DODA: Diffusion for Object-detection Domain Adaptation in Agriculture

Abstract

The diverse and high-quality content generated by recent generative models demonstrates the great potential of using synthetic data to train downstream models. However, in vision, especially in objection detection, related areas are not fully explored, the synthetic images are merely used to balance the long tails of existing datasets, and the accuracy of the generated labels is low, the full potential of generative models has not been exploited. In this paper, we propose DODA, a data synthesizer that can generate high-quality object detection data for new domains in agriculture. Specifically, we improve the controllability of layout-to-image through encoding layout as an image, thereby improving the quality of labels, and use a visual encoder to provide visual clues for the diffusion model to decouple visual features from the diffusion model, and empowering the model the ability to generate data in new domains. On the Global Wheat Head Detection (GWHD) Dataset, which is the largest dataset in agriculture and contains diverse domains, using the data synthesized by DODA improves the performance of the object detector by 12.74-17.76 AP₅₀ in the domain that was significantly shifted from the training data.

1 Introduction

Real-world scenes are often more complex and diverse than training data, and applying models to the real world has been a challenge because of the gap between training data and the real world. In agriculture, object detection is of great significance for yield estimation, breeding, etc. Agricultural scenes are inherently variable and the problem of domain shift between training data and real scenes is particularly important. It’s often necessary for a breeder to process hundreds of varieties to obtain a new variety, and these varieties are all different. In addition, factors such as weather, flowering time, equipment, crop development period, etc. will promote domain shift, which affects the reliability and usability of the object detection model. For object detection, a common solution is semi-supervised learning because of the expense of artificial labeling, where an existing model is used to recognize new scenes and generate pseudo-labels [1, 2, 3, 4], and then the pseudo-labels are used to train another model, thus extending the model’s ability to recognize different scenes. However, this approach amplifies the existing drawbacks and strengths of the model [2], and it cannot work if the domain shift is too strong and the model is completely unable to recognize certain scenes. Global Wheat Head Detection (GWHD) Dataset [5] is the largest object detection dataset in agriculture, but we observed that simply training on the training set of GWHD, the models cannot recognize certain domains reasonably in the test set because of their differences with the training set, the use of object detection model in practice still relies on expensive and laborious artificial annotation of the target scenes [6, 7, 8].

Figure 1: Overview. (a) DODA can generate high-quality object detection data for new domains by encoding the layout image and the reference image from target domain. (b)Fine-tuning with data generated by DODA can significantly improve the recognition of new domain of various detectors.

A growing number of studies have delved into harnessing generative AI as a data reservoir for addressing data-related challenges. The availability of GPT4 [9] marks a new era in text generation, where high-quality content from large language models is extensively employed in crafting datasets for an array of novel and challenging tasks. Wang et al. [10] leveraged Large Language Models (LLMs) for generating data to facilitate fine-grained hallucination detection. Bernsohn et al. [11] utilized GPT4 [9] for generating data aimed at identifying legal violations and associating victims. Choi et al. [12] employed synthetic tweets to train FACT-GPT, achieving effective identification of false information in social media. Furthermore, in computer vision, the synthesized high-quality images of diffusion [13, 14] are widely used in tasks such as visual representation learning [15, 16, 17], image classification [18, 19], and semantic segmentation [20, 21, 22].

However, due to the challenge of acquiring labels, there are few studies on utilizing diffusion to synthesize data for object detection. A straightforward approach is to synthesize the foreground and background separately (or just the foreground) and then paste the foreground onto the background [23, 24]. However, this method often results in composite images with inadequate consistency. Alternatively, generating images directly using a text-to-image model and then deriving labels through some method [25] may yield more consistent images, yet it encounters the same predicament as pseudo-labeling. Letting diffusion to generate images directly based on the layout [26] is a more promising approach. Nevertheless, due to the inadequate alignment between the layout’s features and those of the image, the quality of the synthesized data is not high. Moreover, unlike the preceding approaches, because of the necessity of using the label-image pairs from the object detection dataset to train the diffusion model, the distribution of the generated images will conform to the training set, thereby restricting the utility of synthetic images solely to augmenting the training set.

To enable the model to generate data for different scenes, especially those that are not included in the training set, the approach introduced in this study, referred to as DODA (Diffusion for Object-detection Domain Adaptation), follows the idea of Domain Adaptation, which separates the learning of the differences between different domains from the core components of the model. Specifically, we employ an additional pretrained visual encoder to provide image features for diffusion, ensuring that diffusion does not learn image features from the data when training with label-image pairs. Thus, when generating datasets, DODA can produce high-quality dataset for the target domain by combining the randomly generated layout with the image features specific to the target domain that are extracted through the visual encoder. Furthermore, unlike the previous approach where layouts were encoded through a text encoder for layout-to-image(L2I) generation, we encode layouts in the form of images. This improves the alignment between layouts and images, thereby enhancing the quality of labels.

Below are the main contributions made in this work:

- We proposed an image-based layout encoding method in layout-to-image generation. Results on the COCO dataset [27] demonstrate that this approach achieves higher accuracy in generating labels compared to previous layout-to-image generation (L2I) methods, achieving a new state-of-the-art result.

- We follow the idea of Domain Adaptation by adopting a design that decouples image features from the core components of the model. This allows the generative model to generate images in a completely new domain without the need for addi-tional training. Through training on datasets synthesized by DODA, the performance of object detectors has notably improved. The improvement remains consistent across object detectors of varying sizes and architectures.

- The decoupled design allows us to train different parts of the model with asymmetric data. We found that pretraining the model with more unlabeled data enables the model to acquire better feature composition ability, thereby enhancing performance in downstream tasks.

2 Related Work

2.1 Image Generation Model

Image generation has garnered significant attention in the last decade, leading to the emergence of various approaches. Among them, the autoregressive model [28, 29] is a classic method that focuses on learning the conditional distribution among pixels to generate images pixel by pixel. Generative Adversarial Networks (GANs) [30, 31, 32] have achieved remarkable success by employing adversarial training between a generator and a discriminator to produce realistic images. Additionally, the autoencoder [33, 34] learns the representation of images in the latent space and thus generates new samples, bringing new ideas and methods to image generation. More recently, methods based on diffusion models [13] have received increasing attention. The diffusion model generates images through a multistep denoising process. By combining the strengths of previous methods, such as using classifier guidance [35], encoding images into latent space [14], etc., it can produce more diverse and realistic images than prior methods. This has sparked wide interest and exploration in the field of image generation [36, 37, 38, 39].

2.2 Generative Models for Different Tasks

There have been early attempts to explore the use of image generators such as GANs to generate images and then train models [40, 41, 42, 43]. With the development of the diffusion model, its ability of learning image representations across datasets and generating diversified images has attracted attention. Consequently, an increasing number of studies are exploring the application of diffusion model to different tasks. For example, some studies have tried to directly use diffusion models as representation learners [44, 45], which is still not as effective as contrastive learning [46] or masked-image-modeling [47], but still a meaningful attempt. More research involves generating images using diffusion models [42, 19, 20, 21, 22] and then training downstream models with the synthesized images. Because of the design of the diffusion model, various conditions, such as text [37, 14, 38], can be easily incorporated into the generation process along with other control, making the process of synthesizing images and training downstream models similar to knowledge distillation [17], which helps downstream models learn the implicit knowledge of large model. For instance, in visual representations learning, the utilization of synthetic data for training [17] can compete with approaches like CLIP [46], DINOV2 [48].

2.3 Layout-to-image Generation

Integrating layout guidance into the image generation process to produce images that correspond to the input layout in terms of categories and positions is termed as layout-to-image. Although there were early attempts in GANs [49, 50], the quality of generated images and the fidelity to the layout were constrained by GANs. Inspired by text-to-image generation tasks, diffusion-based L2I methods [26, 14, 51] represent the layout in text form, and then leverage a text encoder (such as CLIP [46] or Bert [52]) to encode the layout, and integrate layout into the diffusion model through cross-attention. Compared with GAN-based methods, this approach achieves significant improvements in image quality and controllability. However, the text encoder is usually trained on image-text pairs or plain text data, so its ability to encode and align layout with image features is relatively weak, and when encoding the category and position of objects together, not only does it heavily consume tokens, but the category and position also compete with each other, resulting in weaker layout control or poorer image quality. Therefore, we propose representing the layout in images form to reduce the length of token to limit the number of generated objects, and to further improve the controllability of generated images and the quality of labels.

2.4 Domain Adaptation

In practical applications, due to various factors, realworld data may have distributions different from the training data, leading to decreased performance of the model. To address this, the goal of domain adaptation is to enable the knowledge learned by the model in the source domain to be used in the target domain [53]. The basic idea of adversarial-based domain adaptation is to use some method (such as additional loss function [54, 55]) to separate the differences between different domains and reduce the model’s discrimination between different domains, and compelling the feature extraction network to extract a domain-invariant feature for downstream tasks. Following this idea, we use an image encoder to separate image features and ensure that the diffusion model learns only how to generate images based on the layout and the image features provided by the image encoder. In this way, when generating new images, through the image features of the target domain provided by the image encoder, the model can generate data for the target domain without training.

3 Method

Given the random layout and the reference images from the target domain, DODA is designed to generate images that match the input layout and domain features, thus providing object detection data for the target domain. Fig. 2 shows the general structure of DODA.

3.1 Problem Formulation

In this study, our objective is to generate data tailored to a specific domain to enhance the performance of the target detector within that domain:

$$I_{i}=\phi_{\text{DODA}}\left(E_{\text{domain}}\left(I_{\text{domain}}^{j}\right),E_{\text{layout}}\left(I_{\text{layout}}^{i}\right),\varepsilon\right)$$ (1)

Figure 2: The overall structure of DODA.

Here, $\phi_{\text{DODA}}(\cdot)$ takes a reference image $I_{\text{domain}}^{j}$ from the target domain, a randomly generated layout image $I_{\text{layout}}^{i}$, and random noise ($\varepsilon\sim N(0,1)$) as input, to generate images ($I_{i}\in\mathbb{R}^{H\times W\times 3}$) adhering to the layout. Sampling occurs in the latent space, where ($\varepsilon\in\mathbb{R}^{h\times w\times c}$), with $h=H/f$, $w=W/f$, and $f$ representing the downsampling factor relative to the original image. In the GWHD dataset, images contain a lot of textured details. To preserve these texture details as much as possible and reduce the loss of details caused by downsampling, $f$ is set to 4 instead of the more common 8 [14].

3.2 Domain Encoding

In our approach, the generative model does not learn the representation between different domains during training, so that it can use randomly generated labels when generating images and obtain zero-shot image generation capability. To avoid the diffusion model from learning domain bias, we employ a pre-trained visual coder to provide features:

$$F_{\text{domain}}^{i}=E_{\text{domain}}\left(I_{\text{domain}}^{i}\right)$$ (2)

Where the $E_{\text{domain}}(\cdot)$ denotes a VIT-B [56] pre-trained with MAE [47], the MAE trains VIT to learn visual representations of images via masked-image-modeling (MIM). Specifically, during training, 75% of image patches are randomly masked, tasking the model with reconstructing the masked patches. Compared to contrastive learning, models trained with MIM will pay more attention to the texture features of the image rather than the structural features, thereby retaining more high-frequency features [57, 58]. These high-frequency texture features effectively encapsulate domain-specific information. $(F_{\text{domain}}^{i}\in\mathbb{R}^{768\times 1})$ denotes the encoded image features of the target domain $(I_{\text{domain}}^{i}\in\mathbb{R}^{224\times 224\times 3})$. To prevent the model from obtaining layout information from the features provided by MAE, we utilize the [CLS] token as the image feature representation. Moreover, reference images use a different data augmentation method from target images. Further details on data augmentation can be found in the appendix.

3.3 Layout Encoding

The previous L2I method [26, 14, 50] used text encoding to represent the layout of the image, and incorporates it into the diffusion model through cross-attention:

$$L_{i}=\{(b_{i,k},c_{i,k})\}_{k=1}^{K}$$ (3)

$$F_{\text{layout}}^{i}=E_{\text{text}}(L_{i})$$ (4)

$$F_{i}=\text{Attention}(Q_{i},K_{L}^{i},V_{L}^{i})$$ (5)

Where $b_{(i,k)}=(x_{\text{min}}^{(i,k)},y_{\text{min}}^{(i,k)},x_{\text{max}}^{(i,k)},y_{\text{max}}^{(i,k)})\in[0,1]^{4}$ represents the four vertices of a bounding box, $c_{(i,k)}\in[0,C]$ denotes the class of the object within the corresponding bounding box. $F_{\text{layout}}^{i}$ is the layout feature encoded by the text encoder $E_{\text{text}}(\cdot)$, and $F_{i}$ is the feature map in diffusion. $Q$, $K_{L}^{i}$, and $V_{L}^{i}$ represent the query, key, and values of the attention operation for cross-attention with text. Specifically, $Q_{i}=W_{q}F_{i}$, $K_{L}^{i}=W_{k}F_{\text{layout}}^{i}$, $V_{L}^{i}=W_{v}F_{\text{layout}}^{i}$.

Cross-attention is suitable for conveying conceptual information, such as the category of objects, the style of the image. However, layout represents the spatial arrangement of objects in the image, so the above method cannot well align layout features with the image. Additionally, this approach heavily consumes tokens, leading to a significant reduction in the number of encoded objects.

In terms of geometric structure control, enhancing the consistency of geometric structures through the ControlNet [59] is a common practice [60, 61, 62, 63]. Here, we represent the layout in the form of images and then incorporate the layout features into the diffusion model refer to ControlNet:

$$F_{\text{layout}}^{i}=E_{\text{layout}}(I_{\text{layout}}^{i})$$ (6)

$$F_{i}=F_{i}+F_{\text{layout}}^{i}$$ (7)

Where $F_{\text{layout}}^{i}$ represents the layout feature encoded by the image encoder $E_{\text{layout}}(\cdot)$.

ControlNet initializes the conditioning encoding module with the weights of the diffusion encoder, which effectively preserves the diversity of the generated image. Additionally, the pointwise addition of condition features with the original features better retains spatial information compared to cross-attention.

Note that the mask-to-image approach used in ControlNet, where carefully labeled masks accurately represent the shape and location of an object, and there is no overlap between masks. In our L2I task, bounding boxes of different objects only reflect their positions and sizes, with potential overlap between them. Thus, we specifically designed the method for drawing layout images.

For a single-category data set like GWHD, we calculate the overlapping relationship between all objects, and use the greedy algorithm to arrange the bounding boxes of overlapped objects into three different channels of the RGB image. For a multi-category data set like COCO, the bounding box of each object are drawn in descending order of area, ensuring that smaller objects are less likely to be occluded. Objects of the same category within an image are depicted with the same hue but slightly weaker brightness for differentiation.

3.4 Dataset Preparation and Model Training

GWHD dataset is composed of nadir view photographs of wheat. In the GWHD dataset, the images have high resolution but are relatively few in number. Therefore, we divided the original $1024\times 1024$ images into 9 images of size $512\times 512$ with a step size of 256. Thus, following the original dataset’s division of images, the training set contains 32,913 images, and the test set contains 25,722 images. In GWHD, all object detectors have particularly poor recognition of the two domains ’Terraref_1’ and ’Terraref_1’. We merge these two domains into a single entity called ’Terraref’ and utilize it to test the effect of DODA for object detection domain adaptation. The merged ’Terraref’ dataset comprises 2250 images.

For the COCO 2017 dataset, we train with the official training set and test the proposed L2I method on the validation set.

The training of DODA for GWHD can be divided into two stages:

1. The diffusion model training stage, also referred to as the pre-training stage. During this stage, training the upper part of the model in Fig. 1, the model is trained to generate images based on the features provided by MAE.

2. The L2I training stage, where the model is trained to generate images based on layouts, which is the lower part of Fig. 2, and the weights of upper part of the model are frozen.

We use all images from the GWHD training and test sets for pre-training, and image-label pairs from the training set for L2I training. Since bounding box annotation is not required during pre-training, and image generation is decoupled from image feature learning, we can use this asymmetric data for two-stage training. Detailed experiments have been conducted regarding the impact of pre-training dataset size on the quality of generated data.

3.5 Evaluation Metrics

Fréchet Inception Distance (FID). [64] reflecting the quality of the generated image. FID measures feature similarity between generated and real images, the features calculated by the pre-trained Inception-V3 [65].

Inception Score (IS) [66] uses a pre-trained Inception-V3 [65] to classify the generated images, reflecting the diversity and quality of the images.

COCO Metrics refers to fine-tuning detectors using synthetic data, and then calculating AP according to the official COCO [27].

YOLO Score uses a pre-trained YOLOX L [67] to detect the generated image, and calculates the AP between the detection result and the input label, which reflects the ability of the generated model to control the layout.

YOLO Distinction. We use DODA to generate images for the target domain, then use the GWHD-pretrained YOLOX L for detection, and calculate the AP50 between the detection result and the input label. Because YOLOX is not fine-tuned on the target domain, it can recognize domains in the GWHD training set but cannot recognize the target domains. It reflects the difference between the generated images and the GWHD training set for the detector.

4 Experiment

4.1 Comparison with T2I-based L2I model

In this section, we compare our proposed L2I method with other text-to-image based L2I methods.

4.1.1 Quantitative Comparison

Table 1: Quantitative results on COCO-stuff-val2017. SD1.5 refers to stable diffusion v1.5. Max obj refers to the maximum number of objects that can be generated in a picture, limited by the length of the token.

Method	Base model	Max obj	YOLO Score$\uparrow$						FID$\downarrow$	IS$\uparrow$
			mAP	AP50	AP75	APs	APm	APl
$256\times 256$
LayoutDiffusion[51]	-	8	6.0	14.9	3.8	0.2	4.7	19.8	20.5	21.8$\pm$1.1
GeoDiffusion[26]	SD1.5	18	27.3	8.5	29.3	2.8	40.3	63.2	34.3	24.7$\pm$1.1
DODA(ours)	SD1.5	-	28.4	45.5	28.5	10.0	38.2	54.8	30.2	31.2$\pm$1.0
$512\times 512$
GeoDiffusion[26]	SD1.5	18	27.7	40.7	29.6	0	13.0	57.8	28.8	26.4$\pm$1.1
DODA(ours)	SD1.5	-	39.7	59.0	41.9	14.7	39.0	56.3	24.0	32.4$\pm$1.0

To provide a fair comparison of the L2I method, we followed the setup of GeoDiffusion [26] and trained the model based on Stable Diffusion [14] v1.5. Since Stable Diffusion is a T2I model, we constructed a simple text prompt for our method. Assuming that an image contains objects of i different categories, the prompt is formulated as:

$$prompt=^{\prime}\text{a photograph with }(N_{\text{cls}}^{1})({Cls}^{1}),(N_{{cls}}^{2})({Cls}^{2}),\ldots,(N_{{cls}}^{i})({Cls}^{i})^{\prime}$$

Where $Cls^{i}$ represents a specific category, and $N_{cls}^{i}$ denotes the number of objects belonging to that category.

The results are shown in Table 1. It can be seen that using the layout-image-to-image method can achieve better overall control of the image layout, and the number of generated objects is not limited by the token length. Compared to previous methods, there is an improvement of 11.3 in mAP. When the object area is small, because of the competition between category and position within the prompt, the positions of small objects may be directly ignored, or result in loss of image diversity (As shown in Fig. 3), while DODA achieved the highest APS and IS, which means it can better generate small objects and preserves more details, and. Additionally, after increasing the resolution, DODA’s image control capability rapidly improves, which is crucial for constructing high-quality datasets.

However, DODA’s ability to handle large objects is not as strong. This is because in drawing layout images, we simply arrange objects in descending order of area without considering their true depth relationships in the image, leading to the possibility of large objects being incorrectly occluded. And in terms of FID, due to the utilization of Stable Diffusion as the base model with frozen weights, although more diversity is retained, the FID score is higher compared to LayoutDiffusion, the model that was fully trained on COCO.

Qualitative Comparison Fig. 3 illustrates a comparison of $256\times 256$ images generated on the COCO validation set using our method and previous SOTA L2I approaches.

Figure 3: Visualization of comparison with other L2I methods on COCO. DODA generates images with more detail and greater control over layout, especially for small objects, and is suitable for building datasets.

Table 2: Performance of different models before and after fine-tuning using synthetic domain-specific data.

Method	Parameters	AP50$\uparrow$	AP75$\uparrow$	APs$\uparrow$	APm$\uparrow$	APl$\uparrow$
Deformable DETR[68]	41M	10.2	0.5	0.3	3.1	10.5
+ synthetic dataset		24.8	3.	0.54	9.9	30.2
YOLOX L[67]	54M	30.5	5.0	0.7	12.9	42.0
+ synthetic dataset		48.3	11.6	3.0	22.6	42.3
YOLOV7 X[69]	71M	27.2	7.5	0.2	13.9	40.4
+ synthetic dataset		42.9	11.7	2.2	21.2	40.4
FCOS X101[70]	90M	20.1	1.4	0.4	6.1	17.5
+ synthetic dataset		33.8	5.4	1.9	14.3	21.1

4.2 Synthetic data for Object Detection Domain Adaptation

In this section, we employ DODA to generate data for the domain ‘Terraref’ and evaluate the effectiveness of the generated data on a range of object detection models with varying sizes and structures. For each detector, we initialize it with weights of a model pre-trained on COCO, train on GWHD’s training set and test their performance on ‘Terraref’ as the baseline. We randomly sample images from ‘Terraref’ as references, combine them with randomly generated layouts, and leverage DODA to create a synthetic dataset comprising 800 images. We then repeatedly fine-tune the object detectors using this synthetic dataset for 5 times, and average the result. The results are illustrated in Fig. 1 (b) and summarized in Table 2.

After fine-tuning with the data synthesized by DODA, all models show significant improvement in recognizing ‘Terraref’, indicating DODA’s capability to extract representation of a specific domain and transform them into knowledge that object detection models can utilize, assisting detectors in recognizing new and different domains.

As shown in Fig. 1 (b), different detectors have different starting points, but improvement rates are relatively close. DODA cannot help reduce the gap between detectors, suggesting that synthetic data only helps the detectors to transfer knowledge, while the final recognition performance still depends on the detectors’ inherent recognition abilities.

4.3 Ablation Study

To further understand the significance of each design in the proposed method for domain adaptation, we conducted an ablation study. We explored and analyzed in detail the impact of domain encoder and pre-training dataset size on the quality of data generation, as well as the specific role of generated data.

Figure 4: (a) Without the domain encoder, DODA learns the average representation of the entire dataset, and images are highly similar to each other. (b)Fine-tuning the YOLO X with dif-ferent numbers of synthetic images, 800 images are enough.

Domain Encoder. After removing the domain encoder module from the model, we tested the model’s ability to generate images. As illustrated in Fig. 4 (a), without the domain encoder to separate image features, the diffusion model learns the average representation of the entire dataset, the generated images are highly similar to each other, and can’t be used for training downstream models. Separating features through the domain encoder not only allows the model to have the ability to generate images for specific domains, but more importantly, it can avoid learning the average representation of the data set during the training process.

Number of generated images. Fine-tuning YOLOX L with more images, as shown in Fig. 4 (b). 800 images are sufficient to convey information about the target domain. Further increasing the number of images only results in oscillation in the performance of the detector without further improvement. The primary role of DODA is to assist the detector in knowledge transfer rather than data augmentation.

Table 3: Performance of different models before and after fine-tuning using synthetic domain-specific data.

Additional unlabeled data	FID$\downarrow$	YOLO Distinc-tion$\downarrow$	AP50$\uparrow$	AP75$\uparrow$
Baseline	-	-	30.5	5.0
0%	177.4	81.1	28.3	5.8
25%	158.4	80.3	30.1	5.4
50%	164.3	76.6	31.1	4.9
75%	144.1	76.5	32.2	5.5
100%	158.8	74.2	36.2	7.6
+Terraref	110.3	57.2	40.6	6.2

Figure 5: Illustrations of images generated for the ‘Terraref’ domain after pre-training with different scales of unlabeled additional data. With more data for pre-training, the model generates images with richer details. After additional pre-training using images in ‘Terraref’, the gen-erated images closely match the reference image’s style.

Scaling dataset. The separation of image features by the domain encoder allows us to pre-train the diffusion model using a larger dataset, without worrying about overfitting during the training of the layout encoder due to differences in data distribution. We explored the variation in data quality generated for ‘Terraref’ when training with different amounts of unlabeled data. Here, the unlabeled data refers to the GWHD test set, which will not be used when training the layout encoder.

Specifically, we specify data quality as image quality and label quality. Image quality is evaluated by FID and YOLO Distinction, and label quality is evaluated based on the performance of the detector that fine-tuned with the generated images. The quantitative results are shown in Table 3. When no additional unlabeled data was used, the FID was relatively high, indicating poor image quality, and the generated data even impaired the performance of the detector. As the size of pre-training dataset increases, FID cannot capture changes in images very well, but YOLO Distinction is steadily decreasing, while data quality is steadily increasing. DODA can better distinguish and extract features of specific domains, and these features are helpful for the detector.

After additionally using images in ‘Terraref’ for pre-training, DODA could generate images with styles almost identical to the original images (as seen in the last column of Fig. 5). The performance of the downstream model has been further improved, but the improvement is not as good as Image quality improvement. Although image similarity increased significantly, control over the image layout weakened, leading to a decline in label quality. This may be because when the data from the target domain is not used for pre-training, the model simply combines the learned knowledge about the foreground and background from other data with the embedding of target domain. After pre-training with data from the target domain, the model will use less typical features from other domains and tend to identify the foreground and background of the target domain by itself. However, due to the difficulty of the data, the recognition is not good, so the layout control is weakened.

5 Limitation

After training with images generated by DODA, the detector shows a significant improvement in recognizing ‘Terraref’. However, compared with some simple domains (AP>=80), the recognition performance is still relatively limited. On GWHD, pretraining with larger unlabeled data can enhance the quality of generated data, and no performance saturation is observed. Collecting a more diverse range of images may be a potential approach to address this issue.

The proposed layout-image-to-image method exhibits a significant enhancement in controlling the layout compared to previous state-of-the-art layout-to-image methods, which means higher label quality. However, there are still erroneous labels present. Due to resource limitations, we performed simple fine-tuning using the generated data without focusing on improving scores, such as by incorporating certain semi-supervised learning methods to further enhance label quality.

6 Conclusion

In this paper, we propose an objection detection data generation model called DODA, which synthesizes data for the target domain zero-shot by utilizing unlabeled reference images from the target domain and the randomly generated layouts. Through comprehensive experiments, we demonstrate its effectiveness in alleviating the performance degradation of object detectors caused by significant domain shifts in agricultural scenes.

Acknowledgments. This material is based upon work supported by the Google Cloud Research Credits program. This study was partially supported by the Japan Science and Technology Agency (JST) AIP Acceleration Research (JPMJCR21U3)

Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

Appendix

A. Visualization of image representationsitle

Figure 6: 2D visualization of image representations of the GWHD training set, ‘Terraref’ domain, and the generated images. The generated images are closer to the ‘Terraref’ domain.

We encode the images with MAE and take the [CLS] token as the representations of the images.and employ the UMAP [71] to reduce feature dimensions to 2 for visualization. As shown in Fig. 6, the GWHD training set is remote from the ’Terraref’ domain. Using images from ’Terraref’ as the reference, DODA generates images closer to the target domain, and synthetic data can help downstream models adapt to new domain.

B. Hyperparameters

Table 4: Hyperparameters for pre-training DODA. DODA leverages latent diffusion (LDM) [14] as the base diffusion model, which uses variational autoencoder (VAE) [33] to encode the image into the latent space and thus reduces the computation, so the pre-training of DODA is divided into two stages: the VAE and LDM.

		VAE	LDM
Dataset		All images in GWHD	All images in GWHD
Target Image Shape		$256\times 256\times 3$	$256\times 256\times 3$
Domain Reference Image Shape		-	$224\times 224\times 3$
Data Augmentation	Target Image	Random Rotation	Random Rotation
		Random Crop	Random Crop
		Random Flip	Random Flip
	Reference Image	-	Random Crop
f		4	4
Channels		128	224
Channel Multiplier		1,2,4	1,2,4
Attention Resolutions		-	2,4
Number of Heads		-	8
Learning Rate		2e-6	2.5e-5
Iterations		170k	315k
Batch Size		8	16

Table 5: Hyperparameters for layout-to-image.

Dataset		COCO 2017 training	COCO 2017 training	GWHD training
Target/Layout Image Shape		$256\times 256\times 3$	$512\times 512\times 3$	$256\times 256\times 3$
Domain Reference Image Shape		-	-	$224\times 224\times 3$
Data Augmentation	Target Image	Random Flip	Random Flip	Random Rotation
				Random Crop
				Random Flip
	Reference Image	-	-	Random Crop
Base Model		SD1.5	SD1.5	LDM in Table 4
f		8	8	4
Channels		320	320	224
Channel Multiplier		1,2,4,4	1,2,4,4	1,2,4
Attention Resolutions		1,2,4	1,2,4	2,4
Number of Heads		8	8	8
Learning Rate		2.5e-5	2.5e-5	1e-5
Iterations		100K	30K	80K
Batch Size		16	8	16

C. More qualitative comparisons with previous L2I methods on COCO

Figure 7: More comparisons with other methods on COCO 2017 validation set.