Examining Deep Learning Pixel-Based Classification Algorithms for Mapping Weed Canopy Cover in Wheat Production Using Drone Data

Abstract

Deep learning models offer valuable insights by leveraging large datasets, enabling precise and strategic decision-making essential for modern agriculture. Despite their potential, limited research has focused on the performance of pixel-based deep learning algorithms for detecting and mapping weed canopy cover. This study aims to evaluate the effectiveness of three neural network architectures—U-Net, DeepLabV3 (DLV3), and pyramid scene parsing network (PSPNet)—in mapping weed canopy cover in winter wheat. Drone data collected at the jointing and booting growth stages of winter wheat were used for the analysis. A supervised deep learning pixel classification methodology was adopted, and the models were tested on broadleaved weed species, winter wheat, and other weed species. The results show that PSPNet outperformed both U-Net and DLV3 in classification performance, with PSPNet achieving the highest overall mapping accuracy of 80%, followed by U-Net at 75% and DLV3 at 56.5%. These findings highlight the potential of pixel-based deep learning algorithms to enhance weed canopy mapping, enabling farmers to make more informed, site-specific weed management decisions, ultimately improving production and promoting sustainable agricultural practices.

1. Introduction

Weed infestation remains a major problem in wheat production [1]. Weeds often grow contemporaneously, through different growth stages in the phenology of winter wheat. The occurrence of certain weed species depends largely on soil properties and environmental conditions, such as climate and management practices [2]. These conditions allow for the recurring appearance of weeds on many winter wheat fields. The concept of manual weed scouting is generally marked by time and labor constraints, little efficacy, close monitoring, and lots of effort from farmers, especially in large agricultural fields [3,4].

Drones have become effective platforms for obtaining field information for both agricultural and non-agricultural activities [5]. It provides flexibility of flight altitudes and high-spatial-resolution imagery (≥1 cm) to support crops and weed mapping [6]. The continuous effort by farmers to improve crop yield and maximize agricultural productivity is greatly enhanced by drone and satellite technologies [7,8]. Drones also offer farmers the possibility of close monitoring and early-stage detection of weeds in agricultural production such as winter wheat [9,10]. The combined use of drone imagery and remote sensing techniques has adequately proven to be reliable in obtaining growth indices across every stage of crop production [11,12].

Deep learning (DL) is a component of machine learning and, to a larger extent, artificial intelligence (AI). Deep learning uses neural networks (NN) to detect image objects or for image classification [13]. It is a data-driven approach that enables more precise and strategic decision-making [14]. Automated weed canopy cover mapping and analysis through deep learning approaches reduce the need for manual labor and traditional surveying methods, both of which are often time-consuming and labor-intensive [15]. Deep learning classification techniques have been commonly used in areas such as computer vision and pattern recognition [16,17]. However, applications in the agricultural sector are still emerging. Efficient processing of both multispectral and hyperspectral remotely sensed data has been reported using deep learning techniques [18,19,20].

For site-specific weed mapping, de Camargo et al. [17] proposed an optimized deep learning (DL) approach that achieved high accuracy (94%) when differentiating crops and weeds using high-resolution images acquired from unmanned aerial vehicles (UAVs). To understand how traditional image classifiers and deep learning classification algorithms perform, Kussul et al. [19] used the random forest classifier and deep learning (DL) models to map land cover and crop types. They found higher overall accuracy rates for the DL models (92.7–94.6%) compared to the random forest classifier (88.7%). Despite their high performance in image segmentation, Tao et al. [13] assert that changes in the depth of input images, the quality of training data, and model overfitting or underfitting may affect the final output and overall accuracy of DL model architectures used for image pixel classification.

While previous studies have employed deep learning techniques for mapping crop and weed types, there remains a gap in research exploring the comparative performance of different pixel-based deep learning algorithms specifically for mapping weed canopy cover in winter wheat production. Most existing studies have focused on broader applications of deep learning in agriculture or have evaluated individual models without delving into their relative strengths and weaknesses in specific contexts, such as weed mapping in high-density crop systems. Recognizing this gap, the aim of this study is to evaluate and compare the performance of U-Net, DeepLabV3, and pyramid scene parsing network (PSPNet) model classifiers in accurately mapping weed canopy cover within a winter wheat field, using drone-acquired imagery. By doing so, this research seeks to build upon the findings of prior studies, providing a more detailed understanding of the potential and limitations of these models in supporting precision and sustainable agriculture.

2. Materials and Methods

2.1. Study Area

The study was conducted on the Tennessee State University’s urban agricultural field, located in Davidson County (Figure 1). The part of the field used for the study lies between latitude 36.176° N and longitude 86.827° W, close to the Cumberland River. Located in the southeastern region of the country, the area experiences cold winters and warm summers. The average annual high temperature of the area is about 70 °F (21.1 °C) and the average annual low temperature is about 49 °F (9.4 °C). The mean annual precipitation is around 47.2 inches. The soil type of the area is predominantly Byler silt loam (ByB), a moderately acidic soil formed from weathered limestone materials [21].

2.2. Methodology

The methodology (Figure 2) used involved the growing of an unknown winter wheat species. At four growing stages of the wheat (tillering, jointing, booting, and maturity), multispectral images of the field were taken using a drone. The captured images were labeled (trained to identify the different features) and used to train deep leaning models for the jointing and booting growth stages. The models were then used to classify the input images to differentiate the major broad-leaved weeds from the winter wheat and other weed species in the wheat plots. An assessment of the model architectures was made to determine their robustness.

2.2.1. Growing of Winter Wheat

An unknown winter wheat variety was planted on 19 October 2022, in 6 m × 6 m plots. The field was burnt with a non-selective herbicide (2 percent Roundup^®) to manage the initial weeds. The planting of the winter wheat was conducted 2 weeks after the burning with a no-till planter. At the tillering stage, 45 kg of nitrogen was applied to the wheat plots based on recommendations from soil testing. The matured wheat was cut down with a rotary cutter mower on 10 June 2023.

2.2.2. Drone Data Acquisition

An Inspire-2 drone equipped with an Altum multispectral camera was used to take images of the cultivated field during four growth stages (tillering, jointing, booting, and mature) of winter wheat. The drone was flown at an altitude of 15 m above ground level at a speed of 3 m/s, with an overlap rate of 80–90% and a spectral resolution of 1 cm. The images were captured within the spectral bands of blue (450 to 520 nm), green (520 to 590 nm), red (630 to 690 nm), red-edge (690 to 730 nm), near-infrared (NIR) (770–890 nm), and longwave infrared thermal (LWIR) (10,600 to 11,200 nm). The drone images were taken every 2 seconds. The drone images had a spatial resolution of about 1 cm. The images captured were geotagged, radiometrically corrected (using the camera and sun radiation), and orthomosaiced in Pix4D mapper (version 4.8.0). The mosaicked imagery was then clipped to the wheat plots.

2.2.3. Deep Learning Image (Pixel) Classification

A single wheat plot, representative of both the jointing and booting growth stages of the winter wheat, was used for training and classification with the three deep learning model architectures to evaluate their performance in segmenting and distinguishing broad-leaved weed species from winter wheat and other weeds. Supervised classification was performed on the images using ArcGIS Pro (version 3.3.1). The jointing and booting stages were selected for this evaluation due to the dominance and competitiveness of weeds during those stages. A training dataset was created from the input images, with over 2012 and 2205 polygons digitized for the jointing and booting growth stages, respectively. Digitization was based on the distinct floral colors of the weed species visible in the drone imagery (Figure 3) and field digital images. These training data were then exported as classified tiles for deep learning model training with the U-Net, DeepLabV3, and pyramid scene parsing network (PSPNet) architectures, utilizing ResNet34 as the backbone model. The models were trained with 224 × 224 pixel-sized inputs, for a maximum of 20 epochs to ensure consistency. The dataset was split into 80% for model training and 20% for the validation of the model. The two images were classified four times with each model.

2.2.4. Overview of Deep Learning Model Architectures

U-Net, DeepLabV3, and pyramid scene parsing network (PSPNet) are commonly used deep learning architectures for semantic segmentation tasks due to their proven effectiveness in extracting spatial features and accurately segmenting complex image data [22,23,24]. These models were evaluated for their capacity to accurately segment broad-leaved weeds in a winter wheat field. The evaluation aimed to determine how well each model could identify and separate broad-leaved weeds from the surrounding winter wheat crop, a process which is essential for effective weed management and precision agriculture. Each of these models is further explained below.

The U-Net model classifier architecture works within an encoder–decoder workflow [23,25,26]. The encoder consists of several convolutional layers followed by rectified linear unit (ReLU) activations and max-pooling layers. The decoder reconstructs the segmented image from the encoded feature representations. Furthermore, it uses upsampling layers to increase the spatial dimensions of feature maps, thus refining the segmentation. Each upsampling step is followed by convolutional layers to improve the resolution of the output (Figure 4).

The DeepLabV3 architecture (Figure 5) uses atrous convolution (AC) and atrous spatial pyramid pooling (ASPP) methods in its workflow [28,29,30]. The AC is utilized to increase the receptive field of the convolutional layers without losing spatial resolution, allowing the network to capture more contextual information without downsampling the feature maps. ASPP is designed to gather multi-scale contextual information by applying atrous convolutions with different dilation rates in parallel. This enables the model to recognize objects at multiple scales and improves its ability to segment images accurately.

The pyramid scene parsing network classifier architecture is centered on a pyramid pooling module (PPM) of contextual information at multiple scales of the input dataset [24,32]. The PPM consists of several parallel pooling layers with different grid sizes. Each pooling layer partitions the input feature map into different regions and performs pooling within these regions to generate fixed-size feature maps (Figure 6). The enriched feature representation from the PPM is fed into a series of convolutional layers that produce the segmented image. Each pixel in the output map is assigned a class label, corresponding to the different objects or features within the input image [33].

2.2.5. Assessment of the Models

The classified weed canopy cover maps derived using the algorithms from the three model architectures (U-Net, DLV3, and PSPNet) were validated to assess the accuracy and performance of the deep learning algorithms. The validation and accuracy assessments were carried out using the precision, recall, and overall accuracy criteria. The digital field images, together with 400 (jointing stage) and 500 (booting stage) equalized random points, were used to assess the classification accuracy of the models. The precision was estimated using Equation (1), recall using Equation (2), and overall accuracy using Equation (3) [17].

$$\mathrm{Precision} \left(p\right)={\displaystyle \frac{{TP}_i}{{TP}_i+{FP}_i}}$$

(1)

$$\mathrm{Recall} \left(r\right)={\displaystyle \frac{{TP}_i}{{TP}_i+{FP}_i}}$$

(2)

$$\mathrm{Overall} \mathrm{Accuracy} (oa)={\displaystyle \frac{{\sum }_i^k=1C_i}{N}}$$

(3)

where:

• ${TP}_i$ = true positive of class i
• ${FP}_i$ = false positive of class i
• $oa$ = overall accuracy
• $Ci$ = the count of true positives for class i
• $\sum _i^k{C}_i$ = the sum of all true positives across all classes
• $N$ = the total number of instances in the matrix

3. Results

The training loss is the training error of a model as it learns from the training data, while the validation loss demonstrates whether the model underfits or overfits the training dataset. At the jointing stage, the training loss (TL) for the U-Net model started high and decreased steadily as the model learned from the training data, while the validation loss (VL) started low initially and stayed constant as it converged with the TL curve. Similarly, for the DLC3 curve, the training loss started high and decreased steadily with the increase in the number of processed batches. The validation curve started low and stayed constant throughout the processed batches. In contrast, both the training and validation losses in the PSPNet started high and significantly dropped throughout the processed batches (Figure 7).

The pattern of the training and validation losses derived from the U-Net, DLV3, and PSPNet classification models (Figure 8) in the booting stage of winter wheat were like the jointing growth stage loss curves. The training loss in both U-Net and DLV3 started high and sharply decreased with the increase in the number of processed batches. In contrast, the validation loss for both U-Net and DVLV3 started low and steadily plateaued out as the number of processed batches increased. Both training and validation losses in the PSPNet classification algorithms started high and sharply decreased with the increase in the number of processed batches.

Figure 9 compares the performance of U-Net, pyramid scene parsing network (PSPNet), and DeepLabV3 (DLV3) in weed mapping during the jointing growth stage of winter wheat. Both U-Net and PSPNet effectively captured the overall weed distribution, though they tended to overclassify the speedwell species. DeepLabV3 delivered the most refined segmentation, particularly in relation to the differentiation of speedwell and mayweed from winter wheat and other weed species. Generally, U-Net and PSPNet produced more detailed classification maps, while DLV3 provided a balance between clarity and broader weed detection.

Figure 10 illustrates the performance of the three deep learning model classifiers in mapping weeds during the booting growth stage of winter wheat. The U-Net model captured a broad distribution of weeds but showed a notable underclassification, particularly in relation to common vetch and the other weed species. PSPNet produced a more intricate segmentation, though it blended multiple weed species, resulting in a denser yet less distinct classification. In contrast, DLV3 delivered a more distinct segmentation, isolating species like mayweed and hairy buttercup while missing the other weed species. Overall, DLV3 offered the most precise weed identification, while U-Net and PSPNet balanced broader weed detection with varying levels of detail and clarity.

The accuracy assessments in

Table 1. Accuracy assessment metrics for weed canopy cover derived using the U-Net classifier at the jointing stage of winter wheat.

U-Net—Jointing
Reference/class	Mayweed	Speedwell	Others	Wheat
Mayweed	65	24	10	1
Speedwell	1	92	5	1
Others	0	8	76	16
Wheat	0	2	5	91
p (%)	98	73	79	83
r (%)	65	92	76	91
OA (%)	81

,

Table 2. Accuracy assessment metrics for weed canopy cover derived using the DLV3 classifier at the jointing stage of winter wheat.

DVL3—Jointing
Reference/class	Mayweed	Speedwell	Others	Wheat
Mayweed	70	12	15	3
Speedwell	22	55	15	8
Others	5	13	55	27
Wheat	0	4	13	81
p (%)	71	65	56	68
r (%)	70	55	55	81
OA (%)	65

and

Table 3. Accuracy assessment metrics for weed canopy cover derived using the PSPNet classifier at the jointing stage of winter wheat.

PSPNet—Jointing
Reference/class	Mayweed	Speedwell	Others	Wheat
Mayweed	63	28	7	2
Speedwell	1	88	0	7
Others	1	2	71	26
Wheat	4	3	6	87
p (%)	0.91	0.73	0.81	0.71
r (%)	0.63	0.88	0.71	0.87
OA (%)	77

for the jointing stage of winter wheat highlight the classification performance of the models across four vegetation classes. U-Net demonstrated the highest overall accuracy at 81% (Table 1), with the highest precision for mayweed (98%) and the lowest for speedwell (73%). DLV3 recorded the lowest overall accuracy at 65%, with mayweed achieving the highest precision (71%) and the other species having the lowest (56%). PSPNet, with an overall accuracy of 77%, delivered strong results with a precision of 91% for mayweed and 71% for winter wheat (Table 3), showing a competitive performance across key metrics.

At the booting growth stage, the U-Net model achieved an overall accuracy of 69%, with mayweed having the highest precision (90%) and the smallest precision attained for other weed species at 80% (

Table 4. Accuracy assessment metrics for weed canopy cover derived using the U-Net classifier at the booting stage of winter wheat.

U-Net—Booting
Reference/class	Mayweed	Hairy buttercup	Common vetch	Others	Wheat
Mayweed	86	6	0	2	6
Hairy buttercup	6	86	0	5	3
Common vetch	0	2	0	2	96
Others	3	0	0	83	14
Wheat	1	2	0	5	92
p (%)	90	90	0	86	44
r (%)	86	86	0	83	92
OA (%)	69

). The DLV3 model recorded an overall accuracy of 48%, with notable precision for mayweed (91%) and no precision or recall for common vetch and other species (

Table 5. Accuracy assessment metrics for weed canopy cover derived using the DLV3 classifier at the booting stage of winter wheat.

DVL3—Booting
Reference/class	Mayweed	Hairy buttercup	Common vetch	Others	Wheat
Mayweed	62	17	0	0	21
Hairy buttercup	2	79	0	0	19
Common vetch	6	4	0	0	96
Others	1	3	0	0	96
Wheat	1	2	0	0	97
p (%)	91	77	0	0	29
r (%)	62	79	0	0	97
OA (%)	48

). PSPNet outperformed the other classifiers with an overall accuracy of 82% (

Table 6. Accuracy assessment metrics for weed canopy cover derived using the PSPNet classifier at the booting stage of winter wheat.

PSPNet—Booting
Reference/class	Mayweed	Hairy buttercup	Common vetch	Others	Wheat
Mayweed	76	3	6	5	10
Hairy buttercup	3	84	3	5	5
Common vetch	2	3	83	4	5
Others	3	4	2	81	10
Wheat	6	3	4	7	80
p (%)	84	88	86	78	75
r (%)	76	84	83	80	80
OA (%)	82

), showing strong precision across weeds, particularly for hairy buttercup (88%).

4. Discussion

In this paper, we examined the performance of three deep learning models—U-Net, DLV3, and PSPNet—in mapping weed canopy cover within a winter wheat field using drone-acquired images. The training and validation loss (TL and VL) curves revealed key factors that influenced the performance of the three models. At the jointing stage, U-Net’s rapid decline in TL and early VL stabilization indicated efficient learning with minimal overfitting, contributing to its high mapping accuracy. DLV3 showed closely aligned TL and VL, reflecting a balanced training and effective generalization. PSPNet demonstrated a sharp initial learning and a steady loss reduction, supporting its balanced detection capabilities. At the booting stage, U-Net maintained low and stable TL and VL, ensuring a consistent performance, while DLV3’s minimal TL-VL gap highlighted its reliability. PSPNet’s gradual and aligned loss reductions indicated robust generalization, aiding its ability to handle complex vegetation patterns. These trends underscore the potential of these models for effective weed detection and management based on their specific strengths [34].

At the jointing growth stage, all three models demonstrated unique strengths and limitations in mapping weeds among the winter wheat. Both the U-Net and PSPNet classifiers effectively captured the overall weed layout but struggled with oversegmentation, particularly when identifying speedwell. DeepLabV3 (DLV3), on the other hand, excelled at producing detailed and refined segmentation, accurately differentiating speedwell and mayweed from winter wheat and other weed species. During the booting growth stage, U-Net provided a broad weed coverage but suffered from underclassification, particularly for common vetch and other weeds, missing critical details. PSPNet delivered a more intricate weed map but tended to blend multiple species, resulting in overly dense and less distinct classifications. DLV3, again, demonstrated its strength in precision, isolating mayweed and hairy buttercup effectively, although it failed to detect some other weed species, indicating gaps in comprehensive weed detection. In general, the outputs from DLV3 are more applicable for targeted weed control strategies, while the outputs from both PSPNet and U-Net, with their broader weed mapping capabilities, are more suitable for individual weed identification.

The varying classification performance of the three models can be attributed to their distinct architectural designs and to how they process spatial features and contextual information. U-Net’s symmetric encoder–decoder architecture is highly effective at capturing fine-grained details and spatial relationships [35], enabling it to distinguish individual weed species such as speedwell and hairy buttercup with notable accuracy. However, the model’s reverse learning process may have contributed to the missed classifications observed during the booting stage [36]. PSPNet, with its pyramid pooling module, effectively captures multi-scale contextual information [37], enabling a balanced performance across feature segmentation and detection [38]. This ability to detect all weed species at both growth stages, though with noticeable blending, can be linked to its emphasis on contextual understanding [39]. In contrast, DeepLabV3 (DLV3) utilizes atrous convolution layers for dense spatial sampling, which, while enhancing broader feature detection, struggles with capturing finer details [40]. This architectural choice likely contributed to its reduced accuracy in identifying smaller or less distinct weeds, leading to occasional misclassifications. The quality of the input image, as well as that of the training dataset, could have also contributed to the variations in the model’s performance.

When evaluating robustness, U-Net achieved an overall accuracy of 75%, indicating a strong ability to generalize weed mapping. This generalization makes it suitable for large-scale applications but less effective in scenarios where high precision is critical [41,42]. PSPNet, with the highest overall accuracy of 80%, demonstrated superior resilience by balancing clarity and weed identification, making it well-suited for mapping complex vegetation patterns [43]. In contrast, DeepLabV3 (DLV3) achieved an overall accuracy of 56.5%, showing a vulnerability to underclassification which limits its effectiveness in highly variable environments [44]. Despite its lower overall accuracy, DLV3 excelled at precision, making it particularly effective for tasks requiring detailed, species-specific weed mapping. PSPNet’s detailed outputs provide an advantage for mapping diverse weed environments but may struggle with species blending [45], which can reduce its robustness in certain applications. U-Net, while reliable for broader weed coverage, may face challenges in detecting less dominant species, affecting its utility in scenarios requiring detailed mapping. Overall, these results suggest that U-Net and PSPNet are best suited for general weed mapping with high accuracy, while DLV3’s precision makes it ideal for focused weed management and species-specific strategies.

The study demonstrates that using high-resolution drone imagery combined with effective deep learning models can significantly enhance weed management practices, promoting sustainability by reducing blanket herbicide applications. Understanding the relative abundance and distribution of weeds necessitates site-specific weed management practices. Precision agriculture techniques, such as variable rate herbicide application, could then be employed to target weed hotspots more effectively. Overall, this study provides valuable insights into the effectiveness of different deep learning models in the context of precision agriculture, with PSPNet emerging as a particularly strong candidate for accurate weed mapping. Future research should explore the impact of different soil tillage systems on weed cover dynamics during winter wheat production.

5. Conclusions

This study sought to evaluate and compare the performance of U-Net, DeepLabV3 (DLV3), and pyramid scene parsing network (PSPNet) classifiers in accurately mapping weed canopy cover within a winter wheat field using drone imagery. PSPNet emerged as the most accurate model with an overall accuracy of 80%, excelling at general weed mapping across complex vegetation patterns. U-Net achieved a 75% accuracy, demonstrating strong generalization suitable for large-scale weed mapping, while DLV3, with an accuracy of 56.5%, provided high precision for species-specific weed identification. The performance of these models reflects their architectural strengths, with PSPNet balancing clarity and detection, U-Net capturing fine details but struggling with underclassification, and DLV3 excelling at detailed segmentation but missing finer distinctions. These findings underscore the potential of deep learning models to enhance site-specific weed management practices, promoting sustainability in agriculture. Future research will focus on the development of an enhanced DLV3 model tailored for distinguishing different weed types, with the aim of generating precise prescription maps for targeted weed control.