An Improved Method for Human Activity Detection with High-Resolution Images by Fusing Pooling Enhancement and Multi-Task Learning

Abstract

Deep learning has garnered increasing attention in human activity detection due to its advantages, such as not relying on expert knowledge and automatic feature extraction. However, the existing deep learning-based approaches are primarily confined to recognizing specific types of human activities, which hinders scientific decision-making and comprehensive environmental protection. Therefore, there is an urgent need to develop a deep learning model to address multiple-type human activity detection with finer-resolution images. In this study, we proposed a new multi-task learning model (named PE-MLNet) to simultaneously achieve change detection and land use classification in GF-6 bitemporal images. Meanwhile, we also designed a pooling enhancement module (PEM) to accurately capture multi-scale change details from the bitemporal feature maps through combining differencing and concatenating branches. An independent annotated dataset at Yellow River Delta was taken to examine the effectiveness of PE-MLNet. The results showed that PE-MLNet exhibited obvious improvements in both detection accuracy and detail handling compared with other existing methods. Further analysis uncovered that the areas of buildings, roads, and oil depots has obviously increased, while the farmland and wetland areas largely decreased over the five years, indicating an expansion of human activities and their increased impacts on natural environments.

1. Introduction

Human activities encompass a diverse range of undertakings conducted continuously for survival, development, and the enhancement of living standards [1]. With the acceleration of industrialization and urbanization globally, human activities have increasingly impacted the natural environment, leading to environmental pollution and ecological degradation, such as deforestation, lake reclamation [2], and overexploitation [3]. Consequently, real-time monitoring of human activities is crucial for urban planning and sustainable development.

Two primary means of monitoring human activities are aerial remote sensing and satellite remote sensing [4]. Aerial remote sensing, leveraging aircraft or other airborne platforms equipped with multi-band scanners, cameras, and radars, offers high accuracy and flexibility but is limited in scope for large-area, long-term monitoring due to aircraft performance and economic constraints [5]. In contrast, satellite remote sensing boasts global coverage, continuous observation, and low data acquisition costs [6,7,8], making it widely applicable in urban planning, land management, resource exploration, and disaster monitoring [9,10,11]. Notably, the advent of high-resolution satellites has provided finer data sources for capturing intricate texture structures and change details and thus facilitating more precise land identification [12], although high-resolution images usually offer less spectral information compared to medium- and low-resolution images, like Landsat TM and MODIS [13].

Change detection, defined as “the process of identifying differences in the state of an object of phenomenon by observing it at different times” [14], is an effective means of extracting human activity information through bitemporal image change mapping [15]. Early change detection methods relied on visual interpretation, which was both knowledge-intensive and impractical for large-scale remote sensing data [16]. With advancements in remote sensing and computer technology, automated and intelligent change detection methods have emerged [17]. Algebra-based methods directly compute pixel differences and apply preset thresholds to generate change maps [18], while transformation-based methods use Principal Component Analysis (PCA), Tasseled Cap transformation, or Gramm–Schmidt transformation to separate change components before applying thresholds [19,20,21]. Both approaches face challenges in accurately setting thresholds due to the complexity of and variability in remote sensing data [22]. Classification-based methods compare results after supervised or unsupervised classification results, reducing image information utilization [23]. Advanced models, like reflectance models and spectral mixture models, convert spectral information into physical parameters for change analysis, showing good performance in specific scenarios but limited generalizability [24].

Deep learning has garnered significant attention in recent years, particularly in the fields of image processing and remote sensing [25]. Compared to traditional change detection methods, deep learning approaches excel in their ability to directly learn change features from bitemporal or multi-temporal remote sensing images [26]. By leveraging these features to segment the images, deep learning models generate change maps with superior robustness [27]. Consequently, deep learning methods have been widely employed in human activity detection [28]. For instance, de Bem et al. [29] utilized three convolutional neural networks (SharpMask, U-Net, and ResUnet) to map deforestation in the Amazon region of Brazil from 2017 to 2019 and all outperformed traditional machine learning approaches. Murdaca et al. [30] introduced a semi-supervised deep learning framework to detect changes in open-pit mines, which presented excellent robustness through the incorporation of pseudo-labels. Meanwhile, D’Addabbo et al. [31] successfully detected newly constructed buildings in urban areas by leveraging pre-trained AlexNet to extract deep features and applying transfer learning techniques. Although these studies underscore the efficacy of deep learning in human activity detection, the majority of current research focuses exclusively on specific types of human activities, lacking a deep learning algorithm capable of addressing the detection of multiple types of human activities.

To address the limitations in the existing research, this study aims to develop a multi-class human activity detection model (PE-MLNet) based on high-resolution imagery utilizing a multi-task learning approach. This model not only performs accurate land use classification of bitemporal remote sensing images but also simultaneously identifies areas of change, thereby enabling comprehensive detection of human activities within the study region. Furthermore, we propose a pooling enhancement module (PEM) with a dual-branch structure for capturing both global and local details of changes in bitemporal images.

2. Materials and Methods

2.1. Study Region

The study area for this research is located in the Yellow River Delta of China. As shown in Figure 1, this region has a diverse array of land use types, encompassing natural landscapes such as rivers, wetlands, and lakes while simultaneously experiencing the impact of human activities, like urban expansion, land development, and pond aquaculture [32]. This study collected three GF-6 images covering the Yellow River Delta across June 2019, July 2021, and June 2023, which include one 2 m panchromatic band and four 8 m multispectral bands, namely, the red, green, blue, and near-infrared bands. To accurately capture the characteristics of complex land objects, we first fused the panchromatic and multispectral bands to produce multispectral imagery with a resolution of 2 m.

The sample collection strategy is illustrated in Figure 2. Specifically, 23 regions of different sizes were selected for this study (Figure 2a), covering the majority of typical human activity types and natural land covers. Manual annotation was then performed at the selected sampling points (Figure 2b). For the annotation of change detection (Change), a binary labeling system was adopted, with 1 representing “changed” and 0 representing “unchanged”. As for land use annotation (Landuse1 and Landuse2), this study focused on nine land types related to human activities, including building, natural water, cropland, aquaculture pond, salt field, road, oil depot, wetland, and others. Subsequently, the images and samples were cropped into 256 × 256 tiles, resulting in 1351 sets of training data. After random partitioning, 1204 sets were designated as the training set, and 135 sets were designated as the validation set. As shown in Figure 2c, the data labels include land use labels (label1 and label2) and change detection labels (label_cd) for bitemporal images.

Table 1. The label category and the number of pixels in the YRD dataset.

Task	Class	Train	Validate
Change Detection	Changed	3.7 × 107	4.7 × 106
	Unchanged	2.0 × 108	2.2 × 107
Semantic Segmentation	Others	9.8 × 107	1.5 × 107
	Building	1.0 × 107	2.0 × 106
	Natural Water	9.4 × 107	9.4 × 106
	Cropland	1.5 × 108	1.5 × 107
	Aquaculture Pond	5.9 × 107	5.0 × 106
	Salt Field	2.3 × 107	2.0 × 107
	Road	3.1 × 106	3.5 × 105
	Oil Depot	1.3 × 106	5.0 × 105
	Wetland	4.4 × 107	4.7 × 106

provides a summary of the label categories and pixel statistics for the YRD dataset.

2.2. Methodology

This study designed a human activity detection model called PE-MLNet based on pooling enhancement and multi-task learning. This model not only identifies the change area of dual-temporal images but also classifies the land use of the front and back images, so as to realize the detection of multiple types of human activities in the study area, with the workflow illustrated in Figure 3. It mainly includes four parts:

(1)

Feature Extractor: It extracts common semantic features from the bitemporal images.

(2)

Semantic Segmentation Module: Utilizing the feature maps obtained from the feature extractor as input, it outputs semantic segmentation results.

(3)

Pooling Enhancement Module: It captures multi-scale change details within the feature maps.

(4)

Change Detection Module: Taking the multi-scale change feature maps extracted by the PEM as input, it produces the change detection results.

Based on the aforementioned four modules, multi-task learning encompasses three tasks: (1) change detection in bitemporal images, (2) semantic segmentation, and (3) measuring the consistency between semantic segmentation results and change detection results.

2.2.1. Feature Extractor

In traditional change detection approaches, a single-branch network is often utilized as a feature extractor, where bitemporal images are superimposed and directly input to obtain a difference map [33]. However, this method is prone to information loss and often makes it challenging for the network to converge [34]. To address this issue, we introduced a dual-branch network based on shared parameters as the feature extractor, which is capable of better extracting semantic information and differences from bitemporal images. The branch network comprises four convolutional blocks, each consisting of Conv 3 × 3, batch normalization, and ReLU. When bitemporal images are input into the feature extractor, they undergo multiple downsampling processes, resulting in a continuous reduction in the size of the feature maps and a gradual increase in the number of channels.

2.2.2. Semantic Segmentation Module

To extract semantic features from dual-phase images and obtain semantic segmentation results, the semantic segmentation module in this study adopts the structure of PSPNet [35]. The spatial pyramid pooling (SPP) module in PSPNet enhances the receptive field of the network and captures richer semantic information from the image. After the feature maps extracted by the feature extractor enter the semantic segmentation module, they first pass through the SPP module where average pooling is applied using different kernel sizes of 1 × 1, 2 × 2, 3 × 3, and 6 × 6. The channel number is then reduced using 1 × 1 convolutions. Subsequently, the feature maps are upsampled to restore their original dimensions and are concatenated with the feature maps before entering the SPP module. Following this, two 1 × 1 convolution blocks are applied to reduce the channel number without altering the dimensions and to predict the probability of each pixel belonging to different categories. Finally, the segmentation results are output after interpolation upsampling.

2.2.3. Pooling Enhancement Module (PEM)

In the field of change detection research, two common methods for generating change feature maps are differencing and concatenating. However, both approaches have inherent limitations [36]. The differencing method creates a change feature map by subtracting and then taking the absolute value. This method is susceptible to lighting and weather conditions, leading to unstable results. Additionally, the high variability in high-resolution imagery spectra introduces substantial noise during differentiation. The concatenating method connects the deep features of dual-temporal images across channels, preserving a majority of the semantic information. Yet it results in information redundancy, increases computational costs, and lacks interpretability.

Addressing the limitations of these methods, this study introduced the PEM to capture global and local change details while preserving semantic information (Figure 4). The PEM comprises concatenation and differentiation branches. Upon receiving the feature maps (F1 and F2) from the feature extractor, it produces a connecting feature map (A) and a differential change map (D) through respective connection and differentiation processes. The connecting feature map employs 3 × 3 and 1 × 1 convolution blocks to detect contextual information and reduce channel numbers. For the differential change map, spatial pyramid pooling is first applied, followed by average pooling using four types of receptive fields (1 × 1, 2 × 2, 3 × 3, and 6 × 6). This extracts multi-scale change information from the map, which is then upsampled and combined with the original map. Finally, contextual information is detected using 3 × 3 convolution blocks, and pixel values are summed with those from the connecting branch. The entire process can be described as follows:

$$A=\left[F_1,F_2\right]$$

(1)

$$D=\left|F_1-F_2\right|$$

(2)

$$A^{\prime }=Conv1\left(Conv3\left(A\right)\right)$$

(3)

$$D^{\prime }=SPP\left(D\right)$$

(4)

$$F^{\prime }=A^{\prime }+D^{\prime }$$

(5)

where $\left[·\right]$ represents concatenation on the channel dimension, $\left|·\right|$ denotes the absolute value, and $Conv3$ and $Conv1$ refer to 3 × 3 and 1 × 1 convolution layers, respectively, followed by batch normalization and ReLU. PEM integrates semantic information from dual-temporal feature maps with multi-scale change details, effectively enhancing the representation of changed regions while suppressing irrelevant distractions. Through the PEM, four pooling-enhanced change maps of different scales are sequentially generated.

2.2.4. Change Detection Module

The PEM extracts four types of pooling-enhanced change maps at different scales, employing skip connections to pass multiple change maps to the change detection module. The change detection module is designed as an alternating combination of convolutional and upsampling blocks. Each upsampling block comprises Transposed Conv 3 × 3, batch normalization, and ReLU. After the deep change map enters the change detection module, it first reduces the channel count through convolution and then overlaps with the change map from the previous layer through upsampling. Throughout the process, four upsampling operations are performed to gradually restore the change map’s size and predict the probability of pixel changes.

2.2.5. Loss Function

This study introduces a multi-task loss function to optimize multi-task learning, consisting of three components: the change detection loss $L_{cd}$, the semantic segmentation loss L_se, and the consistency loss L_cs between the change detection and semantic segmentation results.

The change detection loss $L_{cd}$ quantifies the discrepancy between the binary change map I_cd predicted by the change detection module and label_cd. The BCELoss function is employed to compute L_cd, with the formula for each pixel being the following:

$$L_{cd}=-y_c\log \left(p_c\right)-\left(1-y_c\right)\log \left(1-p_c\right)$$

(6)

where $y_c$ indicates whether the pixel has changed in label_cd, with 1 indicating change and 0 indicating no change. $p_c$ represents the probability of the change detection module predicting a change in the pixel.

Semantic segmentation loss $L_{se}$ represents the discrepancy between the segmentation prediction results I1 and I2 and the corresponding label1 and label2 for the dual-temporal images produced by the semantic segmentation module. In this study, $L_{se}$ is computed using the CELoss, which is frequently employed in multi-classification tasks. The calculation formula for each pixel is as follows:

$$L_{se}=-{\displaystyle \frac{1}{k}}{\displaystyle \sum }_{i=1}^k{y}_i\log \left(p_i\right)$$

(7)

where $k$ denotes the number of semantic segmentation categories. $y_i$ denotes the category in label1 or label2, while $p_i$ signifies the predicted probability for category $i$ output by the semantic segmentation module.

$L_{cd}$ and $L_{se}$ are, respectively, used to optimize the learning of change detection tasks and semantic segmentation tasks. Additionally, this study proposes a loss $L_{cs}$ that connects semantic segmentation and change detection. By calculating the loss between the difference (non-zero difference results are converted to 1) of the prediction results I1 and I2 of semantic segmentation and label_cd, the semantic information changes in the semantic segmentation prediction results are kept consistent with the change detection prediction results. The MSE is used to calculate L_cs, and the calculation formula is as follows:

$$L_{cs}={\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\left(y_i-p_i\right)}^2$$

(8)

where $N$ represents the number of pixel points, $y_i$ is the true value of the $i$-th pixel point in label_cd, and $p_i$ is the predicted value of the $i$-th pixel point in the difference result.

The loss for multi-task learning L is set as the sum of the semantic segmentation loss, change detection loss, and consistency loss.

$$L={\lambda }_1{L}_{cd}+{\lambda }_2\left(L_{s{e}_1}+L_{s{e}_2}\right)+{\lambda }_3{L}_{cs}$$

(9)

where $L_{s{e}_1}$ and $L_{s{e}_2}$ represent the semantic segmentation losses for each temporal phase, respectively. ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ denote the hyperparameters representing weights.

3. Results and Discussion

3.1. Experimental Setup and Accuracy Assessment

The experiments in this study were conducted on a workstation equipped with an NVIDIA RTX A4000 GPU. All the models were built using the PyTorch library [37], with an iteration count of 300. The Adam algorithm [38] was chosen as the gradient optimization method, and the batch size was set to 10. To enhance the diversity and quantity of the data, random flipping and mirroring were employed. For the multi-task loss function L, considering that change detection and semantic segmentation are equally important, we hoped to achieve a balance between them during the training process. Therefore, we empirically set ${\lambda }_1$ = 2 and ${\lambda }_2$ = ${\lambda }_3$ = 1 in order to unify the magnitude of different loss values. The initial learning rate was set at 0.0001 and gradually decayed during the training process, according to the following formula:

$$lr=lr\_init\times {\left(1-{\displaystyle \frac{epoch}{\max \_epoch}}\right)}^{0.9}$$

(10)

where $lr$ represents the current learning rate, $lr\_init$ denotes the initial learning rate, $epoch$ signifies the current iteration count, and $\max \_epoch$ indicates the total number of iterations.

To verify the effectiveness of the model, a quantitative analysis was conducted by comparing the output results with the labels. For change detection, this study selected four evaluation metrics for analysis, i.e., precision (P), recall (R), overall accuracy (OA), and F1-score (F1). A higher P indicates fewer false positives, while a higher R signifies fewer false negatives. Both F1 and OA serve as comprehensive evaluation metrics, and higher values represent better model performance. For semantic segmentation, this study evaluated model performance using five metrics, including the Intersection over Union (IOU), mean Intersection over Union (mIOU), overall accuracy (OA) [39], Average Accuracy (AA), and Kappa coefficient (Kappa) [40]. The IOU represents the ratio of the intersection and union between the predicted and true regions, measuring the overlap between the model’s segmentation results and the ground truth labels. The mIOU is the average of the sum of the IOUs for all the categories. The AA represents the average of the sum of the recalls for each category, and the Kappa assesses the consistency of the segmentation results. The formulas are as follows:

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(11)

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(12)

$$OA={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(13)

$$F1={\displaystyle \frac{2\times P\times R}{P+R}}$$

(14)

$$IOU={\displaystyle \frac{TP}{TP+FN+FP}}$$

(15)

$$mIOU={\displaystyle \frac{1}{k}}{\displaystyle \sum }_{i=1}^{k}IO{U}_i$$

(16)

$$AA={\displaystyle \frac{1}{k}}{\displaystyle \sum }_{i=1}^k{R}_i$$

(17)

$$Kappa={\displaystyle \frac{OA-PRE}{1-PRE}}$$

(18)

$$PRE={\displaystyle \frac{\left(TP+FP\right)\times \left(TP+FN\right)\times \left(TN+FN\right)\times \left(TN+FP\right)}{{\left(TP+TN+FP+FN\right)}^2}}$$

(19)

where $TP$, $FP$, $TN$, and $FN$ represent the number of positive samples correctly detected, the number of negative samples detected as positive, the number of negative samples correctly detected, and the number of positive samples detected as negative, respectively. Additionally, $k$ denotes the number of semantic segmentation categories.

3.2. Comparative Methods

To validate the effectiveness of the proposed model, this study compares it with several classic change detection networks and semantic segmentation networks. For change detection, the following networks are selected:

(1)

Fully Convolutional-Early Fusion (FC-EF) [41]: This approach employs an early fusion strategy, where bitemporal images are concatenated and fed into an FCN to obtain a change map. Additionally, skip connections are incorporated to supplement spatial information.

(2)

Fully Convolutional Siamese-Concatenation (FC-Siam-conc) [41]: Based on FC-EF, this method replaces the encoder with a dual-branch structure sharing weights. Deep features from both temporal phases are extracted, concatenated, and then input into a decoder to generate a change map.

(3)

Fully Convolutional Siamese-Difference (FC-Siam-diff) [41]: This structure is similar to FC-Siam-conc but with one key difference. After the encoder extracts deep features from both temporal phases, they are fused using a difference approach.

(4)

Image Fusion Network (IFN) [42]: This network introduces a Deep Difference Discrimination Network (DDN) based on an attention mechanism. It integrates multi-level deep features with image difference features to construct a change map.

(5)

ChangeNet [43]: This network utilizes ResNet to extract change information at different scales, which is then processed by a unified decoder for change detection, outputting semantically meaningful change detection results.

(6)

DTCDSCN [44]: This network introduces a spatial feature pyramid pooling module as its central component, which expands the receptive field of the feature maps and incorporates contextual features across different scales. Additionally, two extra semantic segmentation decoders are trained simultaneously.

For semantic segmentation, this study selects the following networks:

(1)

Fully Convolutional Network (FCN) [45]: Based on the VGG16 classification network, this approach replaces the fully connected layers with convolutional layers. Additionally, skip connections are added to combine deep semantic information with superficial information, aiming to generate precise segmentation results.

(2)

UNet [46]: This network architecture forms a symmetrical “U” shape, consisting of an encoder and a decoder. The encoder is responsible for feature extraction, while the decoder performs upsampling through deconvolutions layer by layer, restoring the feature map size and outputting the segmentation results.

(3)

SegNet [47]: The encoder adopts the network structure of VGG16, while the decoder utilizes pooling indices for upsampling to achieve pixel-level classification.

(4)

HRNet [48]: The network maintains high-resolution features of the image through the use of parallel connections while simultaneously fusing features of different resolutions through repeated information exchange modules.

3.3. Change Detection

For change detection, all the models showed a stable convergence trend during training (Figure 5a) and high accuracy over 90% (Figure 5b). This indicates that all the models are capable of effectively learning from the training data and enhancing their performance. Specifically, the FC series models exhibited a relatively fast convergence rate, stabilizing after approximately 25 epochs, with an OA reaching around 92%. The IFN model showed a higher initial loss and lower OA during the early stages of training, with significant fluctuations in OA throughout the process. Although ChangeNet experienced significant OA fluctuations in the first 50 epochs, it gradually stabilized afterward. Among all the models, PE-MLNet demonstrated the most outstanding performance. During training, the loss of PE-MLNet remained consistently lower than the other models, with an initial OA of 91% and a final OA of 97%. Additionally, the minimal fluctuation in OA further proved its excellent performance in change detection.

Furthermore, PE-MLNet also surpassed the other models in F1, precision, recall, and OA, achieving scores of 96.52%, 96.80%, 96.24%, and 98.01%, respectively (

Table 2. Quantitative comparison of change detection results among the different models. The best results are highlighted in bold.

Method	F1 (%)	Precision (%)	Recall (%)	OA (%)
FC-EF	87.68	86.17	89.41	93.29
FC-Siam-conc	88.15	86.59	89.95	93.55
FC-Siam-diff	87.00	83.84	91.38	93.28
IFN	90.13	89.72	90.91	94.85
ChangeNet	91.73	91.74	91.72	95.31
DTCDSCN	95.55	96.08	95.04	97.44
PE-MLNet	96.52	96.80	96.24	98.01

). From the visualization results (Figure 6), it can be observed that PE-MLNet is capable of extracting most of the changed areas while producing fewer false positives and negatives. For instance, in Figure 6 (S1), only PE-MLNet detected the specific change from salt fields to aquaculture ponds, whereas the other methods failed to fully identify this transformation. In Figure 6 (S2 and S3), the other methods mistakenly classified unchanged roads as changed areas (S2) and failed to detect the emergence of small buildings (S3). This could be due to the narrow pixel width occupied by roads and small buildings in the imagery, leading to the loss of information during feature extraction and resulting in false detections. The PEM in PE-MLNet is designed to capture and enhance change information at different scales in the image, suppress noise, and detect subtle changes. Simultaneously, the multi-task learning framework allows the model to focus more on semantic information during feature extraction, thereby improving the model robustness.

3.4. Semantic Segmentation

The loss of all the models rapidly decreased within the first 75 epochs and then gradually stabilized. Meanwhile, the OA of all the models steadily increased, with the highest OA reaching over 92% (Figure 7). PE-MLNet and UNet exhibited better performance than the other methods, achieving a maximum OA of over 94%. Considering more detailed evaluation metrics (

Table 3. Quantitative comparison of semantic segmentation results among the different models, with the best outcomes highlighted in bold.

Method	mIOU (%)	OA (%)	AA (%)	Kappa (%)
FCN	76.88	92.96	87.80	91.15
UNet	79.82	94.09	90.34	92.06
SegNet	76.77	92.94	87.43	91.17
HRNet	79.56	92.97	86.42	91.18
PE-MLNet	81.50	94.69	88.06	93.33

), the four evaluation metrics of PE-MLNet and UNet are notably higher than those of the other methods. It is worth noting that, except for a slightly lower AA compared to UNet, PE-MLNet achieved the highest levels in the other evaluation metrics, especially in the mIOU, which is improved by 4.62%, 1.68%, 4.73%, and 1.94% compared to the other methods, respectively. This proved the excellent performance of the model in semantic segmentation tasks.

To compare the segmentation effects of the different models on ground object identification, the Intersection over Union (IOU) for each category was calculated (

Table 4. Quantitative comparison of mean Intersection over Union (mIOU) across various categories for different models. The best results are highlighted in bold.

Method	Building	Natural Water	Cropland	Aquaculture Pond	Salt Field	Road	Oil Depot	Wetland	Others
FCN	69.47	86.05	95.37	88.95	84.62	42.46	55.23	85.87	83.88
UNet	69.88	88.63	93.19	90.22	88.06	50.14	65.39	90.62	82.25
SegNet	66.96	82.28	94.94	88.99	58.54	44.84	50.46	83.59	79.60
HRNet	70.65	83.14	95.93	93.01	83.99	49.75	63.04	87.97	89.58
PE-MLNet	72.94	89.67	96.58	94.58	89.37	50.55	63.65	89.43	86.70

). PE-MLNet also obtained the highest IOU in multiple categories. Combined with the visualization results (Figure 8), its prediction results are more accurate in identifying various features, such as buildings, natural water bodies, and aquaculture ponds. This demonstrates a stronger robustness and stability of PE-MLNet than the others. However, for the identification of roads and oil depots, none of the models achieve satisfactory results. This may be due to the limited sample size of these two categories in the training data, making it difficult for the models to fully learn their features.

3.5. Effects of PEM

To evaluate the significance of the PEM in change detection, we chose to replace the PEM with two methods: concatenating (PE-MLNet_conc) and differencing (PE-MLNet_diff) of the multi-layer feature maps obtained from the feature extractor. Although the PEM increases the complexity of the model, it does not significantly impact computational efficiency. In terms of evaluation metrics, while the improvement in the PEM in OA and recall is not remarkable, it significantly outperforms the other two approaches in the F1, IOU, and precision metrics (

Table 5. Quantitative comparison of change detection results using different feature fusion methods.

Method	FLOPS (G)	Params (M)	F1 (%)	OA (%)	IOU (%)	Precision (%)	Recall (%)
PE-MLNet_PEM	37.29	34.67	96.52	98.01	93.37	96.80	96.25
PE-MLNet_conc	66.93	19.94	95.50	97.47	91.54	95.08	95.93
PE-MLNet_diff	36.21	14.81	95.39	97.39	91.36	95.31	95.48

). This suggests that the PEM plays a crucial role in optimizing the details and integrity of prediction results. It aids the network to concentrate on more detailed information during the decoding process, thereby enhancing the overall network performance. Among the three methods, PE-MLNet_diff performed worst. This could be attributed to the high variability in high-resolution spectral imagery, where relying solely on a difference approach might lead to information loss in detection results.

By visualizing the Class Activation Maps (CAMs) for selected samples from the YRD dataset, as shown in Figure 9, it becomes evident that the PEM method adaptively learns weight-enhanced feature representations while accurately pinpointing the locations of changing regions. This evidence supports that the introduction of the PEM enhances the network ability to capture multi-scale detailed information, ultimately improving the precision and completeness of prediction results.

3.6. Human Activity Detection

By cropping high-resolution GF-6 images from 2019, 2021, and 2023 into a tile set of 256 × 256 pixel size and inputting them into the trained PE-MLNet model, land use changes and human activity intensity were evaluated. The detection results are shown in Figure 10. By comparing the land use types before and after the change area, different human activities can be obtained. The detailed display in Figure 11 shows that PE-MLNet can accurately identify various types of human activities, such as buildings, aquaculture ponds, roads, etc. The results of the land use transition diagrams uncovered that the areas of buildings, roads, and oil depots have increased, while the farmland area largely decreased over the five years, primarily transformed into aquaculture ponds and other land uses (Figure 12). Regarding the spatial distribution, human activities were most intense in the northeastern coastal areas, primarily reflected in the expansion of construction land and the conversion of salt fields into aquaculture ponds. In the southeast estuary region, two main types of changes were identified: The first is the degradation of wetlands, which is primarily a result of recent artificial management efforts to control the invasion of Spartina alterniflora in the area. The second is the extension of natural water bodies, possibly influenced by increased runoff and more precipitation than normal [49].

4. Conclusions

Rapid and intelligent detection of human activities is crucial for regional urban planning and ecological protection. High-resolution imagery provides a more detailed data source for detecting human activities. To break the limitation of most current methods only suitable for detecting specific types of human activities, this study constructed a multi-task learning-based human activity detection model named PE-MLNet to simultaneously conduct change detection and semantic segmentation in bitemporal images. By introducing the PEM to capture multi-scale change details and contextual information from bitemporal images, PE-MLNet largely outperformed other classical change detection and semantic segmentation approaches in identification accuracy and visualization details, with an average increase of 7% in the F1 score for change detection and an average improvement of 4% in the mIOU for semantic segmentation. Utilization data from the Yellow River Delta uncovered that the buildings, roads, and oil depots have obviously expanded, while the farmland area largely decreased over the five years, primarily transformed into aquaculture ponds and other land uses. In future work, we will strive to further deepen and refine the classification system for semantic segmentation, aiming for a more granular categorization to adapt to more diverse application scenarios and needs.