Topological-Preserving Membrane Skeleton Segmentation in Multiplex Immunofluorescence Imaging

Objective.

The target annotation of this work is simplified as a skeleton to maximize topology preservation of all membranes. Instead of segmenting the whole merged membrane highlighted area that is more than 1-pixel width, the proposed model learns skeletons of cell membranes for downstream analytics. Figure 3 outlines the membrane segmentation workflow, which includes gold standard annotation, baseline, and proposed approaches.

Dataset and data pre-processing.

Six sample biopsies were collected from 2 CD patients and 1 healthy control from the AC and TI with IRB approval (Vanderbilt IRB #191738 and #191777). All data were de-identified under Institutional Review Board approval. The MxIF markers were acquired with 20× magnification and stained in the following order – DAPI (first round), MUC2, β-catenin 04, CD4, CD3D, HLA-A, CD8, NaKATPase, Vimentin, ERBB2, and CD45. The standard DAPI-based registration and autofluorescence correction were applied ¹⁹. A semi-automatic data quality check was to ensure marker alignment efficiency ²⁰. We computed the tissue masks that covered the tissue pixels across all staining rounds. The masks were applied to the images and preprocessed with group-wise linear normalization. To prepare membrane annotation data, we split each marker image into 128×128 patches (86 μ𝑚). Except for DAPI and MUC2, the rest of the markers were z-stacked and generated max/mean projection images. Finally, 80 regions of interest (ROIs) were selected for further annotation and learning.

Manual annotation.

A senior digital pathology researcher manually contoured all membrane skeletons of ROIs using MITK Workbench v2022.04 ²¹. As shown in Figure 3, the membrane annotation was first traced mainly at the z-stack max/mean projection images, then iteratively optimized via the nine individual channels, especially for the hard cases and weak signal regions. In summary, it took about 20 minutes to trace one image patch.

Membrane skeleton segmentation.

The baseline membrane segmentation model is implemented by an ilastik random forest pixel classification approach ²². Eight RGB patches (R: z-stack max projection image, G: MUC2, B: DAPI) are interactively traced over the training output from the ilastik feature selection intermediate results by a domain expert in cell and molecular biology. The baseline ilastik model would produce a membrane probability mask per input, following skeletonized morphological operation ²³.

In recent years, deep learning has rapidly become the SOTA basis by showing enhanced performance on 2D and 3D semantic segmentation tasks in various medical image analysis applications ²⁴. To train a deep learning-based membrane segmentation framework, we propose to investigate nine input options by employing five different sets of inputs: (i) nine individual and separate membrane markers, (ii) z-stack max projection image, (iii) z-stack mean projection image, (iv) ilastik probability map (intermediate output from the baseline, referenced from ¹⁴), and (v) DAPI & MUC2. We utilized four SOTA deep neural networks to generate 2D membrane skeletons: U-Net ²⁵, U-Net++ ²⁶, Attention U-Net ²⁷, and SegNet ²⁸. We also evaluated four different loss functions to perform gradient descent: binary cross-entropy (BCE) Loss ²⁹, Dice Loss ³⁰, SoftDice Loss (Dice Loss with Laplacian smoothing) ³¹, and clDice Loss (combining SoftDice with a topology-preserving Loss function specific for tubular structure segmentation) ³¹. Similar to the baseline, the final membrane from the proposed deep learning framework was converted to a skeleton by skeletonizing network output as shown in Figure 3.

Evaluation metric.

We use the adjusted rand index (ARI) to validate the segmentation topology performance to measure the similarity between two cell segmentation clusters. The clusters can be generated from a filled membrane skeleton followed by indexing each closed contour. In addition to ARI, to validate the skeleton’s volumetric performance, the target and predicted skeletons may have multiple paths to segment a broad membrane region without necessarily being fully overlayed but sharing a similar topology. Thus, the regular Dice or clDice metrics ³¹ might not suit for evaluating the volumetric overlapping for such scenarios. We extend the clDice and propose a new metric, $clDic{e}_{SKEL}$, to estimate the volumetric similarity of a ground truth skeleton y and the predicted skeleton p while maintaining the topology (an illustrative example is shown at the bottom of Figure 3), as defined in the equation (1):

where D is a dilation ³² function that dilate the input skeleton. 𝑇_p interprets the topology precision, and 𝑇_s determines the topology sensitivity as defined in equation (2):

Experiment design.

Two primary validation purposes were designed: (1) to investigate the best deep neural network and loss function pair, we trained models with different combinations over three deep learning frameworks (U-Net++, Attention U-Net, and SegNet) and four loss functions. We only implemented U-Net with the SoftDice Loss case. All models used the same input channel option (11 channels that contained z-stack max, z-stack mean, and nine separate membrane images) to reduce the complexity of the experimental configuration. And (2) to validate the sensitivity of model input channels (in total, nine options as described in Figure 3), we tested U-Net++, Attention U-Net, and SegNet with SoftDice Loss equipped only.

All deep neural networks were implemented in PyTorch 1.8 and Python 3.8. The networks followed standard architectures, which can be downloaded from a public GitHub repository (https://github.com/Andy-zhujunwen/UNET-ZOO). The models were trained on a high-performance computing datacenter with GPU nodes equipped with NVIDIA Titan Xp 12GB graphic cards. The training batch size of U-Net, U-Net++, and Attention UNet is 32, while the batch Size is set as 4 for SegNet to saturate the GPU. We empirically set 25 iterations and α = 0.01 (the ratio of SoftDice loss and topology-preserving Loss) for clDice Loss. We conducted a 5-fold cross-validation for each model with static training/validation split per fold. No duplicate validation data were allowed across folds. Each network was trained by 100 epochs and the highest $clDic{e}_{SKEL}$ score on validation data decided the best model to save. A 3×3 isotropic structuring kernel dilates each pixel in function D of $clDic{e}_{SKEL}$ metric.

Exploring the best deep learning framework.

The quantitate results on 5-fold cross validation across the ilastik baseline and 13 proposed deep learning works are shown in Figure 4. In summary, for volumetric performance, the SegNet with SoftDice Loss has the best $clDic{e}_{SKEL}$ score (mean ± std: 0.757±0.068) that outperforms the baseline by 20.2%. For topology performance, Attention U-Net with SoftDice Loss (mean ± std: 0.309 ± 0.143) outperforms the baseline by 41.3%. The Wilcoxon sign rank test was employed as the statistical model, where the significant difference (p-value < 0.05) is marked. Figure 5 demonstrates the qualitative results that include good/average/bad $clDic{e}_{SKEL}$ across all models. The relevant $clDic{e}_{SKEL}$ score is marked for reference.

Sensitivity analysis on different input channel.

Table 1 demonstrates the different deep learning frameworks with training by different input channel options when they are all using the SoftDice Loss. The Wilcoxon sign rank test was computed for each method. Table 1 shows that two of the input options (6 & 7, where input channels integrate z-stack max and mean projection image with/without individual markers) have better performance on average with a significant difference (p-value < 0.05).