Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review

Abstract

Video capsule endoscopy (VCE) is used widely nowadays for visualizing the gastrointestinal (GI) tract. Capsule endoscopy exams are prescribed usually as an additional monitoring mechanism and can help in identifying polyps, bleeding, etc. To analyze the large scale video data produced by VCE exams, automatic image processing, computer vision, and learning algorithms are required. Recently, automatic polyp detection algorithms have been proposed with various degrees of success. Though polyp detection in colonoscopy and other traditional endoscopy procedure based images is becoming a mature field, due to its unique imaging characteristics, detecting polyps automatically in VCE is a hard problem. We review different polyp detection approaches for VCE imagery and provide systematic analysis with challenges faced by standard image processing and computer vision methods.

1. Introduction

Video capsule endoscopy (VCE) is an innovative diagnostic imaging modality in gastroenterology, which acquires digital photographs of the gastrointestinal (GI) tract using a swallowable miniature camera device with LED flash lights [1,2]. The capsule transmits images of the gastrointestinal tract to a portable recording device. The captured images are then analyzed by gastroenterologists, who locate and detect abnormal features such as polyps, lesions, bleeding, etc. and carry out diagnostic assessments. A typical capsule exam consists of more than 50,000 images, during its operation time, which spans a duration of 8 to 10 h. Hence, examining each image sequence produced by VCE is an extremely time-consuming process. Clearly, an efficient and accurate automatic detection procedure would relieve the diagnosticians of the burden of analyzing a large number of images for each patient.

Detecting polyps from VCE imagery is one of the foremost problems in devising automated computer-aided detection and diagnosis systems. The polyps to be detected in the images are characterized by physicians according to human perception of their distinctive shapes, and also in some cases, by their color and texture in these geometric objects. In effect, according to medical information, the geometry of colonic polyps can be classified essentially into two types: pedunculate polyps, which are mushroom-like structures attached by a thin stalk to the colon mucosa, and sessile polyps, which are like caps (mushroom structures with no stalk). Their colors are generally red, reddish or rose, and their textures can be very similar to a human brain. Figure 1 shows example polyps across different regions of the gastrointestinal (GI) tract, illustrating the variability in shape, color, and texture.

Many previous studies have focused on detecting polyps on classical imaging techniques of colonoscopies (not capsule endoscopy images) [3,4,5,6,7,8,9,10]. Polyp detection approaches in colonoscopy imagery include using elliptical features [6], texture [3,11], color and position features [12,13] (see [14] for a review of polyp detection methods in colonoscopy images). Polyp detection schemes applicable to colonoscopy and Computed Tomography (CT) colonography use mainly geometry based techniques (see, for example, [15]). However, due to the different imaging modality in VCE, images have different characteristics and hence require unique methods for efficient polyp detection across various frames. Several shape based schemes that were proposed to find polyps in virtual colonoscopy or computed tomography colonography have been addressed (see, e.g., [15,16,17,18,19,20]). Most of these methods take the already reconstructed surface representing the colon’s interior or rely on some specific imaging techniques (see [21,22] for reviews). In contrast, VCE comes with an un-aided, uncontrolled photographic device, which moves automatically and is highly susceptible to illumination saturation due to near-field lighting [23]. Moreover, the images from VCE differs significantly from images obtained with the traditional colonoscopy. For example, the liquid material in the lumen section is less in colonoscopy and hence the images look more specular. However, in VCE images, the mucosa tissue looks diffusive under the presence of liquid and additionally the trash and turbidity can hinder the view of the mucosal surface [23]. Due to the unaided movement of the capsule camera, blurring effects make the image looks less sharper. Moreover, the color of mucosal tissue under VCE has some peculiar characteristics [24]. Due to these reasons particular to VCE, its sensitivity for detecting colonic lesions is low compared with the use of optical colonoscopy as noted in [25]. Nevertheless, a recent meta-analysis showed that capsule endoscopy is effective in detecting colorectal polyps [26] (at-least in the colon capsules, though the jury is still out on the small-bowel and esophagus). Newer advances in sensors, camera system results in second generation capsule endoscopes and sensitivity and specificity for detecting colorectal polyps was improved [27,28]. However, the increased imaging complexity and higher frame rates, though they provide more information, inevitably put more burden on the gastroenterologists. Thus, having efficient, robust automatic computer aided detection and segmentation of colorectal polyps is of great importance and needed now.

In this comprehensive survey paper, we provide an overview on different automatic image/video data based polyp detection (localization) and segmentation methods proposed in the literature so far (up to December 2016, and we refer the reader to the project website that is updated continuously with links to all the papers presented here to obtain more details about this research area [29]). and discuss the challenges that remain. We organized the rest of the paper as follows: Section 2 provides a review of polyp detection, segmentation and holistic techniques from the literature. In Section 3, we discuss the outlook in this field along with challenges that need to be tackled by future research.

2. Review of Polyp Detection and Segmentation in VCE

Variable lighting and rare occurrence of polyps in a given (full) VCE video creates immense difficulties in devising robust and data-driven methods for reliable detection and segmentation. We can classify polyp detection/segmentation methods into two categories: (a) polyp detection—finding where the polyp frame (polyp frames may contain more than one polyp. We do not make a distinction of detecting one or multiple polyps in a given image) occurs, not necessarily the location of the polyp within that frame; (b) polyp segmentation—once a frame that contains polyp(s) is given a segment mucosal area in which the polyp appears. Note that the first task is a much harder problem than the latter due to the large number of frames that the automatic algorithm needs to sift through in order to find the polyp frames, which is usually a rare occurrence. Naturally, machine learning based approaches are essential for both categories, especially for polyp detection since the occurrence of polyp frames, frames where at least one polyp is visible, are very few in contrast to the typical full length of video frames (typically greater than 50,000) in VCE imagery. The task is more complicated since colorectal polyps do not have common shape, texture, and color features even within a single patient’s video. Nevertheless, there have been efforts in identifying polyp frames using automatic data-driven algorithms. For polyp segmentation, it is relatively an easier problem since the automatic algorithms need only to analyze a given polyp frame to find and localize the polyps that are present. We next review polyp detection and segmentation methods studied so far in the literature and discuss the key techniques used with relevant results.

2.1. Polyp Detection in Capsule Endoscopy Videos

There are two main classes of polyps with respect to their appearance in shape: pedunculated and sessile. In Figure 2, we show some example polyps (selected from images shown in Figure 1) in 3D using the shape from shading techniques [30], indicating the amount of protrusion out of the mucosa surface (visualizations as 3D figures are available in the Supplementary Materials and also at the project website). Figure 3 shows a pedunculated polyp that appears in consecutive frames from a VCE exam of a patient. The appearance changes drastically (rotation, translation, and scale changes), and the polyp that is not so clear in the first few frames (Figure 3 (top row)) then becomes completely visible (Figure 3 (middle row)) and then becomes smaller before vanishing from view. In the last few frames (Figure 3 (bottom row)), the stalk of the polyp is not visible, thus appearing like a sessile polyp attached to the mucosal fold at the bottom. In addition, please note that the texture feature (due to vascularization) on top of the polyp is distinctively different from the surrounding mucosal folds that do contain small scale textures.

One of the earliest works in VCE image processing, and especially in detecting polyps automatically, is by Kodogiannis et al. [31], who studied an adaptive neuro-fuzzy approach. By utilizing texture spectrum from six channels (red-green-blue (RGB) and hue-saturation-value (HSV) color spaces) with an adaptive fuzzy logic system based classifier, they obtained $97\%$ sensitivity in 140 images with 70 polyp frames.

Li et al. [32] compared two different shape features to discriminate polyps from normal regions. They utilized an MPEG-7 shape descriptor (angular radial transform—ART), and Zernike moments as features along with a multi-layer perceptron (MLP) neural network as the classifier. Due to invariance to rotation, translation, and scale change, Zernike moments are well-suited for polyp detection in VCE, which has unconstrained movement of the camera. They tested their approach on 300 representative images, out of which 150 contained polyps, and achieved an accuracy of $86.1\%$. However, their approach only compared two specific shape features and discarded color and texture information entirely. In a related work, Li et al. [33] exploited combined color and shape features for polyp detection with HSI (hue, saturation, intensity) color space. They used compressed 2D chromaticity histogram (from hue and saturation), and Zernike moments (from intensity) and tested MLP and support vector machines (SVM) classifiers. In 300 representative images, out of which 150 contain polyps, a 2D chromaticity histogram plus 5th order Zernike moments with MLP obtained $94.20\%$ accuracy when compared to color wavelet covariance feature based results of $70.50\%$ with SVM. The chromaticity histogram is compressed with discrete cosine transform and lower frequency coefficients. This quantization scheme affects the performance with respect to color discriminability.

Karargyris and Bourbakis [34] performed Log–Gabor filter based segmentation along with a smallest univalue segment assimilating nucleus (SUSAN) edge detector [35], curvature clusters, and active contour segmentation to identify polyp candidates. On a 50-frame video containing 10 polyp frames, they achieved a sensitivity of $100\%$ (all 10 frames were selected as polyp candidates by the scheme). The method relies heavily on a geometric rule that assumes that the polyp region is an ellipse close to a circle, which limits its applicability to detecting polyps of different shapes. Moreover, due to the small test cases of polyps, it is not clear how the proposed approach performs in a full VCE exam where the same polyp can have different geometric boundaries across frames. Karargyris and Bourbakis [36] later extended their previous work [34] by adding an SVM classifier. Hwang and Celebi [37] used watershed segmentation with initial markers selected using Gabor texture features and K-means clustering that avoided the requirement of accurate edge detection considered in [34]. On a set of 128 images with 64 polyp frames, this method achieved a sensitivity of $100\%$. However, a similar assumption as [34] about the elliptical or circular shape was made and curvature clusters were used as an indicator for polyp candidates.

Nawarathna et al. [38] considered texton histograms for identifying abnormal regions with different classifiers such as SVM and K-nearest neighbors (K-NN). With Schmid filter bank based textons and SVM classifier, an accuracy of $95.27\%$ was obtained for polyp detection. Nawarathna et al. [39] later extended this approach with the addition of an local binary patterns (LBP) feature. In addition, a bigger filter bank (Leung–Malik), which includes Gaussian filters, were advocated for capturing texture more effectively. Note that these approaches rely only on texture features and do not include any color or geometrical features. The best results of $92\%$ accuracy was obtained for the Leung–Malik-LBP filter bank with K-NN classifier for 400 images with 25 polyps.

Figueiredo et al. [40] used a protrusion measure based on mean and Gaussian curvature for detecting polyps. They defined a novel protrusion measure which detected $80\%$ polyp frames accurately with localization of polyps. Unfortunately, not all polyps are protrusions (see Figure 2b for examples), and the curvature index derived in [40] fails to pick up VCE frames which contain sessile or flat polyps. Please also note that this method does not rely on any classification and is purely a geometry based approach.

Zhao and Meng [41] proposed using opponent color moments with LBP texture features computed over contourlet transformed images with an SVM classifier, and reported $97\%$ accuracy. Their work unfortunately did not mention how many total frames and polyp frames were used or how the color and texture features were fused.

Zhao et al. [42] studied a supervised classification approach with a hidden Markov model (HMM) by integrating temporal information for polyp detection in VCE videos. They utilized a combination of color, edge and texture features that resulted in a 118-D feature vector that was reduced to 13-D via a Laplacian eigenmap method along with a K-NN classifier. Their approach was tested on 400 images with 200 polyp frames and obtained an accuracy of $83.3\%$. However, the true strength of their method lied in testing image sequences, and, for this purpose, they utilized a second dataset of 1120 images (224 videos of 5-frame duration), out of which there were 560 polyp frames. The obtained accuracy of $91.7\%$ for image sequences of 5-frame length indicates a strong potential for utilizing temporal information. In a related work, Zhao et al. [43] used the same features along with a boosted classifier and conducted evaluations using 1200 VCE images with $90\%$ classification accuracy.

Hwang [44] used a bag of visual words model from computer vision literature by treating polyp regions as positive documents and normal regions as the negative documents. The author used speeded up robust features (SURF) and quantized them with K-means clustering to generate the codebook to represent images as histograms of visual words. These feature vectors were then fed into an SVM classifier, and the results showed $90\%$ sensitivity in a total of 120 images with 60 polyp frames. A very similar approach was undertaken in [45] with color features from HSI space added in addition. Tests were conducted on 250 images with 50 polyp frames and $66\%$ sensitivity was obtained.

Li and Meng [46] applied the uniform LBP on discrete wavelet transformed sub-images to capture texture features on polyps. On a dataset of 1200 images with 600 polyp frames, an accuracy of $91.6\%$ was obtained with uniform LBP on 24 circular set members and radii of 3 with an SVM classifier. The method is based only on texture features and can fail to detect flat polyps with little texture. In addition, there was no mention of the dimension of the feature set based on uniform LBPs, which is usually high.

Condessa and Bioucas-Dias [47] studied an extension of the protrusion measure in [40] by utilizing a two-stage approach. The first stage involved multichannel segmentation, local polynomial approximation, and the second stage extracted contours and curvature features with an SVM classifier. They obtained $92.31\%$ sensitivity. The authors advocated using the temporal information via recursive stochastic filtering, though this has not yet been considered by other researchers.

David et al. [48] utilized the protrusion measure from [40]. However, they observed that not all polyps exhibit high curvature values. By augmenting these high curvature peak based locations with color and illumination segmentation, they applied a training stage with histograms of oriented Gaussians (HOG) features (computed on those locations) with an MLP classifier. Results on $30,540$ with 540 polyp frames indicated that an accuracy of $80\%$ was achieved with this approach.

Figueiredo et al. [49] combined the protrusion measure [40] along with multiscale and uniform LBP texture features. They obtained an accuracy of $98.50\%$ in a Commission internationale de l’éclairage (CIE) Lab color space with monogenic LBP and a linear SVM classifier in 400 images with 200 polyp frames. The protrusion measure alone achieved $65.5\%$ accuracy, indicating that the reliance on polyp protrusion only is not a good assumption. Furthermore, with an addition of basic LBP features, the accuracy increased to $97.25\%$, indicating the importance of texture.

Yuan and Meng [50] used scale invariant feature transform (SIFT) feature vectors with K-means clustering for bag of features representation of polyps. By integrating histograms in both saliency and non-saliency regions, the authors calculated weighted histograms of the visual words. These were fed into an SVM classifier and experiments on 872 images with 436 polyp frames showed that $92\%$ detection accuracy was obtained. In a related work, Yuan and Meng [51] tested Gabor filter and monogenic LBP features with an SVM classifier. Their results indicated an accuracy of $91.43\%$ on the same dataset. Yuan et al. [52] later extended these approaches with the addition of LBP, uniform LBP, complete LBP, and HOG features for capturing texture information along with SIFT features. They tested with SVM and Fisher’s linear discriminant analysis (FLDA) classifiers with different combinations of local features. Their method with SIFT + complete LBP with an SVM classifier achieved top classification accuracy of $93.2\%$ on a set of 2500 images that contained 500 polyp frames.

Mamonov et al. [53] proposed a system based on sphere fitting for VCE frames. The pipeline consisted of considering the gray-scale images and applying a cartoon + texture decomposition. The texture part of the given frame is enhanced with nonlinear convolution and then mid pass filtered to obtain binary segments of possible polyp regions. Then, a binary classifier uses a best-fit ball radius as an important decision parameter to decide whether there is a polyp present in the frame or not. This decision parameter is based on the assumption that the polyps are characterized as protrusions that are mostly round in shape, hence polyps which violate this do not get detected. The method tested on $18,968$ frames with 230 polyp frames obtained $81.25\%$ sensitivity per polyp. Here, per polyp basis was computed as correct detection of at least one polyp frame in the corresponding sequence by the automatic algorithm.

Jia et al. [54] used geometrical features via ellipse fitting (area, ratio of major and minor axis), and multiscale rotation invariant LBP, and HOG texture features. With support vector regression, they reported a $64.8\%$ true positive rate on a total number of $27,984$ frames with $12,984$ polyp frames. The huge number of polyp frames were obtained by perturbing original 541 polyp frames to obtain $12,443$ extra samples for training.

Zhou et al. [55] utilized RGB averaging with variance for polyp localization and radius measurement of suspected protruding polyps using a statistical region merging approach. On a total of 359, this approach with an SVM classifier obtained $75\%$ sensitivity. Gueye et al. [56] used SIFT features and a Bag of Features (BoF) method. On 800 frames with 400 polyp frames, their SVM classifier obtained a classification rate $61.83\%$, and on 400 frames with 200 polyp frames, the rate increased to $98.25\%$.

Yuan and Meng [57] recently proposed a saliency driven model for polyp recognition. They used a global and local saliency coding method to encode with SIFT features computed in patches. Overall, their proposed method with an SVM classifier obtained $92.51\%$ sensitivity, $89.81\%$ specificity and $91.13\%$ accuracy in 1080 frames with 540 polyp frames. Yuan et al. [58] later extended this by adding local constrained linear coding for detecting not only polyps, but also bleeding and ulcers. They applied SIFT on the HSV color space and a feature coding based on saliency and an adaptive locality constrained linear approach. With an SVM classifier with max pooling on 17 polyp videos (containing 500 polyp frames), $83.50\%$ accuracy was reported.

In summary, there have been a small number of automatic polyp detection methods, though, unfortunately, the full dataset description is lacking in many of these published works, which makes it hard for us to benchmark them.

Table 1. Overview of automatic polyp detection methods in video capsule endoscopy (VCE) with salient techniques, classifiers, and total number of frames in VCE videos with polyp frames. Abbreviations: ART—angular radial transform, BoW—bag of words, BoF—bag of features, HMM—hidden Markov model, SIFT—scale invariant feature transform, SURF—speeded up robust features, HoG—histogram of oriented gradients, LLE—locally linear embedding, LBP—local binary patterns, K-NN—k-nearest neighbors, MLP—multilayer perceptron, SVM—support vector machines, LDA—linear discriminant analysis, ×—no classifier used.

Reference	Features/Technique	Classifier(s)	Total Number (Polyps)
[31]	Texture spectrum from RGB, HSV	Neurofuzzy	140 (70)
[32]	ART descriptor + Zernike moments	MLP	300 (150)
[33]	Chromaticity histogram + Zernike moments	MLP, SVM	300 (150)
[34]	Log–Gabor filter + SUSAN edge detector	×	50 (10)
[37]	Gabor filters + Watershed segmentation	×	128 (64)
[38]	Filter banks based texton histogram	K-NN, SVM	400 (25)
[40]	Protrusion measure via curvatures	×	1700 (10)
[36]	Log–Gabor filter + SUSAN edge detector	SVM	50 (10)
[41]	Opponent color moments + LBP + LLE	SVM	2 videos
[42]	Color + edge + texture + HMM	weak k-NN	400 (200), 1120 (560)
[44]	SURF features + BoW + K-means	SVM	120 (60)
[45]	Color + Gabor filters + BoW + K-means	SVM	250 (50)
[43]	Color + edge + texture + HMM	Boosted SVM	1200 (600)
[46]	Uniform LBP + wavelet transform	SVM	1200 (600)
[47]	Local polynomial approximation + geometry	SVM	3 videos (40)
[48]	Geometry + color + HoG	MLP	30,540 (540)
[49]	Geometry + color + Monogenic LBP	SVM	400 (200)
[50]	SIFT + Saliency + BoF	SVM	872 (436)
[51]	Gabor filter + Monogenic LBP + LDA	SVM	872 (436)
[53]	Texture + midpass filtering + ellipse fitting	Binary	18,968 (230)
[39]	Texton histogram + LBP	K-NN	400 (25)
[54]	Geometry + LBP + HoG	Regression	27,984 (12,984)
[55]	RGB + Variance + radius	SVM	359
[56]	SIFT + BoF + K-means	SVM	800 (400)
[52]	SIFT + complete LBP + BoF	SVM	2500 (500)
[57]	SIFT + saliency coding	SVM	1080 (540)
[58]	SIFT + saliency coding	SVM	17 videos (500)

summarizes all of the polyp detection methods covered so far with main techniques, classifiers utilized along with tested dataset details (whenever available). It can be seen that the popular classifier is linear SVM with radial basis function as kernel due to its simplicity and ease of use. However, it is our belief that the majority of these methods either overfit or underfit, as the proposed methods are tuned to obtain best possible detection accuracy results for their corresponding datasets. All of the aforementioned methods highlight the accuracy of polyp detection based on how many polyp frames are detected out of all the input frames given. However, as mentioned by Mamonov et al. [53], per polyp accuracy over per frame basis is very important. This is because detecting a minimum of a single polyp frame in (typically) consecutive frames based sequences will suffice to alert the gastroenterologist/clinician. Figure 3 shows different frames of a single patient VCE exam wherein a sequence of length 55 contains a pedunculated polyp (original video available in the Supplementary Materials section). It is clear that detecting any one of the frames as a polyp frame is enough, as the gastroenterologist can inspect the neighboring frames manually. This prospective automatic polyp alert system can reduce the burden on gastroenterologists, as the number of frames to be inspected can be dramatically reduced, from tens of thousands of frames to a few hundred possible polyp sequences.

2.2. Polyp Localization or Segmentation within a VCE Frame

Polyp localization or segmentation of polyps in a single frame of VCE is an (relatively easier) object identification problem. The general task falls under the category of image segmentation, which is a well-studied problem in various biomedical imaging domains. As we have seen before, color, texture, and shape features individually are not discriminative enough to obtain reliable polyp segmentations from VCE frames. There have been a number of efforts in polyp segmentation from VCE which we classify based on which segmentation techniques are utilized. Note that the majority of the previously discussed polyp detection algorithms employ a polyp segmentation step to identify candidate polyp frames, although accurately segmenting the polyp region is not required for subsequent polyp detection in VCE.

• Localization

  • Filtering: Karargyris and Bourbakis [34] performed Log–Gabor filter based segmentation. Prasath and Kawanaka [59] proposed using a novel vascularization feature, which is based on Frangi vesselness filter.
  • Geometry: Hwang and Celebi [37] derive a localization approach using curvature center ratio on K-means clustering based segmented polyp frames. Figueiredo et al. [40] used a protrusion measure based on Gaussian and mean curvature to highlight possible polyp locations in a given VCE image.
  • Hybrid: Jia [60] used K-means clustering and localizing region-based active contour segmentation. The results shown in this paper seems to be from colonoscopy imagery and not VCE.
• Accurate boundaries, segmentation

  • Active contours: Meziou et al. [61] used alpha divergence based active contours segmentation, though their approach is a general segmentation, and not for polyps in particular (see also [62]). Prasath et al. [30] used an active contours without edges method for identifying mucosal surface in conjunction with shape from shading technique (see also [65]). Eskandari et al. [63] used a region based active contour model to segment the polyp region from a given image that contain a polyp (see also [64]).

Figure 4 highlights some example images and results obtained with the aforementioned automatic polyp localization and segmentation methods. Note that, in localization approaches, though polyp bounding boxes or circles were obtained, no crisp boundary segmentations are available. Obtaining accurate boundaries and segmentations of polyps in a frame requires a robust method that can avoid the pitfalls of bubbles, trash, and illumination artifacts.

Table 2. Methods available for polyp localization and segmentation within a VCE frame.

Reference	Technique
[34]	Log–Gabor filter
[59]	Vascularization + Frangi vesselness
[37]	Curvature center ratio + K-means clustering
[40]	Protrusion measure via curvatures
[60]	K-means clustering + localizing active contour
[61]	Alpha divergence based active contours
[62]	Alpha divergence based active contours
[30]	Active contours + Shape from shading
[63]	Active contours
[64]	Active contours

summarizes the methods available for polyp localization and segmentation so far. A preliminary mucosa segmentation [30,65,66,67] may be required before applying the polyp segmentation step in order to avoid these non-polyp pixels.

2.3. Holistic Systems

Romain et al. [68] studied a preliminary multimodal VCE for detecting polyps by utilizing active stereo vision within the capsule itself. It is based on two steps: (1) 2D identification polyp regions of interest (ROI) using simple geometric shape features; and (2) classification of polyps using 3D parameters computed on ROI using active stereo vision. Although the authors of [68] perform a prototype made of silicone for in vitro learning, the experiments were limited by the fact that there are no liquids (GI juices), trash, or a distinct lack of distention of the colon walls, which affect the performance of automatic image analysis methods in real VCE exams. Similar efforts have been reported in [69,70].

Table 3. Holistic polyp detection and segmentation approaches for endoscopy systems. Note that all of these proposed methodologies are so far only tested with traditional colonoscopy images [8].

Reference	Technique	Classifier(s)	Total Number (Polyps)
[68]	Geometry + Texture	Boosting	1000 (200)
[69]	circular Hough + co-occurrence matrix	Boosting	1500 (300)
[70]	Hough transform + co-occurrence matrix	Boosting	1500 (300)

summarizes these holistic system approaches proposed so far. Note that the polyp detection testing is done using traditional colonoscopy images for now. Apart from these technological constraints, these approaches require robust embeddable computer vision systems that need to operate under a strict energy budget (battery constraints). However, a computer vision embedded VCE system can revolutionize the diagnosis procedures that have been done manually through tedious processes so far.

3. Discussion and Outlook

Automatic polyp detection and segmentation is a nascent area which requires various computer vision methodologies such as geometrical primitives, color spaces, texture descriptors, features matching, along with strong machine learning components. A standard assumption made in classical colonoscopy imagery based polyp detection methods is that polyps have high geometrical features such as well-defined shape [20] or protrusion out of mucosal surfaces [15]. Thus, curvature measures are widely utilized in detecting polyps and the adaptation to VCE imagery is undertaken with limited success [40,47,48,49]. Texture or color features based schemes such as [11,12] applied to colonoscopy images do not work well in WCE when used on their own, as we have seen in Section 2.1. This is because, though the polyps appear to have different textural characteristics than the surrounding mucosa, bubble, trash and capsule motion induced by uncontrollable peristalsis have a strong effect on texture features. In such a scenario, classifying with texture features alone is highly error prone. For example, the polyp shown in Figure 3 (top row) has a unique texture pattern, whereas for the polyp in Figure 3 (middle row), texture appears to be uniformly spread across the mucosal structure surrounding it. Moreover, in a video, the neighboring frames show a partial view of the polyp with mucosa folds having a similar texture. Similarly, the color of the polyp is not homogeneous across different polyp frames within a patient and highly variable across different exams from patients. In a comprehensive learning framework, one can try to incorporate local and global texture features for discerning polyps along with vascularization, and color information. Existing approaches reviewed in Section 2 are plagued by various factors. One of the main problems is the existence of trash and bubbles since no colon cleaning is required in VCE exams. Robust polyp detection approaches must combine efficient trash, and bubble detectors to avoid false positives. To conclude, we believe, a holistic view of combining, motion, geometry, color, and texture with a strong machine learning paradigm may prove to be successful for robust, efficient automatic polyp detection in VCE imagery.

Based on the detailed description of the current state-of-the-art polyp detection and segmentation methods, we observe the following salient points for future outlook:

• Recent excitement generated by deep learning is a very promising direction where massively trained neural network based classifiers can be used to better differentiate polyp frames from normal frames. However, deep learning networks in general require a huge amount of training data, in particular labeled data of positive (polyp frame) and negative (normal frame) samples. One possible remedy for an imbalanced data problem is to use data augmentation, therein one can increase the polyp frames by artificial perturbation (rotation, reflection, ...) (see e.g., [54], for an attempt to create a higher number of polyp frames for training). There have been some works in the last two years on endoscopy image analysis with deep learning [71,72,73]. Other approaches include using deep sparse feature selection (see, e.g., [74]).
• Similar to the Arizona State University (ASU)-Mayo Clinic polyp database for colonoscopy polyp detection benchmarking, VCE polyp detection requires a well-defined database with multiple expert gastroenterologists marked polyp regions. This will make the benchmarking and testing of different methodologies for automatic polyp detection and segmentation standardized.
• Sensor improvements with novel capsule systems [75,76], such as more control in terms of higher image resolution, standardized illumination/contrast, controlled capsule speed, and variable image capturing mechanisms, can help automatic image analysis.
• Finally, embedding the image analysis part within capsule endoscopy imaging systems [70,77] is an exciting research area which will enable the gastroenterologists to make real-time decisions. However, there are a lot of challenges that remain for essential progress [78].