Residual objectness for imbalance reduction

As most object detectors rely on dense candidate samples to cover objects, they have always suffered from the extreme imbalance between very few foreground samples and numerous background samples during training, i.e., the foreground-background imbalance. Although several resampling and reweighting schemes (e.g., OHEM, Focal Loss, GHM) have been proposed to alleviate the imbalance, they are usually heuristic with multiple hyper-parameters, which is difficult to generalize on different object detectors and datasets. In this paper, we propose a novel Residual Objectness (ResObj) mechanism that adaptively learns how to address the foreground-background imbalance problem in object detection. Specifically, we first formulate the imbalance problems on all object classes as an imbalance problem on an “objectness” class. Then, we design multiple cascaded objectness estimators with residual connections for that objectness class to progressively distinguish the foreground samples from background samples. With our residual objectness mechanism, object detectors can learn how to address the foreground-background problem in an end-to-end way, rather than rely on hand-crafted resampling or reweighting schemes. Extensive experiments on the COCO benchmark demonstrate the effectiveness and compatibility of our method for various object detectors: the RetinaNet-ResObj, YOLOv3-ResObj and FasterRCNN-ResObj achieve relative 3 % ∼ 4 % Average Precision (AP) improvements compared with their vanilla models, respectively.