Nowadays with the growing popularity of vehicles, traffic accidents occur more frequently, causing lots of casualties. In this paper, in order to avoid the accident where a vehicle collides with the one ahead, we present a novel vehicle brake behavior detection method by using a colorful camera or mobile device fixed on the windshield of the test car which utilized to capture the front vehicle information. The brake behavior detection in our work includes two procedures, brake lights region detection and brake behavior decision. For the first procedure, we use threshold segmentation and proposed horizontal–vertical peak intersection strategy to filter and generate the credible rear-light regions of the front vehicle in the YCrCb color space converted from the original RGB color space. For the second procedure, the sophisticated SVM classifier is trained to detect the brake behavior of the front vehicle. In this procedure, we extract discriminative features of the rear-light regions generated from the first procedure and then the features are used as the input of the pre-trained classifier. Extensive experiments on various real-world vehicle datasets demonstrate the effectiveness and real-time performance of our proposed brake detection strategy.