Anomaly detection is an important component of computer security defense. As the security situation becomes increasingly severe, abnormality is detected by the sequential pattern mining has become a hot research topic. For normal behavior to identify in anomaly detection, proposed software behavior pattern recognition technology which based on improved prefixspan algorithms and sequence similarity comparison, mainly for software behavior java program under windows platform identification. The method uses sequential pattern mining to model the software behavior of the sample program, Extraction behavior of the software in the training data based on system call sequence support, and established pattern library to represent the outline of a software behavior. In the detection phase, compare the program library in the normal sequence mode with the current pattern to be detected and calculate sequence similarity, identify the target behavior. In this paper, the method has good performance in terms of software behavior recognition. Finally, the software behavior in network behavior, given the specific experimental procedures, demonstrate the effectiveness of this method.