KR102532019B1 - Method for high speed video object tracking based on snake and optical flow, recording medium and device for performing the method - Google Patents

Abstract

An ultra-high-speed moving picture object tracking method based on snake and optical flow includes the steps of calculating an initial snake point by setting the outline of an object of interest from a high-speed captured moving picture; generating feature points by performing an optical flow between the n-th frame and the n+1-th frame of the video; selecting an object region to be searched for by utilizing a previous background separation image and a difference image between the n-th frame and the n+1-th frame; adjusting positional information errors of object outlines in the n+1 th frame by comparing positions between the previous background separation image and the object region generated from the difference image and the n+1 th optical flow feature point; and extracting the outline of the object by converting the error-adjusted feature points into snake points of the n+1th frame. Accordingly, object contour extraction efficiency may be increased by increasing contour extraction accuracy.

Description

High-speed video object tracking method based on snake and optical flow, recording medium and device for performing the same

The present invention relates to a method for tracking a high-speed video object based on a snake and an optical flow, a recording medium and an apparatus for performing the same, and more particularly, to segment a moving object in a video captured by a high-speed camera using a snake model and an optical flow and the technique of tracing the outline.

Image segmentation is a technique of analyzing, classifying, and applying an object of interest in an image by separating it from a region determined as a background. This is the field of recognizing objects by reducing the amount of data required to interpret an image or matching the features of the image. The field of video object segmentation has emerged by recognizing an object of interest in a video and segmenting the area as video segmentation has been expanded.

The object tracking system in video has been widely applied in motion capture, security monitoring systems, etc. Currently, with the development of video recording technology, it is possible to capture high-resolution images without noise in the captured images, and object tracking methods have been developed. It has also been studied in many ways as better methods.

The field of recognizing and tracking objects in these videos has been based on the use of normal speed videos, but as the development and utilization of ultra-high-speed cameras increases, related studies are also being actively conducted in the fields of sports, military, and scientific experiments. . In particular, at a time when the mechanical limitations of ultra-high-speed cameras are disappearing to the extent that ultra-high-speed shooting functions are built into smartphones, the expansion of the field is growing more recently.

Until recently, many tracking algorithms have been studied, and they can be largely divided into four categories. There are 1) shape information object tracking method 2) region-based object tracking method 3) adaptive background generation method 4) background subtraction method.

Among them, the method using shape information is a method of tracking and analyzing after extracting the shape of an object, and can match and extract the outline of an object even in an incomplete or noisy image and express the object with fewer pixels. In addition, there is an advantage in that it can solve the occlusion phenomenon of the object.

However, it is difficult to predict the movement information or change of an unstructured object, although its advantages are recognized for structured objects. Similarly, it is difficult to predict in the case of complex shapes such as accurate contour extraction of a moving object or human motion. In addition, it is necessary to create a training set of objects of interest, and it is a method with many limitations in tracking objects in a video at high speed.

As a result, a segmentation and tracking system using an atypical model or a deformable model requires a function that is robust to object deformation and can overcome occlusion. This shows good results even in area-based methods using templates or color distribution, but templates or color distribution methods require complicated preprocessing.

The shape-based method learns object information in advance to obtain a learning set and projects the learning set to the position closest to the contour of the object. Based on this method, there are a condensation algorithm, an active shape model (ASM), and a contour-based method.

ASM is a representative method for efficient tracking of unstructured objects. However, in ASM, the principal component analysis (PCA) value varies according to the reference point of the initial training set. This has the disadvantage of causing mismatched object shapes and also has the disadvantage of requiring a lot of calculation time.

Active contour model (ACM), which has emerged as another method and is widely used in image segmentation, is a method of finding the contour of an object based on the contour energy information of the object. This method was first proposed by Kass et al. [Non-Patent Document 1 of Prior Art Document], and various snake algorithms that improved it have been proposed and are still being used in various fields.

The advantage of this method is that it is simpler than other methods and shows efficiency in segmenting unstructured objects comparable to it. Due to these advantages, it is used in various imaging fields such as medical, robot, tracking system, and face recognition. However, it is affected by the energy around the object and has a limit in that the more complex the background, the more difficult it is to segment without preprocessing.

KR 10-1191319 B1 KR 10-2006-0023869 A JP Patent Publication 9-161072 A

Kass, M., Witkin A., Terzopulos, D., “Snakes: Active contour models,” In: Proceeding of the First International Conference on Computer Vision, London, Vol l.1, pp. 256-269, 1987. Lucas, B. D., and Kanade, T. “An iterative image registration technique with an application to stereo vision.” International Joint Conference on Artificial Intelligence, pp. 674-679, 1981. Choi, J. W., Whangbo, T. K., and Kim, C. G. “A contour tracking method of large motion object using optical flow and active contour model.” Multimedia Tools and Applications, Vol. 74, no. 1, p. 199-210, 2015. Takaya, K. “Tracking a video object with the active contour (snake) predicted by the optical flow.” 2008 Canadian Conference on Electrical and Computer Engineering. IEEE, pp. 369-372, 2008.

Accordingly, the technical problem of the present invention is conceived in this respect, and an object of the present invention is to provide a high-speed video object tracking method based on snake and optical flow.

Another object of the present invention is to provide a recording medium on which a computer program for performing the snake and optical flow-based high-speed video object tracking method is recorded.

Another object of the present invention is to provide an apparatus for performing the high-speed video object tracking method based on the snake and the optical flow.

A high-speed video object tracking method based on a snake and an optical flow according to an embodiment for realizing the above object of the present invention includes setting the outline of an object of interest from a high-speed captured video and calculating an initial snake point; generating feature points by performing an optical flow between the n-th frame and the n+1-th frame of the video; selecting an object region to be searched for by utilizing a previous background separation image and a difference image between the n-th frame and the n+1-th frame; adjusting positional information errors of object outlines in the n+1 th frame by comparing positions between the previous background separation image and the object region generated from the difference image and the n+1 th optical flow feature point; and extracting the outline of the object by converting the error-adjusted feature points into snake points of the n+1th frame.

In an embodiment of the present invention, the selecting of the object region to be searched may include generating a front-background separation image between the n-th frame and the n+1-th frame to obtain contour line information; obtaining additional contour information by generating a difference image between the n-th frame and the n+1-th frame; and selecting, as an object region to be searched, a periphery of a common object contour of the difference image and the separated former background image.

In an embodiment of the present invention, the step of generating the difference image and obtaining additional contour information may include calculating a difference image between the n-th frame and the n+1-th frame, and using a preset threshold to prevent noise generation Calculation can be performed only for regions with a larger difference.

In an embodiment of the present invention, the method for tracking a high-speed video object based on the snake and optical flow may further include repeating the step of extracting the outline of the object by converting it into the snake point a number of times.

In an embodiment of the present invention, in the step of adjusting the location information error of the contour, when the distance between the feature point and the object region is greater than a preset threshold value, the position of the feature point may be readjusted.

In an embodiment of the present invention, the calculating of the initial snake point may include: dividing an object for a first frame (n = 0) in a video sequence; Utilizing the contour information of the segmented object as a snake point; and initializing a snake point by setting the snake point as a feature point.

In an embodiment of the present invention, generating feature points by performing the optical flow may include calculating optical flow of feature points in the n-th frame and the n+1-th frame; applying contour points of the object contour estimated by the n-th frame as snake points to the n+1-th frame; and repeating the above steps if there is a next frame.

In a computer readable storage medium according to an embodiment for realizing the above object of the present invention, a computer program for performing the snake and optical flow-based ultra-high-speed video object tracking method is recorded.

An apparatus for tracking a high-speed video object based on snake and optical flow according to an embodiment for realizing another object of the present invention described above is a snake that calculates an initial snake point by setting the outline of an object of interest from a high-speed captured video point setting unit; an optical flow unit generating feature points by performing an optical flow between the n-th frame and the n+1-th frame of the video; a search object region selection unit that selects an object region to be searched for by utilizing the previous background separation image and the difference image between the n-th frame and the n+1-th frame; an area comparator for adjusting positional information errors of object outlines in the n+1 th frame by comparing positions between the separated background image and the object region generated from the difference image and the n+1 th optical flow feature point; and a contour extractor extracting the contour of the object by converting the error-adjusted feature points into snake points of the n+1th frame.

In an embodiment of the present invention, the search object region selection unit may include: a front background separation unit generating a front background separation image between the n-th frame and the n+1-th frame to obtain contour line information; a difference image unit generating a difference image between the n-th frame and the n+1-th frame to obtain additional contour information; and an object selection unit that selects a common object contour around the difference image and the separated front background image as an object region to be searched for.

In an embodiment of the present invention, the difference image unit calculates a difference image between the n-th frame and the n+1-th frame, and calculates only a region in which the difference is larger than a preset threshold in order to prevent noise generation. can

In an embodiment of the present invention, the region comparator may readjust the location of the feature point when the distance between the feature point and the object region is greater than a preset threshold value.

In an embodiment of the present invention, the snake point setting unit divides an object for the first frame (n = 0) in a video sequence, sets contour information of the divided object as a feature point, and initializes the snake point. can

According to the snake and optical flow-based ultra-high-speed video object tracking method, a more accurate and continuous snake contour can be extracted compared to existing methods using the optical flow and snake model. In addition, both temporal consistency and spatial accuracy were superior to existing methods and showed strength in object extraction from complex backgrounds.

=5 예시를 보여주는 도면이다.
도 12는 본 발명의 단계별 영상을 보여주는 도면이다.
도 13은 본 발명의 일 실시예에 따른 스네이크와 옵티컬 플로우 기반의 초고속 동영상 객체 추적 방법의 흐름도이다.
도 14는 도 13의 옵티컬 플로우 활용 과정을 보여주는 흐름도이다.
도 15는 본 발명과 기존방법 간의 수집 영상의 각 프레임 및

값에 따른 정량적 평가를 보여주는 그래프이다.
도 16은 본 발명과 기존방법 간의 비정형 객체에 대한 비교 영상을 보여주는 도면이다.1 is a block diagram of a high-speed video object tracking device based on snake and optical flow according to an embodiment of the present invention.
Figure 2 is a diagram showing the direction of the snake contour.
FIG. 3 is a diagram showing an example of a sum of vectors in FIG. 2 .
4 is a diagram showing ultra-high-speed video optical flow results according to window sizes.
5 is a diagram showing an optical flow for initial feature point creation and object contours.
6 is a diagram illustrating a noise problem of ultra-high-speed video and limitations of optical flow.
7 is a diagram showing an image separated from an original image and a full background.
8 is a diagram showing an example of calculating a difference image between frames.
9 is a diagram showing noise in a difference image.
10 is a diagram showing a process of using a front background separation image and a difference image.
11 is a pixel search distance

=5 This is a drawing showing an example.
12 is a diagram showing step-by-step images of the present invention.
13 is a flowchart of a high-speed video object tracking method based on snake and optical flow according to an embodiment of the present invention.
14 is a flowchart illustrating a process of utilizing the optical flow of FIG. 13 .
15 shows each frame of the collected image between the present invention and the existing method and

It is a graph showing the quantitative evaluation according to the value.
16 is a diagram showing a comparison image of an atypical object between the present invention and an existing method.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a high-speed video object tracking device based on snake and optical flow according to an embodiment of the present invention.

The snake and optical flow-based ultra-high-speed video object tracking device (10, hereinafter) according to the present invention excludes noises spread throughout the ultra-high-speed video based on the position of feature points generated through the optical flow, and A method of searching only around the common object contour appearing in the separation image and the difference image is used. Using this, it is possible to solve the problem of object tracking efficiency deterioration due to noise around the search object.

Referring to FIG. 1 , an apparatus 10 according to the present invention includes a snake point setting unit 100, an optical flow unit 200, a search object region selection unit 300, a region comparison unit 400, and an outline extraction unit ( 500) are included.

In the apparatus 10 of the present invention, software (application) for performing super-high-speed video object tracking based on snake and optical flow may be installed and executed, and the snake point setting unit 100, the optical flow unit 200 , The search object region selection unit 300, the region comparison unit 400, and the contour extraction unit 500 perform ultra-high-speed video object tracking based on the snake and optical flow executed in the device 10 It can be controlled by software for

The device 10 may be a separate terminal or a part of a module of the terminal. In addition, the snake point setting unit 100, the optical flow unit 200, the search object area selection unit 300, the area comparison unit 400, and the contour extraction unit 500 are configured as an integrated module. may be formed, or may consist of one or more modules. However, on the contrary, each component may be composed of a separate module.

The device 10 may be mobile or stationary. The apparatus 10 may be in the form of a server or engine, and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS), or a wireless device. It can be called by other terms such as wireless device, handheld device, etc.

The device 10 may execute or manufacture various software based on an operating system (OS), that is, a system. The operating system is a system program for enabling software to use the hardware of the device, and is a mobile computer operating system such as Android OS, iOS, Windows mobile OS, Bada OS, Symbian OS, Blackberry OS, and Windows-based, Linux-based, Unix-based, It can include all computer operating systems such as MAC, AIX, and HP-UX.

In the present invention, an object is segmented and tracked based on a snake algorithm and an optical flow using high-speed video. The noise problem of high-speed video that appears in this process is solved by combining the image feature, difference image, and foreground separation methods, and the problem that it greatly depends on the position and shape of the initial snake points, which is the limitation of the snake algorithm, is solved by using the feature points of the optical flow. Solve it.

The present invention is largely divided into initial snake point calculation, feature point generation using optical flow, object region selection using difference image and foreground separation, comparison between feature point and object region, and snake point adjustment.

The snake point setting unit 100 calculates an initial snake point by setting the outline of an object of interest from a video captured at high speed.

When setting the initial snake point, set it to the outline of the object of interest and use it. After that, the snake point is adjusted by comparing the feature point generated by performing the optical flow between the n-th frame and the n+1-th frame with the set snake point of the current frame.

In the middle of this process, the difference image is used to compensate for the conversion problem of snake points caused by noise in the ultra-high-speed image and partial errors of the optical flow. Noise that appears even when using a difference image is compensated by using Morpholoy operation and extraction of the largest contour using image features.

In addition, the contour information of the search object is additionally acquired through the background separation, and the error is adjusted after contrasting and comparing each contour position information. Finally, the error-adjusted feature point is used as a snake point and converted in the n+1th frame. After adjusting the snake points, the transformation of the snake points is repeated until the shape of the desired object outline is obtained.

The optical flow unit 200 generates feature points by performing optical flow between the n-th frame and the n+1-th frame of the video.

The snake algorithm sets snake points around the outline of an object to be extracted and repeatedly moves the snake points using a defined energy function. Through this process, the energy function is minimized and the outline of the object is extracted. Snake can be defined as in Equation 1 below.

[Equation 1]

The snake function consists of external energy that pulls the outer snake point toward the contour of the object and internal energy that determines the shape of the snake contour. When expressed as an equation, it is expressed as Equation 2 below.

[Equation 2]

이면

로 정의할 수 있다.

는

번째의

좌표고

는

번째의

좌표가 된다.snake point

the other side

can be defined as

Is

second

Coordinate height

Is

second

becomes the coordinates.

The internal energy determines the shape of the snake contour and is composed of continuity energy that determines the distance between snake points constant and curvature energy that causes the position of the snake point to form a gentle curve. The internal energy function is defined as in Equation 3 below.

[Equation 3]

는 스네이크 포인트들 간의 거리를 조절하기 위해서 스네이크 포인트들의 거리의 차를 활용한다. 즉, 윤곽선의 포인트들이 일정한 간격으로 배치되면

는 작은 값을 갖게 되며 점들이 윤곽선 위에서 한쪽 부분으로 쏠리게 되면 큰 값을 가지게 된다. 이를 식으로 표현하면 아래의 수학식 4가 된다.

utilizes the difference in distance between snake points to adjust the distance between snake points. That is, if the points of the contour line are arranged at regular intervals,

has a small value and has a large value when the points are skewed to one side on the contour. Expressing this as an equation, Equation 4 below becomes.

[Equation 4]

를 활용해야 할 필요가 있다. 평균 거리

는 수학식 5와 같으며 이를 통해 수학식 6과 같이

를 나타낼 수 있다.However, the above formula has the effect of shrinking the contour line and reducing the size, so to keep the distance of the snake points constant, the average distance

You need to use . average distance

Is the same as Equation 5, through which, as shown in Equation 6

can represent

[Equation 5]

[Equation 6]

The distance between the current snake point and the next snake point is determined by the interference by the average distance between all snake points. As a result, it can be seen that the interval between snake points is maintained as the snake outline continues to be processed repeatedly.

는 도 2에서 보이는 것과 같이 스네이크 윤곽선을 변화가 적은 쪽으로 끌어당겨 스네이크 윤곽선을 객체 윤곽으로 움직이게 하는 역할을 한다.curvature energy

As shown in FIG. 2, serves to move the snake contour to the object contour by pulling the snake contour to the side with less change.

는 윤곽선이 완만한 곡선이 되도록 한다. 즉, 윤곽선이 진동(Oscillate)하거나 급격하게 방향을 바꾸는 것을 방지하는 기능을 한다.

는 다음과 같은 수학식 7로 표현한다. 이전 포인트

와 이후 포인트

간의 에너지 함수를 활용하여 유도된다. 정규화는 이웃한 포인트들 중 변화율이 제일 큰 값인

로 정규화 시킨다.

makes the outline a gentle curve. That is, it functions to prevent the contour line from oscillating or rapidly changing its direction.

is expressed by the following Equation 7. previous point

and after point

It is derived using the energy function of the liver. Normalization is the value that has the largest rate of change among neighboring points.

normalize to

[Equation 7]

를 표현한 것으로 세 개의 포인트가 이루는 벡터들의 합을 의미한다. 즉,

는 현재 스네이크 포인트와 근접한 포인트가 이루는 벡터의 합으로 나타낼 수 있고, 그 값이 작은 방향으로 스네이크 포인트를 이동시키게 된다.Figure 3 is

It represents the sum of vectors formed by three points. in other words,

can be expressed as the sum of vectors formed by points adjacent to the current snake point, and moves the snake point in a direction with a smaller value.

를 갖는 2차원 가우시안 함수와 영상이 컨벌루션(Convolution)시켜 표현하면 수학식 8과 같이 된다. The external energy serves to pull the snake contour toward the feature part or contour direction of the object to be searched. Usually, the edge information of the image is used, and among the characteristics of the image, gradient information or intensity of the image is used. standard deviation

When the two-dimensional Gaussian function having , and the image are expressed by convolution, Equation 8 is obtained.

[Equation 8]

If the edge information is used, the snake point is placed on the contour of the object by having a small value where the slope is large.

Here, as shown in FIG. 2, an external energy function is calculated at 8 neighboring positions to find a part where the energy is minimized at the current position. If there is no change in the gradient or intensity of the image, the point is pulled inward due to internal energy, and when it encounters an edge with a large change in the image, it is regarded as a minimum portion at that position, and the movement of the snake point stops.

,

,

가 들어간다. 이 가중치들의 역할은 만들어지는 윤곽선의 형태에 관여한다.

값이 다른 가중치에 비해 상대적으로 크면 윤곽선의 형태가 원에 가까운 형태가 된다. 한 스네이크 포인트에서

값이 0이면 그 점에서 불연속이 허용되며

값이 0이면 구석(Corner)이 허용된다. 따라서, 윤곽선이 물체 경계의 구석 부분에서 일치할 수 있도록 한다.Each energy term has a weight

,

,

goes in The role of these weights is involved in the shape of the contour line being created.

If the value is relatively large compared to other weights, the shape of the outline becomes a shape close to a circle. at one snake point

A value of 0 allows discontinuity at that point and

A value of 0 allows Corners. Thus, it allows the contour lines to match at the corners of the object boundary.

Referring to the optical flow feature point and snake point once again, first, the object is directly segmented by the user for the first frame (n = 0) in the video sequence. The contour information of the divided object is used as a snake point. That is, it is set as a feature point.

와

사이의 옵티컬 플로우 계산은 다음의 수학식 9와 같이 표현된다.After that, the optical flow of feature points is calculated in the n+1th frame. The calculation method used here was the Lucas-Kanade algorithm [Non-Patent Document 2 of Sunhan Technical Document], which is widely used as a method of tracking pixel motion. The Lucas-Kanade algorithm is a method based on three hypotheses: brightness constancy, temporal persistence, and spatial coherence. time in this hypothesis

and

The optical flow calculation between

[Equation 9]

축과

축을 추가시켜 계산한다. 그 후, 속도 요소인

와

를 영상의 밝기 정도를 나타내는

를 활용하여 다음의 수학식 10과 같이 구한다.Here, the partial differential chain rule and the speed factor

axis

Calculate by adding an axis. Then, the speed factor

and

represents the brightness level of the image.

Using , it is obtained as in Equation 10 below.

[Equation 10]

When performing the optical flow, the window size must be set to a higher value than when used in general video in order to be used in ultra-high-speed video. This is because optical flow is not smoothly performed due to noise generated during shooting in high-speed video.

4 shows an example in which optical flow is applied to ultra-high-speed video. As shown in FIG. 4, it is difficult to extract feature points by using a general optical flow due to noise.

The initial feature point is set by the user and then used as the initial snake point when applying the snake model. After applying the optical flow together with the next frame, the optical flow feature point of the next frame is used as a snake point.

The white outline in (c) of FIG. 5 is the object outline of the previous frame. That is, it is the initial snake point in the previous frame. The red outline is the object outline in the next frame. Contour points of the red object contour estimated by optical flow are used as snake points and applied in the next frame.

These existing optical flow utilization methods have a problem in that feature point extraction is not performed smoothly when noise in an image or shaking or occlusion of an object or background occurs. In particular, a situation in which feature points are lost occurs in a video showing complex movements of an object, and a situation in which feature points are scattered due to noise generated around an object occurs when using high-speed video.

In general, when an optical flow is applied to an ultra-high-speed image, a problem occurs in which feature points are lost or are not properly extracted due to noise. In addition, in the case of a sports video as shown in FIG. 5, a problem in which feature points are not properly extracted is revealed even in a part where the movement of the object is complicated or the contours of the object overlap.

In FIG. 6 , it can be seen that the feature points of the optical flow are lost or scattered in the lower body. This is because noise is generated by grass in the background of the lower body area, and feature points are scattered by noise. In addition, there is also a problem that feature points are lost due to the occlusion of the legs.

The search object region selection unit 300 selects an object region to be searched for using a difference image between the n-th frame and the n+1-th frame and a previous background separation image.

A background separation method was used to obtain contour information of an object even in such a complex motion. The method used is a background separation method based on a Gaussian synthesis model.

This method adaptively and automatically determines the number of Gaussian distributions in the existing Gaussian synthesis model instead of directly determining the number of Gaussian distributions, and increases the processing time by adjusting the number of distributions according to the amount of data belonging to the distribution. As a result, it shows the result that no temporal problem occurs during separation due to the high speed of ultra-high-speed imaging compared to other methods.

7 shows an original video frame and a frame after separating the foreground. It can be seen that even after the front background separation, a lot of noise appears due to the high-speed image characteristics, but the outline of the search object is displayed well. Table 1 below shows an example of an adaptive Gaussian synthesis model algorithm.

Algorithm 1: Adaptive Gaussian Synthesis Model Algorithm

,

,

,

Output : 수정된 확률 분포

번 째 성분에 속하는 샘플의 개수를

라고 가정한다.
가정된

을 통한 우도 함수

를 계산한다.
※혼합 계수는 1로 고정한다.
라그랑제 승수를 도입하여 최대우도

를 구하고 라그랑제 승수를 제거하여

를 구한다.
확률 분포

에 대한 수정 방정식을 구한다.

켤레 사전분포인 Dirichlet 사전분포를 구한다.
※

은 가우시안 분포

에 대해 이전에 속했던 샘플 수(prior evidence)를 의미하며

는 이전에 속했던 샘플 수를 측정해 분포의 생존 유무를 판단하는 계수이다.

, where

,

최대우도 + Dirichlet 사전분포 ※

최종 확률 분포

수정 방정식 ※

Input: data sample

,

,

,

Output: modified probability distribution

The number of samples belonging to the first component

Assume that
assumed

Likelihood function via

Calculate
※The mixing factor is fixed at 1.
Maximum likelihood by introducing Lagrangian multipliers

and remove the Lagrangian multiplier to

save
probability distribution

Find the corrected equation for

Find the Dirichlet prior distribution, which is a conjugate prior distribution.
※

is Gaussian distribution

It means the number of samples previously belonging to (prior evidence) for

is a coefficient that determines whether a distribution is alive or not by measuring the number of samples that belonged to it previously.

, where

,

Maximum likelihood + Dirichlet prior ※

final probability distribution

Correction equation ※

값보다 차이가 큰 영역에 대해서만 계산을 진행한다.After that, in order to improve the performance of contour extraction, as in the example of FIG. 8, the difference between the current frame and the next frame is calculated, and in order to prevent noise

Calculation is performed only in the area where the difference is greater than the value.

However, in the case of ultra-high-speed video, a lot of noise is generated around the object area and throughout the background, so it is not possible to achieve effective noise prevention through the use of thresholds. 9 is an enlarged picture of a difference image, and it can be seen that a lot of noise is generated around the object area.

[Equation 11]

In order to solve this noise problem, the present invention excludes noise spread throughout the high-speed image based on the feature point position generated through the optical flow, and uses a method of searching only around the common object contour appearing in the front background separation image and the difference image. Thus, the object tracking efficiency deterioration problem caused by noise around the search object can be solved.

The area comparison unit 400 compares the feature points with the object area and adjusts an error in the location information of the object contour in the n+1 th frame.

The process of FIG. 10 proceeds only for a region adjacent to a feature point that appears after applying the optical flow. Looking at this, a common area that is not classified as a background is revealed when the results of the image separated from the previous background and the difference image are compared. It is a part indicated by a square in FIG. 10 . It can be seen that the fine noises appearing in the difference image and the noises appearing in the image separated from the background are different.

These noises are compared and if they do not appear in common, they are all processed as background. After that, in order to move the position of the feature point of the optical flow, comparison between the difference image and the image generated through the separation of the previous background image is performed. The search starts from the location of the initial feature point, and since the comparison image has only contour information, neighboring contours are searched.

The contour extractor 500 extracts the contour of the object by converting the error-adjusted feature point into a snake point of the n+1th frame.

The movement of feature points proceeds as shown in FIG. 11 . As a result, the method combining the foreground separation rather than the method using the existing snake algorithm and optical flow increases the accuracy of contour extraction through comparison of contour information, thereby increasing the efficiency of object contour extraction. Table 2 is an example of an algorithm using a front background separation image and a difference image.

Algorithm 2: Utilization process of pre-background separation image and difference image

, 전배경 분리 영상

, 옵티컬 플로우 영상

Output : 배경이 분리된 윤곽선 영상

,

윤곽선 비교 영상
전배경 분리 영상과 차영상을 비교하여 윤곽선 비교 영상을 생성
For each

do

end for
옵티컬 플로우 특징점을 기초로 하여 윤곽선 비교 영상과 대조 후 위치 재조정
For each

do
For each

,

do
if

then

특징점 위치 재조정

end for
재조정된 특징점을 활용하여 예측 윤곽선 영상 생성
For each

do

end forInput: Cha Young-sang

, Full-background separation image

, optical flow imaging

Output: Contour image with background separated

,

Contour Comparison Video
Contour comparison image is generated by comparing the previous background separation image and the difference image.
For each

do

end for
Position readjustment after comparison with contour comparison image based on optical flow feature points
For each

do
For each

,

do
if

then

Repositioning feature points

end for
Generate a predicted contour image using the readjusted feature points
For each

do

end for

프레임과

프레임 사이의 변화가 하얀색 윤곽선과 빨간색 윤곽선으로 표시되어 있다. 그 후, (c) 영상에 보이는 것과 같이 전배경 분리를 진행하여 윤곽선 정보를 습득한다. 전배경 분리 영상의 경우, 많은 잡음이 나타나는 것을 볼 수 있다. Referring to FIG. 12, (a) is an image in which a user sets an initial snake point. Based on the set snake point, the optical flow proceeds in (b). (b) the video is

frame and

Changes between frames are indicated by white and red outlines. After that, (c) as shown in the image, the front background is separated to acquire the contour line information. In the case of the background-separated image, it can be seen that a lot of noise appears.

프레임과

프레임 사이의 차영상을 습득하여 전배경 분리 영상에 이은 추가적인 윤곽선 정보를 습득한다. 마지막으로 (e)에서는 전배경 분리된 영상에서의 윤곽선 정보와 옵티컬 플로우를 통해 얻어진 특징점, 차영상을 통해 얻어진 객체 영역을 활용한 최종적인 윤곽선 추출 영상을 보여준다. 마찬가지로, 표 3은 단계별 영상에 따른 알고리즘 순서를 나타낸다.In step (d)

frame and

By acquiring the difference image between frames, additional contour information following the previous background separation image is obtained. Finally, (e) shows the final contour extraction image using the contour information from the image separated from the whole background, the feature points obtained through the optical flow, and the object region obtained through the difference image. Similarly, Table 3 shows the algorithm sequence according to the step-by-step image.

Algorithm 1: Object tracking in ultra-high-speed video

영상과

영상
Output : 배경이 분리된 윤곽선 영상
1.1 스네이크 포인트 설정 및 옵티컬 플로우
사용자에 의한 초기 윤곽선 추출
스네이크 포인트 초기화
while

do
스네이크 수행 및 스네이크 포인트 이동
end while

특징점 =

영상 스네이크 포인트

특징점과

영상으로 옵티컬 플로우를 계산

영상 스네이크 포인트 = 계산되어진

옵티컬 플로우 특징점

1.2 전배경 분리 및 차영상 계산
GMM(

영상과

영상) => 전배경 분리 영상 생성

영상과

영상을 통한 차영상 생성

1.3 옵티컬 플로우 특징점과 객체 영역간의 비교
전배경 분리 영상과 차영상을 통해 생성된 객체 영역과

옵티컬 플로우 특징점 간의 위치 비교
for each 특징점

객체 영역

,

do
특징점과 객체 영역 사이의 거리 <

:

특징점 위치 재조정

end for

1.4 윤곽선 추출

재조정된 옵티컬 플로우 특징점 영상을 통해 스네이크 포인트 설정

while

do
스네이크 수행 및 스네이크 포인트 이동
end whileInput:

video department

video
Output: Contour image with background separated
1.1 Snake point setup and optical flow
Initial contour extraction by user
Snake point reset
while

do
Perform snake and move snake point
end while

feature point =

video snake point

feature point

Calculate optical flow from images

Video Snake Point = Calculated

Characteristics of optical flow

1.2 Background Separation and Difference Image Calculation
GMM(

video department

Image) => Generates a background-separated image

video department

Create difference image through image

1.3 Comparison between optical flow feature points and object areas
The object area created through the previous background separation image and the difference image

Position comparison between optical flow feature points
for each feature

object area

,

do
Distance between feature point and object area <

:

Repositioning feature points

end for

1.4 Contour Extraction

Snake point setup through recalibrated optical flow feature point image

while

do
Perform snake and move snake point
end while

Accordingly, the present invention can extract a more accurate and continuous snake contour compared to conventional methods using an optical flow and a snake model. In addition, both temporal consistency and spatial accuracy were superior to existing methods and showed strength in object extraction from complex backgrounds. In the future, it is expected to show better results when using the snake model and optical flow using neural networks.

13 is a flowchart of a high-speed video object tracking method based on snake and optical flow according to an embodiment of the present invention. 14 is a flowchart illustrating a process of utilizing the optical flow of FIG. 13 .

The method for tracking a high-speed video object based on snake and optical flow according to the present embodiment may be performed in substantially the same configuration as the apparatus 10 of FIG. 1 . Accordingly, components identical to those of the apparatus 10 of FIG. 1 are given the same reference numerals, and repeated descriptions are omitted.

Also, the snake and optical flow-based ultra-high-speed video object tracking method according to the present embodiment may be executed by software (application) for performing snake- and optical flow-based ultra-high-speed video object tracking.

Referring to FIG. 13 , in the snake and optical flow-based ultra-high-speed video object tracking method according to the present embodiment, an initial snake point is calculated by setting the outline of an object of interest from an ultra-high-speed captured video (step S10).

Referring to FIG. 14 , an object is divided for a first frame (n = 0) in a video sequence (step S11). The contour information of the segmented object is used as a snake point (step S12), and the snake point is set as a feature point to initialize the snake point (step S13).

Feature points are generated by performing an optical flow between the n-th frame and the n+1-th frame of the video (step S20).

Referring to FIG. 15, the optical flow of feature points in the nth frame and the n+1th frame is calculated (step S21), and the two frames are compared (step S22). Contour points of the object contour estimated by the n-th frame are used as snake points and applied to the n+1-th frame (step S23). If there is a next frame (step S24), the above steps are repeated (step S25).

An object region to be searched is selected using the previous background separation image and the difference image between the n-th frame and the n+1-th frame (step S30).

In the step of selecting the object region to be searched, a front background separation image between the n th frame and the n + 1 th frame is generated to obtain contour line information, and a difference image between the n th frame and the n + 1 th frame to obtain additional contour information. Based on this, the area around the common object contour of the difference image and the separated former background image is selected as the object region to be searched.

In the step of obtaining additional contour information by generating the difference image, a difference image between the n-th frame and the n+1-th frame is calculated, and only for an area where the difference is larger than a preset threshold in order to prevent noise generation. proceed with the calculation

A location information error of the contour of the object in the n+1 th frame is adjusted by comparing the position between the previous background separation image and the object region generated from the difference image and the n+1 th optical flow feature point (step S40). . In this case, when the distance between the feature point and the object area is greater than a preset threshold value, the location of the feature point may be readjusted.

The contour of the object is extracted by converting the error-adjusted feature points into snake points of the n+1th frame (step S50).

In the step of extracting the outline of the object by transforming into snake points, the transformation of the snake points is repeated a number of times until a desired shape of the object outline is obtained.

Hereinafter, experimental results compared with prior art Choi [Non-Patent Document 3 of Prior Art Document] and K Takaya [Non-Patent Document 4 of Prior Art Document] are described in order to verify the performance of the present invention.

값이 늘어날수록 전반적으로 성능이 올라가는 것을 볼 수 있으며 본 발명에서 제안하는 방법이 다른 방법들보다 우수한 성능을 나타내는 것을 확인할 수 있다.15 shows the result of measuring the average of temporal coherence and spatial accuracy by varying the threshold for each frame. Referring to Figure 15,

It can be seen that overall performance increases as the value increases, and it can be confirmed that the method proposed in the present invention exhibits better performance than other methods.

16 is for qualitative evaluation, and is a result of an experiment on an atypical object and an experiment on how existing methods and the proposed method respond to an occlusion phenomenon. Referring to FIG. 16, it can be confirmed that the method proposed in the present invention is more robust than the existing methods even in a complex background such as a grass background of a soccer video.

Such a snake and optical flow-based ultra-high-speed video object tracking method may be implemented as an application or implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although the above has been described with reference to embodiments, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will understand.

The present invention uses a method of excluding noises spread throughout the ultra-high-speed image based on feature point locations generated through optical flow and searching only around common object contours appearing in the front-background separation image and the difference image. Using this, it is possible to solve the problem of object tracking efficiency deterioration due to noise around the search object.

As a result, the method combining the foreground separation rather than the method using the existing snake algorithm and optical flow increases the accuracy of contour extraction through comparison of contour information, thereby increasing the efficiency of object contour extraction. Accordingly, it is expected to be useful in various fields such as sports image analysis and scientific experiment image analysis.

10: high-speed video object tracking device
100: snake point setting unit
200: optical flow unit
300: search object area selection unit
400: area comparison unit
500: contour extraction unit

Claims (13)

calculating an initial snake point by setting an outline of an object of interest from a high-speed captured video;
generating feature points by performing an optical flow between the n-th frame and the n+1-th frame of the video;
selecting an object region to be searched for by utilizing a previous background separation image and a difference image between the n-th frame and the n+1-th frame;
adjusting positional information errors of object outlines in the n+1 th frame by comparing positions between the previous background separation image and the object region generated from the difference image and the n+1 th optical flow feature point; and
Extracting the outline of the object by converting the error-adjusted feature points into snake points of the n+1th frame;
In the step of selecting the object area to be searched,
obtaining contour line information by generating a background separation image between the n-th frame and the n+1-th frame;
obtaining additional contour information by generating a difference image between the n-th frame and the n+1-th frame; and
A high-speed video object tracking method based on snake and optical flow, comprising: selecting a common object contour around the difference image and the front background separation image as an object region to be searched for.
delete The method of claim 1, wherein generating the difference image to obtain additional contour information comprises:
High-speed video object tracking based on snake and optical flow that calculates the difference image between the n-th frame and the n+1-th frame and calculates only the area where the difference is greater than the preset threshold to prevent noise generation method.
According to claim 1,
Further comprising, repeating the step of extracting the outline of the object by converting it to the snake point multiple times; further comprising a snake and optical flow-based high-speed video object tracking method.
The method of claim 1, wherein the step of adjusting the location information error of the contour line comprises:
An ultra-high-speed video object tracking method based on snake and optical flow, which readjusts the position of a feature point when the distance between a feature point and an object region is greater than a preset threshold.
The method of claim 1, wherein calculating the initial snake point comprises:
Segmenting an object for a first frame (n = 0) in a video sequence;
Utilizing the contour information of the segmented object as a snake point; and
A method for tracking a high-speed video object based on snake and optical flow, comprising: initializing a snake point by setting the snake point as a feature point.
7. The method of claim 6, wherein generating feature points by performing the optical flow comprises:
calculating an optical flow of feature points in the n-th frame and the n+1-th frame;
applying contour points of the object contour estimated by the n-th frame as snake points to the n+1-th frame; and
A method for tracking a high-speed video object based on snake and optical flow, comprising: repeating the above steps if there is a next frame.
A computer-readable storage medium on which a computer program for performing the snake and optical flow-based ultra-high-speed video object tracking method according to claim 1 is recorded.
a snake point setting unit that calculates an initial snake point by setting the outline of an object of interest from a high-speed captured video;
an optical flow unit generating feature points by performing an optical flow between the n-th frame and the n+1-th frame of the video;
a search object region selection unit that selects an object region to be searched for by utilizing the previous background separation image and the difference image between the n-th frame and the n+1-th frame;
an area comparator for adjusting positional information errors of object outlines in the n+1 th frame by comparing positions between the separated background image and the object region generated from the difference image and the n+1 th optical flow feature point; and
A contour extractor extracting the contour of the object by converting the error-adjusted feature point into a snake point of the n+1th frame;
The search object area selection unit,
a front-background separator for obtaining contour information by generating a front-background separation image between the n-th frame and the n+1-th frame;
a difference image unit generating a difference image between the n-th frame and the n+1-th frame to obtain additional contour information; and
A high-speed video object tracking device based on snake and optical flow, comprising: an object selection unit for selecting a common object contour of the difference image and the front background separation image as an object region to be searched for.
delete The method of claim 9, wherein the difference image unit,
High-speed video object tracking based on snake and optical flow that calculates the difference image between the n-th frame and the n+1-th frame and calculates only the area where the difference is greater than the preset threshold to prevent noise generation Device.
10. The method of claim 9, wherein the area comparison unit,
A high-speed video object tracking device based on snake and optical flow that readjusts the position of a feature point when the distance between a feature point and an object region is greater than a preset threshold.
The method of claim 9, wherein the snake point setting unit,
A high-speed video object tracking device based on snake and optical flow that divides an object for the first frame (n = 0) in a video sequence and initializes the snake point by setting the contour information of the divided object as a feature point.

Images