CN111967360B - Target vehicle posture detection method based on wheels - Google Patents
Target vehicle posture detection method based on wheels Download PDFInfo
- Publication number
- CN111967360B CN111967360B CN202010784979.1A CN202010784979A CN111967360B CN 111967360 B CN111967360 B CN 111967360B CN 202010784979 A CN202010784979 A CN 202010784979A CN 111967360 B CN111967360 B CN 111967360B
- Authority
- CN
- China
- Prior art keywords
- vehicle
- wheel
- coordinates
- target
- wheels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 claims abstract description 62
- 239000011159 matrix material Substances 0.000 claims description 34
- 230000009466 transformation Effects 0.000 claims description 23
- 238000004364 calculation method Methods 0.000 claims description 22
- 230000011218 segmentation Effects 0.000 claims description 14
- 238000013528 artificial neural network Methods 0.000 claims description 12
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 claims description 11
- 230000000007 visual effect Effects 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 5
- 238000013136 deep learning model Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 238000003062 neural network model Methods 0.000 claims description 2
- 238000013135 deep learning Methods 0.000 claims 1
- 230000003447 ipsilateral effect Effects 0.000 abstract 1
- 238000006243 chemical reaction Methods 0.000 description 10
- 238000005259 measurement Methods 0.000 description 5
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000011426 transformation method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/08—Detecting or categorising vehicles
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
A method of wheel-based target vehicle attitude detection, comprising: (1) acquiring a current camera image; (2) Inputting the obtained current camera image, and detecting a vehicle target of the input image to obtain the vehicle target; (3) Detecting the vehicle target to obtain a wheel target; (4) extracting two ipsilateral wheels from the wheel target; (5) And obtaining the inclination angle posture of the vehicle by using the connecting line of the central coordinates of the wheels on the two same sides. The method can quickly realize the turning angle gesture of the vehicle without marking the gesture of the vehicle, can accurately detect the gesture of the target in real time in a larger range in a medium-high speed state and a low speed state, and is very suitable for practical application in automatic driving or auxiliary driving.
Description
Technical Field
The invention relates to the field of automatic driving and computer vision, in particular to a technology for analyzing the position and the posture of a target vehicle in automatic driving based on computer vision recognition, and particularly relates to a method and a device for measuring and calculating the posture of the target vehicle based on wheels.
Background
With technological progress and technological development, autopilot is becoming a research hotspot in the field of current transportation vehicles.
During the course of autonomous vehicle travel, the position and attitude of the target vehicle have a critical effect on control decisions. In the automatic driving process, the road conditions are complex and changeable, the running speed of the vehicle is high, and in order to improve the driving safety, the running state of the object vehicle needs to be accurately detected in real time, and the running intention of the object vehicle is prejudged in advance. When the host vehicle is automatically connected or suddenly inserted from the side by other vehicles, the behavior intention of the object vehicle is needed to be judged in advance, so that a control decision can be adjusted in real time according to the intention of the object vehicle, and safe driving is ensured. It is therefore important to obtain the attitude, especially the turning angle, of the subject vehicle.
In the prior art, methods for acquiring a vehicle corner and a vehicle driving intention mainly include two types:
1) Predicting through a neural network;
2) And a method of estimating using a history trajectory.
The Chinese patent document CN110232379A (invention name: a vehicle posture detecting method and system) proposes a method for detecting the vehicle posture by adopting a target detection network model, detecting the image area of a target vehicle in an image to be detected, and outputting the inclination angle of the target vehicle at the same time when the network model predicts. However, the method requires a large amount of vehicle marking data with angles to train, is not suitable for practical implementation, and the stability of the result obtained by the detection method is poor, so that the practical use requirement cannot be met.
US20030235332A1 (title of the invention: SYSTEM AND method for pose-angle estimation) proposes a method for gesture detection, which uses a target detection network, trains a neural network through regression estimation, and performs angular gesture detection on an input image. However, the method also requires a large amount of attitude image annotation data with angles for training, and the detection result depends on the precision of data calibration; and the neural network is adopted for prediction, so that the calculation consumption is high, and the real-time requirement of vehicle gesture detection in automatic driving is not facilitated.
Chinese patent document CN109305171a (title of the invention: a forward vehicle running posture observation apparatus and method thereof) uses the state of comparison of the vehicle front wheel position with the lane line to determine the forward vehicle running safety, but does not give vehicle posture information.
Chinese patent document CN109949364a (title: an optimization method for vehicle attitude detection accuracy based on a drive test monocular camera) describes a method for estimating a vehicle traveling direction based on vehicle trajectory tracking. However, this method requires knowledge of the history of the vehicle and is not effective at low speeds.
By adopting the method for forecasting through the neural network, a large amount of vehicle attitude data are required to be collected, higher calibration precision is required, the workload is complex, the quick implementation of vehicle attitude estimation is not facilitated, larger calculation force support of an operation platform is generally required, the real-time performance is poor, and the method is difficult to apply in actual automatic driving.
The method for estimating the historical track is only suitable for being applied in a medium-high speed state, is difficult to realize in a low-speed and congestion state, and cannot acquire the posture of the object vehicle.
Therefore, how to accurately detect the gesture of the target in real time in a large range is a technical problem to be solved.
Disclosure of Invention
Aiming at the technical problems, the invention provides a novel method and a device for measuring and calculating the posture of a target vehicle based on wheels, which are used for researching a target vehicle posture measuring and calculating system with large application range, high measurement precision and low calculation consumption and improving the precision and efficiency of front vehicle target detection in automatic driving.
To solve the above technical problem, according to an aspect of the present invention, there is provided a method for detecting a target vehicle posture based on a wheel, including:
step 1, acquiring a current camera image;
Step 2, inputting the obtained current camera image, and detecting a vehicle target on the input image to obtain the vehicle target;
step 3, detecting the wheels of the vehicle target to obtain the wheel target;
step 4, extracting characteristic points of two wheels on the same side in the wheel target;
and 5, obtaining the inclination angle posture of the vehicle by utilizing the connecting line of the coordinates of the characteristic points of the wheels on the same side.
Preferably, before the current camera image is acquired, calibrating the camera is required, including calibrating camera internal parameters and external parameters;
The camera intrinsic parameter matrix is shown in formula (1):
Wherein f x,fy represents the focal length of the camera in the X, Y coordinate direction, and p x,py represents the offset of the center point of the camera relative to the upper left corner;
the camera external parameter matrix is shown in formula (2):
Where r ij (i=1, 2,3; j=1, 2, 3) represents an element of a rotation matrix of the camera with respect to the world coordinate system, and t i (i=1, 2, 3) represents a translational relationship of the camera coordinate system with respect to the world coordinate system;
And, the relation matrix EM of the camera internal parameters and external parameters is shown in formula (3):
Where m ij (i=1, 2,3; j=1, 2,3, 4) represents an element of a parameter matrix consisting of camera internal and external parameters.
Preferably, the camera comprises an RGB camera, an infrared camera or a depth camera.
Preferably, the camera is a video camera mounted at the front end of the vehicle.
Preferably, a trained deep learning model is adopted to detect a vehicle target, and coordinates and sizes (x, y, w, h) of a two-dimensional frame of the vehicle target V, namely, a target frame in an image are obtained, wherein (x, y) represents the left upper corner coordinates of the two-dimensional frame, and w and h are the width and height of the two-dimensional frame.
Preferably, the wheel detection is performed within the already acquired target frame, a wheel frame in the image is obtained, denoted W i(xi,yi,wi,hi), wherein i=1, 2..n, N denotes the number of detected wheels, wherein x i、yi、wi and h i denote the x-coordinate, y-coordinate, wheel frame width and height, respectively, of the upper left corner of the wheel frame.
Preferably, firstly, the vehicle targets in the camera view are divided into a left target and a right target, then the left vehicle and the right vehicle are respectively detected for the wheels, and the two same-side wheel frames are extracted;
Extracting the detected feature point pixel coordinates P i(ui,vi) of the wheel frame W i(xi,yi,wi,hi), converting the feature point pixel coordinates into a world coordinate system, and acquiring coordinates in the world coordinate system
And obtaining the top, center or bottom coordinates of two wheels of the vehicle based on the world coordinates of the wheels obtained in the process, and obtaining the inclination posture of the vehicle according to the connecting line of the top, center or bottom coordinates of the two wheels.
Preferably, the trained deep neural network model is used for detecting the wheels in the target frame of the vehicle, and the detection result is the coordinates and the dimension W i(xi,yi,wi,hi of the wheels in the image.
Preferably, the feature point detection of the wheel uses a segmentation map of the target vehicle frame and the corresponding vehicle, wherein the segmentation map is obtained based on a deep neural network segmentation model.
Preferably, the detection of the target vehicle wheel characteristic point includes the following steps:
Step 1), according to the position of the target vehicle in the current image, the method is divided into two cases of left side and right side: if the abscissa x+w/2 of the center point of the target vehicle frame is on the left side of the image, the target vehicle is considered to be on the left side of the current vehicle, and if x+w/2 is on the right side of the image, the target vehicle is considered to be on the right side of the current vehicle;
step 2), when the target vehicle is on the right side of the current vehicle, firstly searching and finding the coordinates of a first row of vehicle pixel points in a target block image area from bottom to top based on the vehicle pixels in the segmentation map to serve as a first vehicle wheel reference coordinate P w1, then counting a first proportion r wpix1 of the number of the vehicle pixels in the whole neighborhood delta in the neighborhood delta of the coordinates, if r wpix1 is larger than a first threshold T wpix1, considering the coordinates to be valid, otherwise, recognizing the coordinates to be invalid, and continuing to search upwards;
Step 3), taking the wheel reference coordinate P w1 meeting the conditions in the step as a starting point, taking a point on the right boundary of a vehicle target frame as an end point P wr (x+w, y+h-s), forming a scanning line, wherein s is the scanning times, 0< s < h, the scanning sequence is that s increases from 1, then counting a second proportion r spix of the whole scanning line occupied by the vehicle pixel point on the scanning line, if r spix is larger than a second threshold T spix, the line is considered to be an effective vehicle scanning line, then finding a point closest to the end point on the scanning line as a second wheel reference coordinate P w2, then counting a third proportion r wpix2 of the vehicle pixel number in the whole neighborhood delta in the neighborhood delta of the coordinate, if r wpix2 is larger than a third threshold T wpix2, otherwise, recognizing the coordinate to be invalid, and continuing scanning;
Step 4), if the effective coordinates are found after the scanning in the step is finished, taking P w1 and P w2 as vehicle feature points;
Step 5), when the target vehicle is on the left side of the current vehicle, the coordinates of the end point P wr are expressed as (x, y+h-s) in the step 3), and the rest steps are the same.
Preferably, when the image resolution is 640×480, the neighborhood δ is a 5×5 rectangular neighborhood, the first threshold T wpix1 is 0.2, the second threshold T spix is 0.2, and the third threshold T wpix2 is 0.2.
Preferably, a depth learning method is adopted to detect the visual proportion or the shielding proportion of the wheels;
If the visual proportion is greater than a fourth threshold T vis, the wheels are considered to be complete in the image;
If the occlusion ratio is less than the fifth threshold T hid, the wheel is considered to appear complete in the image.
Preferably, if the difference δy=abs (y bi-yb) between the detected bottom ordinate y bi＝yi+hi of the wheel frame W i(xi,yi,wi,hi) and the two-dimensional bottom ordinate y b =y+h of the vehicle object V where it is located is smaller than the sixth threshold T, and the difference between y b and the bottom edge of the image is also smaller than the sixth threshold T, then the wheel is considered to be incomplete in the image at this time.
Preferably, the sixth threshold T takes a value of 5px when the resolution of the original image is 640×480.
Preferably, when the wheel appears intact in the image, the wheel bottom and ground contact point is used as a feature point; when the wheel appears incomplete in the image, the midpoint of the wheel top is used as a feature point.
Preferably, if the target vehicle is a vehicle with more than 2 wheels on one side, when the vehicle is detected on the same side, detecting all N w wheels on the same side, selecting N w characteristic point coordinates of the wheels on the same side of the vehicle, wherein N w is more than 2, acquiring a straight line equation y w＝kwxw+bw of the wheels by a fitting method, then selecting x coordinates of two coordinates with maximum and minimum transverse coordinates of N w characteristic points in N w wheels, respectively marking the x coordinates as x wmax and x wmin, substituting the x coordinates into the straight line equation y w＝kwxw+bw to obtain two corresponding y coordinates, respectively marking the y wmax and y wmin, and obtaining the characteristic point coordinates of the wheels of the final calculated vehicle pose.
Preferably, the target vehicle type is a truck, a large truck or an engineering vehicle.
Preferably, the feature point pixel coordinates are transformed into a world coordinate system using a perspective transformation matrix.
Preferably, the feature point pixel coordinates are converted into a world coordinate system using equations (4) and (5)
Wherein,
Wherein m ij (i=1, 2,3; j=1, 2,3, 4) represents an element of a parameter matrix consisting of camera internal and external parameters; u i、vi denotes the x-coordinate and y-coordinate of the wheel-frame feature point numbered i in the image coordinate system; z represents the coordinate value of the Z axis of the feature point in the world coordinate system; x i w、yi w represents the coordinate value of the X-axis and the coordinate value of the Y-axis of the pixel coordinate in the world coordinate system, respectively.
Preferably, when using the contact point of the wheel bottom with the ground as the characteristic point, z=0, where equations (4) and (5) are simplified to the form of equations (6) and (7)
Preferably, when using the midpoint of the top of the wheel as the feature point, z=d wheel, where D wheel represents the diameter of the wheel.
Preferably, determining whether the vehicle target is traveling on the LEFT side (LEFT) or the RIGHT side (RIGHT) based on a position where the vehicle target appears in the field of view; if traveling on the LEFT side (LEFT), the two side wheels of the target vehicle seen in the field of view are located on the right side of the target frame; conversely, if the target vehicle is traveling on the RIGHT (RIGHT), then the two side wheels of the target vehicle are seen to be on the left side of the target frame.
Preferably, for the feature point pixel coordinates P i(ui,vi), if all the detected wheels are entirely present in the field of view, the contact point of the wheel bottom and the ground is used as the feature point, which is noted as formula (8)
Preferably, for the feature point pixel coordinates P i(ui,vi), if it is detected that the wheel is incomplete in the image, the midpoint of the wheel top is used as the wheel feature point, which is noted as formula (9)
In this case, when the point on the calculated image is down-converted to the world coordinate system, the point (A, B, C, D) in the world coordinate system taken during the calculation of the perspective transformation matrix is not on the ground plane, but is parallel to the ground plane z=d wheel, where D wheel represents the diameter of the wheel.
Preferably, converting the feature point pixel coordinates to the world coordinate system includes:
4 points under a fixed world coordinate system are marked as points A, B, C, D, and a rectangular range formed by the points is used as a bird's eye view diagram representable area;
converting the 4 points under the world coordinate system into the image coordinate system by using the camera external reference matrix and the internal reference matrix as shown in formula (10)
Here, [ Xw s Yws Zws ], (s=a, B, C, D) represents the coordinates of 4 points A, B, C, D in the world coordinate system, zc represents the value of the corresponding point in the Z-axis direction of the coordinates of the camera coordinate system, [ u s,vs ], (s=a, B, C, D) is the obtained coordinates of the corresponding point in the image coordinate system.
Preferably, 4 vertexes of the aerial view are taken as perspective transformation reference points and combined with 4 points under an original image coordinate system to obtain a perspective transformation matrix Warp from the original image to the aerial view;
Calculating a proportional relation G w between pixel values in the aerial view and coordinates of the world coordinate system according to the scale of 4 points in the world coordinate system and the scale relation of the aerial view;
The points on the image coordinate system are converted to the world coordinate system by equation (11)
Wherein Z t is not practically significant in the formula; x i w、yi w represents the x-coordinate value and y-coordinate value of the pixel coordinate in the world coordinate system, respectively.
Preferably, the calculation steps of the transformation matrix Warp are as follows:
Step 1), taking 4 points on the ground in a world coordinate system As 4 vertices in the bird's eye view field, where i=1, 2,3,4 denote the indices of the points,Is a coordinate value in a world coordinate system; in the world coordinate system, the 4 points form a rectangle, the sides of the rectangle are parallel to the coordinate axes on the world coordinate system, and in the aerial view, the coordinates of the 4 vertexes are respectivelyW img、Himg are coordinate values on a world coordinate system;
step 2), according to the internal parameters and external parameters of the camera, the 4 points are processed Projected onto image coordinates, expressed asWhere i=1, 2,3,4 denotes the index of the point,Coordinate values in an image coordinate system;
Step 3), passing through the above points And (3) withThe corresponding relation of (2) constitutes the following equation set:
and solving a homography matrix Warp between the aerial view and the original image.
Preferably, if the characteristic point of the wheel is the contact position of the wheel with the ground, thenIf the characteristic point of the wheel is the wheel center, thenIf the characteristic point of the wheel is the top of the wheel, then
The diameter D wheel of the wheel is obtained based on the trained deep neural network.
Preferably, the actual height Z w of the wheel feature at this time is recorded as 0 when the wheel is traveling on the ground.
Preferably, the world coordinates of the two wheel feature points are recorded asWherein Z w represents the actual height of the wheel feature; the attitude angle of the vehicle is shown as formula (12)
Preferably, the motion model is used to track vehicle coordinates and pose.
Preferably, the coordinate and the posture of the vehicle are tracked and filtered by using a CTRV motion model, and the coordinate value and the vehicle posture value calculated by each frame of image are taken as observed quantity, so that the prediction is performed by using a Kalman tracking method.
In order to solve the above technical problem, according to another aspect of the present invention, there is provided a vehicle-wheel-based target vehicle posture detection apparatus including:
an image acquisition device that acquires a current camera image;
the vehicle target detection device inputs the acquired current camera image, and carries out vehicle target detection on the input image to obtain a vehicle target;
the wheel target detection device is used for detecting the wheels of the vehicle target to obtain the wheel target;
the same-side wheel extraction device is used for extracting characteristic points of two wheels on the same side in a wheel target;
and the inclination angle posture calculating device obtains the inclination angle posture of the vehicle by utilizing the connecting line of the coordinates of the characteristic points of the wheels on the same side.
The invention has the beneficial effects that:
1. According to the invention, the judgment of the vehicle corner attitude can be rapidly realized without marking the vehicle attitude, and good performance is still realized under low-speed congestion road conditions; the method overcomes the defect that the real-time performance cannot be ensured because a large amount of vehicle attitude data are required to be calibrated when the traditional method is used for predicting through the neural network, and solves the problem that the traditional historical track calculation method is only suitable for being applied in a medium-high speed state and cannot be applied in a low-speed state;
2. the wheel is used as a main basis for posture judgment, and posture estimation is carried out under a bird's eye view, so that the accuracy of posture calculation is improved, and the real-time accurate posture detection can be carried out on the target in a larger range;
3. The method for measuring and calculating the attitude of the target vehicle based on the wheels is simple and feasible, has high measuring and calculating precision, and is suitable for practical application in automatic driving or auxiliary driving.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention. The above and other objects, features, and advantages of the present invention will become more apparent by describing in detail embodiments thereof with reference to the attached drawings.
FIG. 1 is a flow chart of a method of wheel-based target vehicle attitude measurement;
FIG. 2 is a coordinate and size diagram of a two-dimensional frame of a vehicle target in an image;
FIG. 3 is a flow chart for obtaining wheel feature points based on a segmentation map;
FIG. 4 is a graph of wheel feature point results obtained based on a segmentation map;
FIG. 5 is a flow chart of the conversion of original coordinates to world coordinates;
FIG. 6 is a perspective transformation relationship diagram from original to bird's eye view;
FIG. 7 is an original view and a perspective transformed bird's eye view;
FIG. 8 is an attitude angle of a vehicle;
fig. 9 is a diagram of the actual target vehicle posture measurement result.
Detailed Description
The present invention will be described in further detail with reference to the drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the substances, and not restrictive of the invention. It should be further noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
In addition, the embodiments of the present invention and the features of the embodiments may be combined with each other without collision. The present invention will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Unless otherwise indicated, the exemplary implementations/embodiments shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure may be practiced. Thus, unless otherwise indicated, features of the various implementations/embodiments may be additionally combined, separated, interchanged, and/or rearranged without departing from the technical concepts of the present disclosure.
In the description of the present specification, reference to the terms "one embodiment/manner," "some embodiments/manner," "example," "a particular example," "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment/manner or example is included in at least one embodiment/manner or example of the application. In this specification, the schematic representations of the above terms are not necessarily for the same embodiment/manner or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments/modes or examples. Furthermore, the various embodiments/modes or examples described in this specification and the features of the various embodiments/modes or examples can be combined and combined by persons skilled in the art without contradiction.
Aiming at the problems that in the prior art, a method for predicting through a neural network is adopted, a large amount of vehicle attitude data is required to be collected, the real-time performance is poor, and a method for estimating a historical track is difficult to realize in a low-speed and congestion state, the invention provides a novel method for measuring and calculating the attitude of a target vehicle based on wheels. The flow chart of the target vehicle attitude measuring and calculating method based on the wheels is shown in fig. 1, and mainly comprises the following steps of;
Step 1, acquiring a camera image;
step 2, acquiring the image, detecting a vehicle target from the image, and detecting a vehicle-related wheel;
step 3, selecting the same-side wheel characteristic points according to the integrity degree of the wheels;
step 4, converting the wheel characteristic points into a world coordinate system by using camera external parameters;
Step 5, calculating the vehicle posture by utilizing the world coordinates of the wheel characteristic points;
and 6, reducing measurement errors by using Kalman tracking.
Before the camera image is acquired, camera parameters also need to be calibrated.
1) Calibrating internal and external parameters of a camera: calibrating the internal parameters and the external parameters of the camera to obtain a parameter matrix of the internal parameters of the camera as follows:
where f x,fy denotes the focal length of the camera in the x, y directions, and p x,py denotes the offset of the center point of the camera relative to the upper left corner, all of which can be obtained by calibration methods.
The camera extrinsic matrix is as follows:
Where r ij (i=1, 2,3; j=1, 2, 3) represents the rotation matrix of the camera relative to the world coordinate system, and t i (i=1, 2, 3) represents the translational relationship of the camera coordinate system relative to the world coordinate system.
And the camera internal and external parameter relation is recorded as follows:
2) Vehicle target detection and wheel detection: the camera is a camera arranged at the front end of the vehicle, and is used for acquiring images and can acquire the images, and the images are marked as I; the trained deep learning model is used for target detection, as shown in fig. 2, coordinates and sizes (x, y, w, h) of a two-dimensional frame of the vehicle target V in the image can be obtained, wherein (x, y) represents the left upper corner coordinates of the two-dimensional frame, and w and h are the sizes of the two-dimensional frame.
Within the already acquired target frame, a wheel frame in the image is obtained using the method of wheel detection and extraction, denoted W i(xi,yi,wi,hi), as a basis for subsequent calculations, where i=1, 2..n, N denotes the number of detected wheels, where x i、yi、wi and h i denote the x-coordinate, y-coordinate, wheel frame width and height of the upper left corner of the wheel frame, respectively.
3) Calculating the target attitude of the vehicle:
When the number of wheels N that can be detected by one vehicle target V is not less than 2, the following steps can be used to calculate the attitude of the vehicle:
Checking wheel frames of the left side vehicle and the right side vehicle respectively, and extracting two wheels with the same side;
Selecting the pixel coordinates P i(ui,vi of the feature points of the wheel frame W i detected by the above steps), converting the feature points into a world coordinate system, and obtaining the coordinates in the world coordinate system
Based on the world coordinates of the bottom (or top) of the wheel obtained in the above process, the center coordinates of the two wheels of the vehicle can be obtained, and the inclination posture of the vehicle can be obtained according to the connecting line of the center coordinates.
In addition, for the situation that the real-time transformation is not very strict or the configuration of the vehicle gesture detection system is high and the calculation force is strong, the following formula can be adopted to convert the pixel coordinates of the feature points into the world coordinate system:
Wherein,
Gc＝m11·m22-m12·m21+
(m21·m32-m22·m31)·ui-
(m11·m32-m12·m31)·vi
Wherein m ij (i=1, 2,3; j=1, 2,3, 4) represents an element of a parameter matrix consisting of camera internal and external parameters; u i、vi denotes the x-coordinate and y-coordinate of the wheel-frame feature point numbered i in the image coordinate system; z represents the value of the feature point in the Z axis of the world coordinate system; x i w、yi w represents the x-coordinate value and y-coordinate value of the pixel coordinate in the world coordinate system, respectively.
When the contact point of the wheel bottom and the ground is used as the characteristic point, z=0, at which time the formulas (4) and (5) are simplified to the form of formulas (6) and (7)
When using the midpoint of the wheel top as a feature point, z=d wheel, where D wheel denotes the diameter of the wheel.
The formula conversion method is characterized in that only the coordinate points needing to be converted are needed to be calculated, but the calculation process is complex, and the method can be directly used when the system configuration is high and the calculation force is strong; for the embedded platform in automatic driving, because the computing power is limited and the real-time requirement is high, the direct perspective transformation method is used for carrying out coordinate transformation, at the moment, the world coordinate value corresponding to each pixel point in the image is only needed to be obtained in advance once, and the coordinate transformation is carried out directly when the coordinate transformation is carried out subsequently, so that the method has high transformation speed and is suitable for the embedded real-time computing platform. The direct perspective transformation method is adopted to carry out coordinate transformation, so that the system calculation consumption is greatly reduced, the calculation accuracy can be ensured, and the method is suitable for both detection in a medium-high speed state and detection in a low-speed state, and is a simple and rapid calculation method with wide application range.
3.1 Method for detecting wheels on the same side
If the number of wheels N >2 detected by the same vehicle target, the wheels belonging to the same side of the vehicle need to be proposed.
If the target vehicle is a vehicle with more than 2 wheels on one side, when the same-side vehicle is detected, detecting all N w wheels on the same side, selecting N w characteristic point coordinates of the wheels on the same side of the vehicle, wherein N w is more than 2, acquiring a vehicle linear equation y w＝kwxw+bw by a fitting method, then selecting x coordinates of two coordinates with maximum and minimum transverse coordinates in N w characteristic points in N w wheels, respectively marking the x coordinates as x wmax and x wmin, substituting the x coordinates into the vehicle linear equation y w＝kwxw+bw to obtain two corresponding y coordinates, respectively marking the y wmax and y wmin, and finally calculating the vehicle characteristic point coordinates of the vehicle pose.
As shown in fig. 2, it is first determined whether the vehicle target is traveling on the LEFT side (LEFT in the figure) or the RIGHT side (RIGHT in the figure) based on the position where the vehicle target appears in the visual field; if traveling on the LEFT (LEFT), then the two lateral vehicles of the target vehicle that are visible in the field of view should be located to the right of the target frame; conversely, if the target vehicle is traveling on the RIGHT (RIGHT), the wheels that can be seen should be on the left side of the target frame.
3.2 -Feature point pixel coordinates P i(ui,vi)
In general, if all the detected wheels are in the field of view, the contact point between the bottom of the wheel and the ground is first used as the characteristic point to record
In practice, the wheels may be incomplete in the image when the vehicle just enters the view or disappears from the view, and in the case that the wheels are incomplete, the midpoint of the bottom of the wheels cannot be obtained, and at this time, the midpoint of the top of the wheels is taken as the wheel characteristic point P i(ui,vi).
In this case, when the point on the calculated image is down-converted to the world coordinate system, the point (a, B, C, D) in the world coordinate system taken during the calculation of the perspective transformation matrix is not on the ground plane, but is parallel to the ground plane Z w＝Dwheel, where D wheel represents the diameter of the wheel.
The judging method for the completeness of the wheel comprises the following steps:
judging whether the wheel image in the wheel frame W i detected by the steps is complete, wherein the method comprises the following steps:
if the difference δy=abs (y bi-yb) between the bottom ordinate y bi＝yi+hi of the wheel frame W i and the two-dimensional frame bottom y b =y+h of the target V where it is located is smaller than the acceptable threshold T, and the difference between y b and the bottom edge of the image is also smaller than the threshold T, it can be considered that the wheel is incomplete at this time, and the general T is preferably 5px.
3.3 Conversion of original image coordinates into world coordinates)
Fig. 5 is a flowchart of the conversion from original image coordinates to world coordinates, wherein the conversion uses a direct perspective conversion method to perform coordinate conversion, and only the world coordinate value corresponding to each pixel point in an image is required to be obtained in advance once, and the coordinate conversion is performed directly by a coordinate mapping interface, so that the conversion speed is high, and the method is suitable for an embedded real-time computing platform. The specific process is as follows:
3.3.1 Rectangular range consisting of 4 points (A, B, C, D) under a fixed world coordinate system, and can be used as a bird's eye view representation area; the perspective transformation relationship is shown in fig. 6;
3.3.2 Using the camera external and internal matrices, converting 4 points in the world coordinate system into the image coordinate system
Here, [ Xw s Yws Zws ] (s=a, B, C, D) represents the coordinates of 4 points A, B, C, D in the world coordinate system, zc represents the value of the corresponding point in the Z-axis direction of the coordinates of the camera coordinate system, and [ u s,vs ] (s=a, B, C, D) is the obtained coordinates of the corresponding point in the image coordinate system. Generally, Z ws is denoted as 0 herein, since the wheel is traveling on the ground.
3.3.3 Taking 4 vertexes of the aerial view as perspective transformation reference points, combining the perspective transformation reference points with 4 points under the original image coordinate system obtained in the previous step, and obtaining a perspective transformation matrix Warp from the original image to the aerial view
3.3.4 According to the scale of the four points of the world coordinate system and the scale relation of the aerial view, calculating the proportional relation G w between the pixel value in the aerial view and the coordinates of the world coordinate system; the effect after conversion is shown in fig. 7, where fig. 7 (a) is an original view and fig. 7 (b) is a bird's eye view.
3.3.5 The points on the image coordinate system can be transferred to the world coordinate system by the following formula
Wherein Z t is not practically significant in the formula.
The calculation steps of the transformation matrix Warp are as follows:
Step 1), taking 4 points on the ground in a world coordinate system As 4 vertices in the bird's eye view field, where i=1, 2,3,4 denote the indices of the points,Is a coordinate value in a world coordinate system; in the world coordinate system, the 4 points form a rectangle, the sides of the rectangle are parallel to the coordinate axes on the world coordinate system, and in the aerial view, the coordinates of the 4 vertexes are respectivelyW img、Himg are coordinate values on a world coordinate system;
step 2), according to the internal parameters and external parameters of the camera, the 4 points are processed Projected onto image coordinates, expressed asWhere i=1, 2,3,4 denotes the index of the point,Coordinate values in an image coordinate system;
Step 3), passing through the above points And (3) withThe corresponding relation of (2) constitutes the following equation set:
and solving a homography matrix Warp between the aerial view and the original image.
Calculating a vehicle attitude angle:
the world coordinates of the two wheel characteristic points calculated through the steps are Where Z w is the actual height of the calculated wheel feature. The attitude angle of the vehicle can be calculated as:
as shown in fig. 8, the coordinates of the vehicle center position are P c(xc,yc), wherein W, L are the track and wheel base of the target vehicle, respectively.
4) Tracking the coordinates and the gesture of the vehicle by using the motion model, and obtaining stable measurement results
For example, the CTRV motion model can be used for tracking and filtering the coordinates and the gesture of the vehicle, the coordinate value and the vehicle gesture value calculated by each frame of image are taken as observed quantity, and a Kalman tracking method is used for prediction.
Fig. 9 is a graph of the calculation result of the target vehicle calculated by the method proposed by the patent, and the top view on the right side can see that the target posture obtained by the method is consistent with the actual posture of the vehicle, so that the purpose of calculating the target vehicle posture can be achieved.
Therefore, the invention can rapidly judge the rotation angle posture of the vehicle without marking the vehicle posture, and has good performance under low-speed congestion road conditions; the method overcomes the defect that the real-time performance cannot be ensured because a large amount of vehicle attitude data are required to be calibrated when the traditional method is used for predicting through the neural network, and solves the problem that the traditional historical track calculation method is only suitable for being applied in a medium-high speed state and cannot be applied in a low-speed state; according to the invention, the wheel is used as a main basis for posture judgment, and posture estimation is carried out under a bird's eye view, so that the accuracy of posture calculation is improved, and the real-time accurate posture detection can be carried out on the target in a larger range; the method is simple and feasible, has high measuring and calculating precision, and is suitable for practical application in automatic driving or auxiliary driving.
While the present invention has been described with reference to the preferred embodiments shown in the drawings, it will be understood by those skilled in the art that the above embodiments are for clarity of illustration only and not intended to limit the scope of the invention, which is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such modifications and substitutions will fall within the scope of the present invention.
Claims (29)
1. A method for detecting a posture of a target vehicle based on wheels, comprising:
step 1, acquiring a current camera image;
Step 2, inputting the obtained current camera image, and detecting a vehicle target on the input image to obtain the vehicle target;
step 3, detecting the wheels of the vehicle target to obtain the wheel target;
step 4, extracting characteristic points of two wheels on the same side in the wheel target;
step 5, obtaining the inclination posture of the vehicle by utilizing the connecting line of the coordinates of the characteristic points of the wheels on the two same sides;
firstly, dividing a vehicle target in a camera view into a left target and a right target, then respectively detecting wheels of a left vehicle and a right vehicle, and extracting two required wheel frames with the same side;
Extracting the detected feature point pixel coordinates P i(ui,vi) of the wheel frame W i(xi,yi,wi,hi), converting the feature point pixel coordinates into a world coordinate system, and acquiring coordinates in the world coordinate system Wherein wheel detection is performed within the target frame that has been acquired, a wheel frame in the image is obtained, denoted W i(xi,yi,wi,hi), wherein i=1, 2..n, N represents the number of detected wheels, wherein x i、yi、wi and h i represent the x-coordinate, y-coordinate, wheel frame width and height, respectively, of the upper left corner of the wheel frame;
Obtaining the coordinates of the top, the center or the bottom of two wheels of the vehicle based on the obtained world coordinates of the wheels, and obtaining the inclination angle posture of the vehicle according to the connecting line of the coordinates of the top, the center or the bottom of the two wheels;
The feature point detection of the wheels uses a segmentation map of a target vehicle frame and a corresponding vehicle, wherein the segmentation map is obtained based on a deep neural network segmentation model;
the detection of the characteristic points of the wheels of the target vehicle comprises the following steps:
Step 1), according to the position of the target vehicle in the current image, the method is divided into two cases of left side and right side: if the abscissa x+w/2 of the center point of the target vehicle frame is on the left side of the image, the target vehicle is considered to be on the left side of the current vehicle, and if x+w/2 is on the right side of the image, the target vehicle is considered to be on the right side of the current vehicle;
step 2), when the target vehicle is on the right side of the current vehicle, firstly searching and finding the coordinates of a first row of vehicle pixel points in a target block image area from bottom to top based on the vehicle pixels in the segmentation map to serve as a first vehicle wheel reference coordinate P w1, then counting a first proportion r wpix1 of the number of the vehicle pixels in the whole neighborhood delta in the neighborhood delta of the coordinates, if r wpix1 is larger than a first threshold T wpix1, considering the coordinates to be valid, otherwise, recognizing the coordinates to be invalid, and continuing to search upwards;
Step 3), taking the wheel reference coordinate P w1 meeting the conditions in the step as a starting point, taking a point on the right boundary of a vehicle target frame as an end point P wr (x+w, y+h-s), forming a scanning line, wherein s is the scanning times, 0< s < h, the scanning sequence is that s increases from 1, then counting a second proportion r spix of the whole scanning line occupied by the vehicle pixel point on the scanning line, if r spix is larger than a second threshold T spix, the line is considered to be an effective vehicle scanning line, then finding a point closest to the end point on the scanning line as a second wheel reference coordinate P w2, then counting a third proportion r wpix2 of the vehicle pixel number in the whole neighborhood delta in the neighborhood delta of the coordinate, if r wpix2 is larger than a third threshold T wpix2, otherwise, recognizing the coordinate to be invalid, and continuing scanning;
Step 4), if the effective coordinates are found after the scanning in the step is finished, taking P w1 and P w2 as vehicle feature points;
Step 5), when the target vehicle is on the left side of the current vehicle, the coordinates of the end point P wr are expressed as (x, y+h-s) in the step 3), and the rest steps are the same.
2. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
Before the current camera image is acquired, calibrating the camera, including calibrating internal parameters and external parameters of the camera;
The camera intrinsic parameter matrix is shown in formula (1):
Wherein f x,fy represents the focal length of the camera in the X, Y coordinate direction, and p x,py represents the offset of the center point of the camera relative to the upper left corner;
the camera external parameter matrix is shown in formula (2):
Where r ij (i=1, 2,3; j=1, 2, 3) represents an element of a rotation matrix of the camera with respect to the world coordinate system, and t i (i=1, 2, 3) represents a translational relationship of the camera coordinate system with respect to the world coordinate system;
And, the relation matrix EM of the camera internal parameters and external parameters is shown in formula (3):
Where m ij (i=1, 2,3; j=1, 2,3, 4) represents an element of a parameter matrix consisting of camera internal and external parameters.
3. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
The camera includes an RGB camera, an infrared camera, or a depth camera.
4. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
The camera is a camera head arranged at the front end of the vehicle.
5. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
And detecting the vehicle target by adopting the trained deep learning model to obtain coordinates and dimensions (x, y, w, h) of a two-dimensional frame of the vehicle target V, namely the target frame in the image, wherein (x, y) represents the left upper corner coordinates of the two-dimensional frame, and w and h are the width and height of the two-dimensional frame.
6. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
And (3) detecting the wheels in the target frame of the vehicle by using the trained deep neural network model, wherein the detection result is the coordinates and the dimension W i(xi,yi,wi,hi of the wheels in the image.
7. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
When the image resolution is 640×480, the size of the neighborhood δ is a 5×5 rectangular neighborhood, the value of the first threshold T wpix1 is 0.2, the value of the second threshold T spix is 0.2, and the value of the third threshold T wpix2 is 0.2.
8. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
Detecting the visual proportion or shielding proportion of the wheels by adopting a deep learning method;
If the visual proportion is greater than a fourth threshold T vis, the wheels are considered to be complete in the image;
If the occlusion ratio is less than the fifth threshold T hid, the wheel is considered to appear complete in the image.
9. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
If the difference δy=abs (y bi-yb) between the detected bottom ordinate y bi＝yi+hi of the wheel frame W i(xi,yi,wi,hi) and the two-dimensional bottom ordinate y b =y+h of the vehicle target V is smaller than the sixth threshold T, and the difference between y b and the bottom edge of the image is also smaller than the sixth threshold T, then the wheel is considered to be incomplete in the image.
10. The wheel-based target vehicle posture detection method of claim 9, characterized in that,
The sixth threshold T takes a value of 5px at an original image resolution of 640 x 480.
11. The wheel-based target vehicle posture detection method according to claim 8 or 9, characterized in that,
When the wheels appear in the image completely, using the contact point between the bottom of the wheels and the ground as a characteristic point; when the wheel appears incomplete in the image, the midpoint of the wheel top is used as a feature point.
12. The wheel-based target vehicle posture detection method according to claim 1 or 6, characterized in that,
If the target vehicle is a vehicle with more than 2 wheels on one side, when the same-side vehicle is detected, detecting all N w wheels on the same side, selecting N w characteristic point coordinates of the wheels on the same side of the vehicle, wherein N w is more than 2, acquiring a vehicle linear equation y w＝kwxw+bw by a fitting method, then selecting x coordinates of two coordinates with maximum and minimum transverse coordinates in N w characteristic points in N w wheels, respectively marking the x coordinates as x wmax and x wmin, substituting the x coordinates into the vehicle linear equation y w＝kwxw+bw to obtain two corresponding y coordinates, respectively marking the y wmax and y wmin, and finally calculating the vehicle characteristic point coordinates of the vehicle pose.
13. The wheel-based target vehicle posture detection method according to claim 1 or 6, characterized in that,
The target vehicle type is a truck, a large truck or an engineering operation vehicle.
14. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
And converting the pixel coordinates of the characteristic points into a world coordinate system by adopting a perspective transformation matrix.
15. The wheel-based target vehicle posture detection method of claim 1, characterized in that,
Converting the feature point pixel coordinates into a world coordinate system using equations (4) and (5)
Wherein,
Wherein m ij (i=1, 2,3; j=1, 2,3, 4) represents an element of a parameter matrix consisting of camera internal and external parameters; u i、vi denotes the x-coordinate and y-coordinate of the wheel-frame feature point numbered i in the image coordinate system; z represents the coordinate value of the Z axis of the feature point in the world coordinate system; x i w、yi w represents the coordinate value of the X-axis and the coordinate value of the Y-axis of the pixel coordinate in the world coordinate system, respectively.
16. The wheel-based target vehicle posture detection method of claim 15, characterized in that,
When the contact point of the wheel bottom and the ground is used as the characteristic point, z=0, at which time the formulas (4) and (5) are simplified to the form of formulas (6) and (7)
17. The wheel-based target vehicle posture detection method of claim 15, characterized in that,
When using the midpoint of the wheel top as a feature point, z=d wheel, where D wheel denotes the diameter of the wheel.
18. The wheel-based target vehicle posture detection method according to claim 1 or 6, characterized in that,
Judging whether the vehicle target is traveling on the LEFT side (LEFT) or the RIGHT side (RIGHT) according to the position of the vehicle target in the visual field; if traveling on the LEFT side (LEFT), the two side wheels of the target vehicle seen in the field of view are located on the right side of the target frame; conversely, if the target vehicle is traveling on the RIGHT (RIGHT), then the two side wheels of the target vehicle are seen to be on the left side of the target frame.
19. The wheel-based target vehicle posture detection method according to claim 1 or 6, characterized in that,
For the feature point pixel coordinates P i(ui,vi), if all the detected wheels are entirely present in the field of view, the contact point between the bottom of the wheel and the ground is used as the feature point, which is noted as formula (8)
20. The wheel-based target vehicle posture detection method according to claim 1 or 6, characterized in that,
For the feature point pixel coordinates P i(ui,vi), if it is detected that the wheel is incomplete in the image, the midpoint of the wheel top is used as the wheel feature point and is recorded as formula (9)
In this case, when the point on the calculated image is down-converted to the world coordinate system, the point (A, B, C, D) in the world coordinate system taken during the calculation of the perspective transformation matrix is not on the ground plane, but is parallel to the ground plane z=d wheel, where D wheel represents the diameter of the wheel.
21. The wheel-based target vehicle posture detection method according to claim 1 or 6, characterized in that,
Converting the feature point pixel coordinates to a world coordinate system, comprising:
4 points under a fixed world coordinate system are marked as points A, B, C, D, and a rectangular range formed by the points is used as a bird's eye view diagram representable area;
converting the 4 points under the world coordinate system into the image coordinate system by using the camera external reference matrix and the internal reference matrix as shown in formula (10)
Here, [ Xw s Yws Zws ], (s=a, B, C, D) represents the coordinates of 4 points A, B, C, D in the world coordinate system, zc represents the value of the corresponding point in the Z-axis direction of the coordinates of the camera coordinate system, [ u s,vs ], (s=a, B, C, D) is the obtained coordinates of the corresponding point in the image coordinate system.
22. The wheel-based target vehicle posture detection method of claim 21, characterized in that,
Taking 4 vertexes of the aerial view as perspective transformation reference points, and combining the vertexes with 4 points under an original image coordinate system to obtain a perspective transformation matrix Warp from the original image to the aerial view;
Calculating a proportional relation G w between pixel values in the aerial view and coordinates of the world coordinate system according to the scale of 4 points in the world coordinate system and the scale relation of the aerial view;
The points on the image coordinate system are converted to the world coordinate system by equation (11)
Wherein Z t is not practically significant in the formula; x i w、yi w represents the x-coordinate value and y-coordinate value of the pixel coordinate in the world coordinate system, respectively.
23. The wheel-based target vehicle posture detection method of claim 22, characterized in that,
The calculation steps of the transformation matrix Warp are as follows:
Step 1), taking 4 points on the ground in a world coordinate system As 4 vertices in the bird's eye view field, where i=1, 2,3,4 denote the indices of the points,Is a coordinate value in a world coordinate system; in the world coordinate system, the 4 points form a rectangle, the sides of the rectangle are parallel to the coordinate axes on the world coordinate system, and in the aerial view, the coordinates of the 4 vertexes are respectivelyW img、Himg are coordinate values on a world coordinate system;
step 2), according to the internal parameters and external parameters of the camera, the 4 points are processed Projected onto image coordinates, expressed asWhere i=1, 2,3,4 denotes the index of the point,Coordinate values in an image coordinate system;
Step 3), passing through the above points And (3) withThe corresponding relation of (2) constitutes the following equation set:
and solving a homography matrix Warp between the aerial view and the original image.
24. The wheel-based target vehicle posture detection method of claim 23, characterized in that,
If the characteristic point of the wheel is the contact position of the wheel and the ground, thenIf the characteristic point of the wheel is the wheel center, thenIf the characteristic point of the wheel is the top of the wheel, then
The diameter D wheel of the wheel is obtained based on the trained deep neural network.
25. The wheel-based target vehicle posture detection method of claim 24, characterized in that,
The actual height Z w of the wheel feature at this time is noted as 0 when the wheel is traveling on the ground.
26. The wheel-based target vehicle posture detection method of claim 25, characterized in that,
The world coordinates of two wheel characteristic points are recorded asWherein Z w represents the actual height of the wheel feature; the attitude angle of the vehicle is shown as formula (12)
27. The wheel-based target vehicle posture detection method of any one of claims 22-25, characterized in that,
The motion model is used to track vehicle coordinates and pose.
28. The wheel-based target vehicle posture detection method of any one of claims 22-25, characterized in that,
And tracking and filtering the coordinates and the gesture of the vehicle by using a CTRV motion model, taking the coordinate value and the vehicle gesture value calculated by each frame of image as observed quantity, and predicting by using a Kalman tracking method.
29. A wheel-based target vehicle posture detection apparatus, characterized by comprising:
an image acquisition device that acquires a current camera image;
the vehicle target detection device inputs the acquired current camera image, and carries out vehicle target detection on the input image to obtain a vehicle target;
the wheel target detection device is used for detecting the wheels of the vehicle target to obtain the wheel target;
the same-side wheel extraction device is used for extracting characteristic points of two wheels on the same side in a wheel target;
the inclination angle posture calculation device is used for obtaining the inclination angle posture of the vehicle by utilizing the connecting line of the coordinates of the characteristic points of the wheels on the two same sides;
the method comprises the steps of firstly dividing a vehicle target in a camera view into a left target and a right target, then respectively detecting wheels of a left vehicle and a right vehicle, and extracting two required wheel frames with the same side;
Extracting the detected feature point pixel coordinates P i(ui,vi) of the wheel frame W i(xi,yi,wi,hi), converting the feature point pixel coordinates into a world coordinate system, and acquiring coordinates in the world coordinate system Wherein wheel detection is performed within the target frame that has been acquired, a wheel frame in the image is obtained, denoted W i(xi,yi,wi,hi), wherein i=1, 2..n, N represents the number of detected wheels, wherein x i、yi、wi and h i represent the x-coordinate, y-coordinate, wheel frame width and height, respectively, of the upper left corner of the wheel frame;
Obtaining the coordinates of the top, the center or the bottom of two wheels of the vehicle based on the obtained world coordinates of the wheels, and obtaining the inclination angle posture of the vehicle according to the connecting line of the coordinates of the top, the center or the bottom of the two wheels;
The feature point detection of the wheels uses a segmentation map of a target vehicle frame and a corresponding vehicle, wherein the segmentation map is obtained based on a deep neural network segmentation model;
the detection of the characteristic points of the wheels of the target vehicle comprises the following steps:
Step 1), according to the position of the target vehicle in the current image, the method is divided into two cases of left side and right side: if the abscissa x+w/2 of the center point of the target vehicle frame is on the left side of the image, the target vehicle is considered to be on the left side of the current vehicle, and if x+w/2 is on the right side of the image, the target vehicle is considered to be on the right side of the current vehicle;
step 2), when the target vehicle is on the right side of the current vehicle, firstly searching and finding the coordinates of a first row of vehicle pixel points in a target block image area from bottom to top based on the vehicle pixels in the segmentation map to serve as a first vehicle wheel reference coordinate P w1, then counting a first proportion r wpix1 of the number of the vehicle pixels in the whole neighborhood delta in the neighborhood delta of the coordinates, if r wpix1 is larger than a first threshold T wpix1, considering the coordinates to be valid, otherwise, recognizing the coordinates to be invalid, and continuing to search upwards;
Step 3), taking the wheel reference coordinate P w1 meeting the conditions in the step as a starting point, taking a point on the right boundary of a vehicle target frame as an end point P wr (x+w, y+h-s), forming a scanning line, wherein s is the scanning times, 0< s < h, the scanning sequence is that s increases from 1, then counting a second proportion r spix of the whole scanning line occupied by the vehicle pixel point on the scanning line, if r spix is larger than a second threshold T spix, the line is considered to be an effective vehicle scanning line, then finding a point closest to the end point on the scanning line as a second wheel reference coordinate P w2, then counting a third proportion r wpix2 of the vehicle pixel number in the whole neighborhood delta in the neighborhood delta of the coordinate, if r wpix2 is larger than a third threshold T wpix2, otherwise, recognizing the coordinate to be invalid, and continuing scanning;
Step 4), if the effective coordinates are found after the scanning in the step is finished, taking P w1 and P w2 as vehicle feature points;
Step 5), when the target vehicle is on the left side of the current vehicle, the coordinates of the end point P wr are expressed as (x, y+h-s) in the step 3), and the rest steps are the same.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010784979.1A CN111967360B (en) | 2020-08-06 | 2020-08-06 | Target vehicle posture detection method based on wheels |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010784979.1A CN111967360B (en) | 2020-08-06 | 2020-08-06 | Target vehicle posture detection method based on wheels |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111967360A CN111967360A (en) | 2020-11-20 |
| CN111967360B true CN111967360B (en) | 2024-09-06 |
Family
ID=73364687
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010784979.1A Active CN111967360B (en) | 2020-08-06 | 2020-08-06 | Target vehicle posture detection method based on wheels |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111967360B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112785653B (en) * | 2020-12-30 | 2024-06-21 | 中山联合汽车技术有限公司 | Vehicle-mounted camera attitude angle calibration method |
| CN112802341B (en) * | 2020-12-30 | 2022-07-01 | 中山联合汽车技术有限公司 | Method for measuring position and attitude of front vehicle |
| CN112800873A (en) * | 2021-01-14 | 2021-05-14 | 知行汽车科技（苏州）有限公司 | Method, device and system for determining target direction angle and storage medium |
| CN113065393B (en) * | 2021-02-25 | 2024-04-02 | 惠州华阳通用电子有限公司 | Blind area monitoring method based on rearview camera |
| CN113267137B (en) * | 2021-05-28 | 2023-02-03 | 北京易航远智科技有限公司 | Real-time measurement method and device for tire deformation |
| JP2023017341A (en) * | 2021-07-26 | 2023-02-07 | トヨタ自動車株式会社 | Model production method, model production device, model production program, moving body posture estimation method, and moving body posture estimation device |
| CN113658449B (en) * | 2021-07-27 | 2022-09-23 | 上海交通大学 | Parking lot vehicle positioning system and method |
| CN117252880B (en) * | 2023-11-17 | 2024-01-16 | 宝德华南(深圳)热能系统有限公司 | Adas comprehensive evaluation system of new energy automobile |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109584300A (en) * | 2018-11-20 | 2019-04-05 | 浙江大华技术股份有限公司 | A kind of method and device of determining headstock towards angle |
| CN110065494A (en) * | 2019-04-09 | 2019-07-30 | 魔视智能科技(上海)有限公司 | A kind of vehicle collision avoidance method based on wheel detection |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004227293A (en) * | 2003-01-23 | 2004-08-12 | Nissan Motor Co Ltd | Side vehicle detector |
| DE102013022076A1 (en) * | 2013-12-23 | 2015-06-25 | Valeo Schalter Und Sensoren Gmbh | Method for determining a width of a target vehicle by means of a camera system of a motor vehicle, camera system and motor vehicle |
| CN110246183B (en) * | 2019-06-24 | 2022-07-15 | 百度在线网络技术（北京）有限公司 | Wheel grounding point detection method, device and storage medium |
| CN110555402B (en) * | 2019-08-27 | 2024-10-29 | 纵目科技（上海）股份有限公司 | Congestion car following method, system, terminal and storage medium based on look-around |
| CN110648360B (en) * | 2019-09-30 | 2023-08-01 | 的卢技术有限公司 | Method and system for avoiding other vehicles based on vehicle-mounted camera |
| CN110738181B (en) * | 2019-10-21 | 2022-08-05 | 东软睿驰汽车技术（沈阳）有限公司 | Method and device for determining vehicle orientation information |
-
2020
- 2020-08-06 CN CN202010784979.1A patent/CN111967360B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109584300A (en) * | 2018-11-20 | 2019-04-05 | 浙江大华技术股份有限公司 | A kind of method and device of determining headstock towards angle |
| CN110065494A (en) * | 2019-04-09 | 2019-07-30 | 魔视智能科技(上海)有限公司 | A kind of vehicle collision avoidance method based on wheel detection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111967360A (en) | 2020-11-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111967360B (en) | Target vehicle posture detection method based on wheels | |
| CN110745140B (en) | Vehicle lane change early warning method based on continuous image constraint pose estimation | |
| EP2209091B1 (en) | System and method for object motion detection based on multiple 3D warping and vehicle equipped with such system | |
| CN111553252B (en) | Road pedestrian automatic identification and positioning method based on deep learning and U-V parallax algorithm | |
| CN108256413B (en) | Passable area detection method and device, storage medium and electronic equipment | |
| EP2372304B1 (en) | Vehicle position recognition system | |
| US10424081B2 (en) | Method and apparatus for calibrating a camera system of a motor vehicle | |
| CN105300403B (en) | A kind of vehicle mileage calculating method based on binocular vision | |
| JP5023186B2 (en) | Object motion detection system based on combination of 3D warping technique and proper object motion (POM) detection | |
| CN110969064B (en) | Image detection method and device based on monocular vision and storage equipment | |
| JP2006053757A (en) | Plane detector and detection method | |
| CN104204726A (en) | Moving-object position/attitude estimation apparatus and method for estimating position/attitude of moving object | |
| CN113421289A (en) | High-precision vehicle track data extraction method for overcoming unmanned aerial vehicle shooting disturbance | |
| US11904843B2 (en) | Autonomous parking systems and methods for vehicles | |
| CN112489106A (en) | Video-based vehicle size measuring method and device, terminal and storage medium | |
| Zhang et al. | Deep learning based object distance measurement method for binocular stereo vision blind area | |
| CN107220632B (en) | Road surface image segmentation method based on normal characteristic | |
| CN111256651B (en) | Week vehicle distance measuring method and device based on monocular vehicle-mounted camera | |
| JP2006053754A (en) | Plane detection apparatus and detection method | |
| US11461944B2 (en) | Region clipping method and recording medium storing region clipping program | |
| CN107248171B (en) | Triangulation-based monocular vision odometer scale recovery method | |
| JP4106163B2 (en) | Obstacle detection apparatus and method | |
| JP2006053755A (en) | Device for calculating moving quantity of mobile body | |
| CN116311136A (en) | Lane line parameter calculation method for driving assistance | |
| CN112634354B (en) | Road side sensor-based networking automatic driving risk assessment method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |