CN115908913A - RGBD-based parcel category detection method and electronic equipment - Google Patents
RGBD-based parcel category detection method and electronic equipment Download PDFInfo
- Publication number
- CN115908913A CN115908913A CN202211432120.XA CN202211432120A CN115908913A CN 115908913 A CN115908913 A CN 115908913A CN 202211432120 A CN202211432120 A CN 202211432120A CN 115908913 A CN115908913 A CN 115908913A
- Authority
- CN
- China
- Prior art keywords
- parcel
- package
- rgbd
- determining
- category
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of image recognition, and particularly discloses a package type detection method based on RGBD and an electronic device, wherein the package type detection method comprises the following steps: acquiring RGBD image acquisition data; determining the position state of a parcel according to RGB image information and depth information in the RGBD image acquisition data, wherein the position state of the parcel comprises a stacked parcel and a non-stacked parcel; when the position state of the parcel is a single parcel in the non-stacked parcels, extracting a parcel area image of RGB image information where the single parcel is located; and determining the parcel category of the single parcel according to the parcel area image and the depth information corresponding to the single parcel, and outputting the parcel category. The RGBD-based package type detection method and the electronic equipment have the advantages of low cost and high package type detection precision.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to a package type detection method based on RGBD and electronic equipment.
Background
The modern logistics industry generally carries out classification treatment according to large pieces and small pieces, wherein the small pieces are generally conveyed by adopting advanced high-efficiency sorting equipment; the large pieces comprise single large pieces and transfer mailbag total packages, and because the large pieces are heavy, the large pieces are generally sorted by adopting sorting equipment with lower conveying speed, or the packages are classified by volume measuring equipment according to the volume of the packages or whether the packages are regular cubes.
At present, the traditional parcel classification processing method is that parcels are firstly transmitted to a workshop after being unloaded manually, and then manual classification processing is carried out, so that the processing procedure is complex, the times of reversing hands are increased, and the labor cost is high; or intelligent equipment is adopted for classification, and the classification failure ratio is high due to the fact that the materials of the package shape are complex, the equipment cost is high, and the occupied area is large.
Therefore, how to provide a method with low cost and high class detection precision becomes a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The invention provides a package type detection method based on RGBD and electronic equipment, which solve the problems of low detection precision and high cost in the related technology.
As a first aspect of the present invention, there is provided an RGBD-based package category detection method, including:
acquiring RGBD image acquisition data;
determining the position state of a parcel according to RGB image information and depth information in the RGBD image acquisition data, wherein the position state of the parcel comprises a stacked parcel and a non-stacked parcel;
when the position state of the parcel is a single parcel in the non-stacked parcels, extracting a parcel area image of RGB image information where the single parcel is located;
and determining the parcel category of the single parcel according to the parcel area image and the depth information corresponding to the single parcel, and outputting the parcel category.
Further, determining the position state of the parcel according to the RGB image information and the depth information in the RGBD image acquisition data includes:
preprocessing RGB image information in the RGBD image acquisition data to obtain a preprocessed image;
positioning the position state of the parcel in the preprocessed image according to a target detection algorithm;
if the position state of the parcel conforms to a first preset category, outputting the position state of the parcel as a non-stacked parcel, wherein the non-stacked parcel comprises a nearby parcel and a single parcel;
and if the position state of the parcel conforms to a second preset category, outputting the position state of the parcel as a stack parcel.
Further, when the position state of the parcel is a non-stack parcel, if the preprocessed image includes a single parcel, determining that the position state of the parcel is a single parcel;
if the preset processing image comprises a plurality of independent packages, determining whether the distance between every two adjacent independent packages is a nearby package according to the depth information of every two adjacent independent packages;
and if the difference between the depth values of every two adjacent independent parcels is smaller than a preset threshold value, determining that the position state of the parcel is a distance-approaching parcel.
Further, determining whether the parcel is a nearby parcel according to the depth information of each two adjacent independent parcels comprises the following steps:
calculating the actual Y-axis physical coordinate of each independent parcel according to the pixel coordinate of the parcel area in the preset processing image and the height value corresponding to the pixel coordinate;
calculating the Y-axis distance between every two adjacent independent parcels according to the actual Y-axis physical coordinates of each independent parcel;
comparing the Y-axis distance between every two adjacent independent parcels with a preset threshold value;
and if the Y-axis distance between every two adjacent independent parcels is smaller than a preset threshold value, determining that the non-stacked parcel is a distance-adjacent parcel.
Further, when the position state of the parcel is a single parcel, extracting a parcel area image of RGB image information where the single parcel is located includes:
extracting a parcel region comprising a single parcel from the RGB image information to obtain an extracted image;
and carrying out image scaling and normalization processing on the extracted image to obtain a parcel area image.
Further, determining the parcel category of the single parcel according to the parcel area image and the depth information corresponding to the single parcel, and outputting the parcel category, includes:
carrying out classification processing on the single parcel in the parcel region image according to an object classification algorithm;
determining a package category according to the classification processing result, wherein the package category comprises a carton, a foam box, a bag collecting bag and a soft bag;
and if the package type is a paper box or a soft package, determining the fine-grained classification of the paper box or the soft package according to the depth information corresponding to the single package, wherein the fine-grained classification of the paper box comprises breakage and normality, and the fine-grained classification of the soft package comprises a film-coated paper box and a soft package bag.
Further, determining a package category according to a result of the classification process includes:
extracting a corresponding parcel picture from the RGBD image acquisition data according to the minimum external matrix of the single parcel;
adjusting the size of the parcel picture, and performing normalization processing on the parcel picture;
and performing package type identification on the package picture after the normalization processing according to an object classification model to obtain a package type identification result.
Further, if the package type is a carton or a soft package, determining the fine-grained classification of the carton or the soft package according to the depth information corresponding to the single package, including:
acquiring the actual coordinates of the single parcel, and determining the depth value of the area where the single parcel is located;
calculating the actual spatial coordinate value of the area according to the depth value of the area where the single parcel is located;
fitting the actual space coordinate value by combining a fitting algorithm to obtain a plane of a fitting package surface and obtain a plane fitting equation formula;
substituting the actual coordinates of the single parcel into the plane fitting equation formula, and calculating the error value of each point of the single parcel from the plane of the surface of the fitted parcel;
marking the points with the error values larger than a preset error threshold value as deviation points, and counting the number of the deviation points;
when the parcel type is a carton, if the ratio of the number of the deviation points to the number of the actual points of the single parcel is greater than a preset ratio threshold value, determining that the single parcel is a damaged carton, otherwise, determining that the single parcel is a normal carton;
and when the package type is soft package, if the ratio of the number of the deviation points to the number of the actual points of the single package is smaller than a preset ratio threshold value, determining that the single package is a film-coated carton, otherwise, determining that the single package is a soft package bag.
Further, when the location status of the package is a stack package, information that does not require category determination of the package is output.
As another aspect of the present invention, an electronic device is provided, which includes a memory and a processor, wherein the memory stores computer program instructions, and the processor is configured to load and execute the computer program instructions to implement the RGBD-based package category detection method described above.
According to the RGBD-based package type detection method provided by the invention, the package type is further determined by adopting RGBD image acquisition data after the position state of the package is determined to be a single package, and the package type and the position state of the package are determined according to the depth information in the RGBD image acquisition data, namely based on the combination of the depth data and the RGB data, so that the package type can be determined more accurately, and the algorithm precision can be ensured and the operation speed of the algorithm can be greatly increased by adopting a cascading algorithm framework when the package position state detection and the package type determination are carried out. In addition, the full automatic detection does not need personnel participation, so the labor cost is reduced. Therefore, the RGBD-based package category detection method provided by the invention has the advantages of high detection precision, low cost and high detection efficiency.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.
Fig. 1 is a flowchart of an RGBD-based package category detection method provided in the present invention.
Fig. 2 is an original drawing of an RGBD image acquired by an RGBD camera provided by the present invention.
FIG. 3 is a flow chart for determining the status of a package location provided by the present invention.
Fig. 4a is a schematic diagram of the present invention providing a proximity package.
Fig. 4b is a schematic view of an individual package provided by the present invention.
Fig. 4c is a schematic view of a stack package provided by the present invention.
FIG. 5 is a schematic diagram of a depth value of a parcel area provided by the present invention.
Fig. 6 is a flow chart of package category determination provided by the present invention.
Fig. 7a is a schematic diagram of the package type provided by the present invention being a soft package.
Fig. 7b is a schematic view of the present invention providing a package of the type foam box.
Fig. 7c is a schematic diagram of the package provided by the present invention being a bag.
Fig. 7d is a schematic illustration of the present invention providing a package category as a carton.
Fig. 8 is a block diagram of an electronic device according to the present invention.
Detailed Description
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged as appropriate in order to facilitate the embodiments of the invention described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In the present embodiment, an RGBD-based package category detection method is provided, and fig. 1 is a flowchart of an RGBD-based package category detection method according to an embodiment of the present invention, as shown in fig. 1, including:
s100, acquiring RGBD image acquisition data;
in the embodiment of the invention, the RGBD images of the parcels on the parcel sorting line are acquired in real time by the RGBD camera to obtain RGBD image acquisition data, as shown in fig. 2.
S200, determining the position states of the parcels according to RGB image information and depth information in the RGBD image acquisition data, wherein the position states of the parcels comprise stacked parcels and non-stacked parcels;
it should be understood that after the RGBD image acquisition data is acquired, the RGBD image acquisition data is preprocessed, and the position of the package in the image is located by using the target detection algorithm to determine the position state of the package.
In the embodiment of the present invention, as shown in fig. 3, the method specifically includes:
s210, preprocessing RGB image information in the RGBD image acquisition data to obtain a preprocessed image;
in some embodiments, the RGB image in the acquired RGBD image acquisition data may be specifically scaled to (416 ) a size (which can meet the requirement of the subsequent target detection algorithm, and of course, other sizes may be set as needed, which is only an example), and then the image is normalized, where the specific normalization formula is as follows:
wherein mean is value Mean, std, of three channels representing an RGB image value Representing the variance of three channels of an RGB image, x represents the pixel value of the image,
represents the normalized pixel value, and in this embodiment, mean is the value determined since the RGB image size is (416 ) value ＝[0.485,0.456,0.406]，std value ＝[0.229,0.224,0.225]。
S220, positioning the position state of the parcel in the preprocessed image according to a target detection algorithm;
in the embodiment of the invention, an improved NanoDet target detection algorithm can be specifically adopted, the backbone network and the characteristic pyramid structure are optimized, the algorithm precision is ensured, the model parameters are reduced, and the algorithm running speed is increased.
For a model trunk part (based on ShuffleNet V2), when the convolution step length is 2, fusing the characteristics of different channel information by adding depth separable convolutions with convolution kernel size of 3 and convolution kernel size of 1; when the convolution step is 1, a Ghost module (more features can be generated by using fewer parameters) is added into the backbone network so as to improve the feature extraction capability of the backbone network.
For the feature pyramid structure part (based on CSP-PAN), in the original CSP-PAN structure, the number of channels of each output feature map is kept the same as the input feature map from the backbone network, and for the edge calculation, the structure with large number of channels has expensive calculation cost, so the embodiment of the present invention uses convolution with convolution kernel size of 1 to make the number of channels of all feature maps equal to the minimum number of channels (i.e. both 96), and realizes the feature fusion of top-down and bottom-up through the CSP structure, and the reduced features make the calculation cost lower and do not lose the model performance.
When the target detection model is obtained, the activation function is replaced from ReLU to H-Swish in the model training process, calculation is faster, and the method is friendly to a mobile terminal. The learning rate mechanism is replaced by a cosine learning rate mechanism (linear step learning rate mechanism), so that the decline of the learning rate is more gradual, and the model training is facilitated. The concrete formula is as follows:
wherein T represents the total number of model training rounds, T represents the current round, L represents the initial set learning rate, and L represents the initial set learning rate t Representing the current learning rate.
S230, if the position state of the parcel conforms to a first preset category, outputting the position state of the parcel as a non-stacked parcel, wherein the non-stacked parcel comprises a close parcel and a single parcel;
it should be understood that, if the position status of the parcel is determined according to the output result of the object detection algorithm, if the output is a non-stack parcel, since the non-stack parcel may include a single parcel (i.e. only one parcel) and a plurality of independent parcels not stacked together, it is necessary to further determine which case the current non-stack parcel is, i.e. a single parcel or a parcel close to the current parcel.
Fig. 4a shows a schematic view of a close-by parcel, fig. 4b shows a schematic view of an individual parcel, and fig. 4c shows a schematic view of a stacked parcel.
In the embodiment of the present invention, when the position status of the parcel is a non-stack parcel, if a single parcel is included in the preprocessed image, it is determined that the position status of the parcel is a single parcel;
if the preset processing image comprises a plurality of independent packages, determining whether the distance between every two adjacent independent packages is a nearby package according to the depth information of every two adjacent independent packages;
and if the difference between the depth values of every two adjacent independent parcels is smaller than a preset threshold value, determining that the position state of the parcel is a distance-approaching parcel.
It should be understood that, when detecting the state of the parcels, the number of parcels in the image and the distance between the parcels need to be determined first. And when the number of the packages is more than 1, the stack judgment logic is required to be entered. The specific coordinates of the parcels in the images are obtained through parcel positioning, but the real distance between the parcels cannot be reflected due to the fact that the parcel distance detected by the RGB images has parallax, so that the depth data of the parcels are obtained through the image depth data extracted by the RGBD camera, and the actual distance between the two parcels is calculated.
In the embodiment of the invention, the step of determining whether the parcel is a nearby parcel according to the depth information of each two adjacent independent parcels comprises the following steps:
calculating the actual Y-axis physical coordinate of each independent parcel according to the pixel coordinate of the parcel area in the preset processing image and the height value corresponding to the pixel coordinate;
calculating the Y-axis distance between every two adjacent independent parcels according to the actual Y-axis physical coordinates of each independent parcel;
comparing the Y-axis distance between every two adjacent independent parcels with a preset threshold value;
and if the Y-axis distance between every two adjacent independent parcels is smaller than a preset threshold value, determining that the non-stacked parcel is a distance-adjacent parcel.
As shown in fig. 4a, the position of two packages is obtained by package positioning, the package above the view is designated as package a, and the package below the view is designated as package b. The Depth value of the parcel area is as shown in fig. 5, the Depth value of the parcel a is marked as Depth _ a, and the Depth value of the parcel b is marked as Depth _ b.
Because the pixel coordinate of the parcel area and the height value corresponding to the coordinate are calculated all the time, the actual physical coordinate Y of the parcel on the Y axis and the actual physical coordinate X of the parcel on the X axis can be calculated according to the following formula.
The pixel position of the RGBD camera is determined according to the position of the main optical axis of the RGBD camera in the Y-axis direction, the position of the pixel point is determined according to the position of the main optical axis of the RGBD camera in the Y-axis direction, and the position of the pixel point is determined according to the position of the pixel point in the Y-axis direction.
The distance between the two parcels can be accurately calculated through the actual Y-axis physical coordinate Y of each position of the parcel area, and then the parcel state of the current frame is obtained.
S240, if the position state of the parcel conforms to a second preset category, outputting the position state of the parcel as a stack parcel.
In the embodiment of the invention, when the position state of the parcel is a stack parcel, information that does not need to perform category determination on the parcel is output.
It will be appreciated that in embodiments of the invention, subsequent package category detection is only performed on individual packages, and information not requiring a package to be category-determined is output directly for both stacked packages and for nearby packages.
And pile up the parcel and close to the parcel apart from and can continue to circulate on the parcel letter sorting line, can pass through next time or circulate for a plurality of times and continue to carry out the classification detection of parcel after becoming individual parcel promptly to can improve the degree of accuracy that the parcel classification detected.
S300, when the position state of the parcel is a single parcel in the non-stacked parcels, extracting a parcel area image of RGB image information where the single parcel is located;
after the location status of the package is determined, if the package does not belong to a close-distance package or a stacked package type, i.e., only if the package status detection output is a single package, the package type detection is performed. Extracting parcel areas of individual parcels from the image; the extracted image of the parcel area is preprocessed, scaled to a fixed size, and then normalized.
Specifically, when the position state of the parcel is a single parcel, extracting a parcel area image of RGB image information where the single parcel is located includes:
extracting a parcel region comprising a single parcel from the RGB image information to obtain an extracted image;
and carrying out image scaling and normalization processing on the extracted image to obtain a parcel area image.
Firstly, detecting the parcel state, and when the state is a single parcel, extracting a parcel picture of the area position from an original image according to the minimum circumscribed matrix of the parcel. Due to the different sizes of the parcel sizes, the parcel pictures need to be fixed to the sizes (224 ) firstly, then the images are normalized, and the specific formula of the normalization processing refers to the normalization processing formula in the foregoing.
S400, determining the parcel category of the single parcel according to the parcel area image and the depth information corresponding to the single parcel, and outputting the parcel category.
In the embodiment of the invention, the packages are classified by an object classification algorithm, and the categories are totally divided into four main categories: carton, collection bag, foam case, soft package, wherein the carton needs to carry out the fine grit classification with soft package classification, and the carton needs to judge whether normal carton or damaged carton, and soft package needs to judge whether normal soft package or tectorial membrane carton.
In the fine particle classification, the depth data is converted into point cloud data. If the package is judged to be a carton, judging whether the carton is damaged or not by fitting the flatness of the surface of the package; if the package is judged to be soft, the package flatness is fitted, and whether the soft package is a film-coated carton or not is judged.
In the embodiment of the present invention, determining the parcel category of the single parcel according to the parcel area image and the depth information corresponding to the single parcel, and outputting the parcel category, as shown in fig. 6, includes:
s410, carrying out classification processing on the single parcel in the parcel region image according to an object classification algorithm;
specifically, the object classification algorithm may specifically adopt a MobileNetV2 classification model. The MobileNet V2 improves the performance of the algorithm and reduces the parameter number by improving the residual error structure. Specifically, when entering the residual module, the feature map is upscaled by 1*1 convolution, and then the operand and parameters are reduced by depth-wise separable convolution (depth-wise separable convolution), and the ReLU activation function after the depth-wise separable convolution is cancelled. Therefore, the embodiment of the invention selects the MobileNetV2 model based on the balance between the accuracy of the algorithm and the operation efficiency.
S420, determining the package types according to the classification processing result, wherein the package types comprise a carton, a foam box, a bag collecting bag and a soft bag; fig. 7a shows a schematic view of the package category being a soft bag, fig. 7b shows a schematic view of the package category being a foam box, fig. 7c shows a schematic view of the package category being a bag, and fig. 7d shows a schematic view of the package category being a carton.
Specifically, extracting a corresponding parcel picture from the RGBD image acquisition data according to the minimum external matrix of the single parcel;
adjusting the size of the parcel picture, and performing normalization processing on the parcel picture;
and performing package type identification on the package picture after the normalization processing according to an object classification model to obtain a package type identification result.
S430, if the package type is a paper box or a soft package, determining the fine-grained classification of the paper box or the soft package according to the depth information corresponding to the single package, wherein the fine-grained classification of the paper box comprises breakage and normality, and the fine-grained classification of the soft package comprises a film-coated paper box and a soft package bag.
Specifically, the actual coordinates of the single parcel are obtained, and the depth value of the area where the single parcel is located is determined;
calculating the actual spatial coordinate value of the area according to the depth value of the area where the single parcel is located;
fitting the actual space coordinate value by combining a fitting algorithm to obtain a plane of a fitting package surface and obtain a plane fitting equation formula;
substituting the actual coordinates of the single parcel into the plane fitting equation formula, and calculating the error value of each point of the single parcel from the plane of the surface of the fitted parcel;
marking the points with the error values larger than a preset error threshold value as deviation points, and counting the number of the deviation points;
when the parcel type is a carton, if the ratio of the number of the deviation points to the number of the actual points of the single parcel is greater than a preset ratio threshold value, determining that the single parcel is a damaged carton, otherwise, determining that the single parcel is a normal carton;
and when the package type is soft package, if the ratio of the number of the deviation points to the number of the actual points of the single package is smaller than a preset ratio threshold value, determining that the single package is a film-coated carton, otherwise, determining that the single package is a soft package bag.
In embodiments of the present invention, further determinations need to be made regarding the identification of the broad classes as carton and soft pack types. Judging whether the surface of the carton is damaged or not according to the type of the carton; for the soft bag type, it is necessary to determine whether the carton is a film-covered carton (a bag wrapped with a soft bag) or not.
During specific implementation, the parcel surface data is fitted by extracting the depth value and utilizing an algorithm of a PCL (Point Cloud Library) Point Cloud Library to obtain a fitting plane, and the error between the actual surface depth value and the plane is calculated, so that whether the parcel plane is flat or not is judged. The method comprises the following specific steps:
(1) Acquiring the actual coordinate of the parcel position to obtain the depth value of the area;
(2) Calculating the actual (X, Y, z) value of the parcel area by the depth value, wherein the specific conversion formula can refer to the calculated parcel actual X-axis physical coordinate and the parcel actual Y-axis physical coordinate;
(3) Filtering the depth value of the belt falling part through the RGBD camera mounting height;
(4) Fitting the plane of the wrapping surface by a random sampling consistency algorithm (RandomSampleConsensus) of PCL to obtain a plane fitting equation formula:
ax+by+cz+d＝0，
(5) Substituting the actual coordinates of the package into the plane fitting equation formula, calculating to obtain an error value of each point from the plane, and recording as a deviation point if the error value is greater than a set threshold value;
(6) And finally, calculating the proportion between the number of the deviated points and the number of the actual points. For cartons, if the deviation point/actual point threshold is too large, the carton is subdivided into broken cartons; for soft packs, if the offset/actual point threshold is too small, it is subdivided into film-covered cartons.
In summary, according to the RGBD-based package type detection method provided by the invention, the package type is further determined by adopting the RGBD image acquisition data after determining that the position state of the package is a single package, and the package type and the position state of the package are determined according to the depth information in the RGBD image acquisition data, namely based on the combination of the depth data and the RGB data, so that the package type can be determined more accurately, and the algorithm precision can be ensured and the operation speed of the algorithm can be greatly increased by adopting the cascade algorithm architecture when the package position state detection and the package type determination are performed. In addition, the full automatic detection does not need personnel participation, so the labor cost is reduced. Therefore, the RGBD-based package category detection method provided by the invention has the advantages of high detection precision, low cost and high detection efficiency.
As another embodiment of the present invention, an electronic device is provided, which includes a memory and a processor, wherein the memory stores computer program instructions, and the processor is configured to load and execute the computer program instructions to implement the RGBD-based package category detection method described above.
As shown in fig. 8, the electronic device may include: at least one processor 81, such as a CPU (Central Processing Unit), at least one communication interface 83, memory 84, and at least one communication bus 82. Wherein a communication bus 82 is used to enable the connection communication between these components. The communication interface 83 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 83 may also include a standard wired interface and a standard wireless interface. The Memory 84 may be a high-speed RAM Memory (volatile Random Access Memory) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 84 may alternatively be at least one memory device located remotely from the processor 81. Wherein the memory 84 stores an application program, and the processor 81 calls the program code stored in the memory 84 for executing any of the above method steps.
The communication bus 82 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 82 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 8, but this is not intended to represent only one bus or type of bus.
The memory 84 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (english: flash memory), such as a hard disk (HDD) or a solid-state drive (SSD); the memory 84 may also comprise a combination of the above types of memory.
The processor 81 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of CPU and NP.
The processor 81 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The aforementioned PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Optionally, the memory 84 is also used to store program instructions. Processor 81 may invoke program instructions to implement the RGBD-based package category detection method as shown in the embodiment of fig. 1 of the present invention.
As another embodiment of the present invention, a storage medium is provided, wherein the storage medium stores computer instructions which are loaded by a processor to execute the RGBD-based package category detection method described above.
In an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, where the computer-readable storage medium stores computer-executable instructions, where the computer-executable instructions may execute the RGBD-based package category detection method in any of the above method embodiments. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk Drive (Hard Disk Drive, abbreviated as HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
It will be understood that the above embodiments are merely exemplary embodiments taken to illustrate the principles of the present invention, which is not limited thereto. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the spirit and substance of the invention, and these modifications and improvements are also considered to be within the scope of the invention.
Claims (10)
1. A parcel category detection method based on RGBD is characterized by comprising the following steps:
acquiring RGBD image acquisition data;
determining the position state of a parcel according to RGB image information and depth information in the RGBD image acquisition data, wherein the position state of the parcel comprises a stacked parcel and a non-stacked parcel;
when the position state of the parcel is a single parcel in the non-stacked parcels, extracting a parcel area image of RGB image information where the single parcel is located;
and determining the parcel category of the single parcel according to the parcel area image and the depth information corresponding to the single parcel, and outputting the parcel category.
2. The RGBD-based package category detection method according to claim 1, wherein determining the position status of the package according to the RGB image information and the depth information in the RGBD image acquisition data comprises:
preprocessing RGB image information in the RGBD image acquisition data to obtain a preprocessed image;
positioning the position state of the parcel in the preprocessed image according to a target detection algorithm;
if the position state of the parcel conforms to a first preset category, outputting the position state of the parcel as a non-stacked parcel, wherein the non-stacked parcel comprises a nearby parcel and a single parcel;
and if the position state of the parcel conforms to a second preset category, outputting the position state of the parcel as a stack parcel.
3. The RGBD-based package category detection method according to claim 2,
when the position state of the parcel is a non-stacked parcel, if the preprocessed image comprises a single parcel, determining that the position state of the parcel is the single parcel;
if the preset processing image comprises a plurality of independent parcels, determining whether the parcel is a nearby parcel according to the depth information of every two adjacent independent parcels;
and if the difference between the depth values of every two adjacent independent parcels is smaller than a preset threshold value, determining that the position state of the parcel is a distance-approaching parcel.
4. The RGBD-based package category detection method according to claim 3, wherein the determining whether the package is a nearby package according to the depth information of each two adjacent independent packages comprises:
calculating the actual Y-axis physical coordinate of each independent parcel according to the pixel coordinate of the parcel area in the preset processing image and the height value corresponding to the pixel coordinate;
calculating the Y-axis distance between every two adjacent independent parcels according to the actual Y-axis physical coordinates of each independent parcel;
comparing the Y-axis distance between every two adjacent independent parcels with a preset threshold value;
and if the Y-axis distance between every two adjacent independent parcels is smaller than a preset threshold value, determining that the non-stacked parcel is a distance-adjacent parcel.
5. The RGBD-based package category detection method according to any one of claims 1 to 4, wherein when the position status of the package is a single package, extracting a package area image of RGB image information where the single package is located comprises:
extracting a parcel region comprising a single parcel from the RGB image information to obtain an extracted image;
and carrying out image scaling and normalization processing on the extracted image to obtain a parcel area image.
6. The RGBD-based package category detection method according to any one of claims 1 to 4, wherein determining the package category of the single package according to the package area image and the depth information corresponding to the single package and outputting the package category comprises:
carrying out classification processing on the single parcel in the parcel area image according to an object classification algorithm;
determining a package category according to the classification processing result, wherein the package category comprises a carton, a foam box, a bag collecting bag and a soft bag;
and if the package type is a paper box or a soft package, determining the fine-grained classification of the paper box or the soft package according to the depth information corresponding to the single package, wherein the fine-grained classification of the paper box comprises breakage and normality, and the fine-grained classification of the soft package comprises a film-coated paper box and a soft package bag.
7. The RGBD-based package category detection method of claim 6, wherein determining a package category from the result of the classification process comprises:
extracting a corresponding parcel picture from the RGBD image acquisition data according to the minimum circumscribed matrix of the single parcel;
adjusting the size of the parcel picture, and performing normalization processing on the parcel picture;
and performing package type identification on the package picture after the normalization processing according to an object classification model to obtain a package type identification result.
8. The RGBD-based package category detection method according to claim 6, wherein if the package category is carton or soft package, determining the fine-grained classification of the carton or soft package according to the depth information corresponding to the single package comprises:
acquiring the actual coordinates of the single parcel, and determining the depth value of the area where the single parcel is located;
calculating the actual spatial coordinate value of the area according to the depth value of the area where the single parcel is located;
fitting the actual space coordinate value by combining a fitting algorithm to obtain a plane of a fitting package surface and obtain a plane fitting equation formula;
substituting the actual coordinates of the single parcel into the plane fitting equation formula, and calculating the error value of each point of the single parcel from the plane of the surface of the fitted parcel;
marking the points with the error values larger than a preset error threshold value as deviation points, and counting the number of the deviation points;
when the parcel type is a carton, if the ratio of the number of the deviation points to the number of the actual points of the single parcel is greater than a preset ratio threshold value, determining that the single parcel is a damaged carton, otherwise, determining that the single parcel is a normal carton;
and when the package type is soft package, if the ratio of the number of the deviation points to the number of the actual points of the single package is smaller than a preset ratio threshold value, determining that the single package is a film-coated carton, otherwise, determining that the single package is a soft package bag.
9. The RGBD-based package category detection method according to any one of claims 1 to 4, wherein when the location status of the package is a stacked package, information that does not require category determination of the package is output.
10. An electronic device comprising a memory having stored therein computer program instructions and a processor configured to load and execute the computer program instructions to implement the RGBD-based package category detection method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211432120.XA CN115908913A (en) | 2022-11-16 | 2022-11-16 | RGBD-based parcel category detection method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211432120.XA CN115908913A (en) | 2022-11-16 | 2022-11-16 | RGBD-based parcel category detection method and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115908913A true CN115908913A (en) | 2023-04-04 |
Family
ID=86489083
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211432120.XA Pending CN115908913A (en) | 2022-11-16 | 2022-11-16 | RGBD-based parcel category detection method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115908913A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116273914A (en) * | 2023-05-18 | 2023-06-23 | 中科微至科技股份有限公司 | Multi-channel sorting machine for separating package single piece |
| CN116902559A (en) * | 2023-08-23 | 2023-10-20 | 中科微至科技股份有限公司 | Visual positioning correction method for conveying sheet-like object |
-
2022
- 2022-11-16 CN CN202211432120.XA patent/CN115908913A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116273914A (en) * | 2023-05-18 | 2023-06-23 | 中科微至科技股份有限公司 | Multi-channel sorting machine for separating package single piece |
| CN116273914B (en) * | 2023-05-18 | 2023-08-04 | 中科微至科技股份有限公司 | Multi-channel sorting machine for separating package single piece |
| CN116902559A (en) * | 2023-08-23 | 2023-10-20 | 中科微至科技股份有限公司 | Visual positioning correction method for conveying sheet-like object |
| CN116902559B (en) * | 2023-08-23 | 2024-03-26 | 中科微至科技股份有限公司 | Visual positioning correction method for conveying sheet-like object |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108009675B (en) | Goods packing method, device and system | |
| CN115908913A (en) | RGBD-based parcel category detection method and electronic equipment | |
| CN112950547B (en) | Machine vision detection method for lithium battery diaphragm defects based on deep learning | |
| CN103489200A (en) | Image processing apparatus and image processing method | |
| CN113167569B (en) | Volume measurement device, system, method, and program | |
| CN111008561A (en) | Livestock quantity determination method, terminal and computer storage medium | |
| CN113608916A (en) | Fault diagnosis method and device, electronic equipment and storage medium | |
| CN110349216A (en) | Container method for detecting position and device | |
| CN113065810A (en) | Method, device, computing equipment, logistics system and storage medium for detecting packages | |
| CN112513877A (en) | Target object identification method, device and system | |
| CN111814739A (en) | Method, device and equipment for detecting express package volume and storage medium | |
| CN111160450A (en) | Fruit and vegetable weighing method based on neural network, storage medium and device | |
| CN112785595A (en) | Target attribute detection, neural network training and intelligent driving method and device | |
| CN114219402A (en) | Logistics tray stacking identification method, device, equipment and storage medium | |
| CN109829401A (en) | Traffic sign recognition method and device based on double capture apparatus | |
| CN111696152B (en) | Method, device, computing equipment, system and storage medium for detecting package stack | |
| CN109759339B (en) | Sorting method, sorting device and sorting terminal for logistics packages | |
| CN110909633B (en) | Method and device for determining accumulation degree, storage medium, and electronic device | |
| CN113379826A (en) | Method and device for measuring volume of logistics piece | |
| CN113978987B (en) | Pallet object packaging and picking method, device, equipment and medium | |
| CN113160414B (en) | Automatic goods allowance recognition method, device, electronic equipment and computer readable medium | |
| CN114519793A (en) | Target object detection method and device, electronic equipment and storage medium | |
| CN115730638A (en) | Network pruning method, device and equipment and computer storage medium | |
| CN114708230A (en) | Vehicle frame quality detection method, device, equipment and medium based on image analysis | |
| CN115631125A (en) | Parcel detection method and parcel processing apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |