CN117389742A - Edge computing method, device and storage medium for machine vision - Google Patents
Edge computing method, device and storage medium for machine vision Download PDFInfo
- Publication number
- CN117389742A CN117389742A CN202311512133.2A CN202311512133A CN117389742A CN 117389742 A CN117389742 A CN 117389742A CN 202311512133 A CN202311512133 A CN 202311512133A CN 117389742 A CN117389742 A CN 117389742A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- edge
- visual
- subtasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 34
- 230000000007 visual effect Effects 0.000 claims abstract description 92
- 238000012545 processing Methods 0.000 claims abstract description 91
- 238000000605 extraction Methods 0.000 claims abstract description 44
- 238000000034 method Methods 0.000 claims abstract description 44
- 230000008569 process Effects 0.000 claims abstract description 26
- 238000010223 real-time analysis Methods 0.000 claims abstract description 23
- 239000000284 extract Substances 0.000 claims abstract description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 37
- 238000001514 detection method Methods 0.000 claims description 29
- 238000004458 analytical method Methods 0.000 claims description 26
- 230000006399 behavior Effects 0.000 claims description 18
- 238000013527 convolutional neural network Methods 0.000 claims description 15
- 238000007405 data analysis Methods 0.000 claims description 15
- 230000004438 eyesight Effects 0.000 claims description 11
- 238000003709 image segmentation Methods 0.000 claims description 10
- 238000010801 machine learning Methods 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 10
- 239000013598 vector Substances 0.000 claims description 9
- 238000013135 deep learning Methods 0.000 claims description 5
- 230000004927 fusion Effects 0.000 claims description 5
- 238000002372 labelling Methods 0.000 claims description 5
- 230000009467 reduction Effects 0.000 claims description 5
- 238000013461 design Methods 0.000 claims description 4
- 238000003672 processing method Methods 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 238000004148 unit process Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 61
- 230000004044 response Effects 0.000 abstract description 10
- 238000007781 pre-processing Methods 0.000 description 22
- 238000012790 confirmation Methods 0.000 description 15
- 230000000694 effects Effects 0.000 description 13
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 241001522296 Erithacus rubecula Species 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000004382 visual function Effects 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an edge computing method, an edge computing device and a storage medium for machine vision. The method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network; the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks; and carrying out real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results. Thus, the data transmission delay can be reduced, and the real-time data processing and response can be realized. By calculating and deducing on the edge equipment, the visual data can be rapidly analyzed and processed, and the requirements of real-time application scenes are met.
Description
Technical Field
The invention provides an edge computing method, device and storage medium for machine vision, and belongs to the technical field of machine vision and edge computing.
Background
Machine vision is a discipline involving the fields of artificial intelligence, computer science and engineering, aimed at studying how to simulate the visual functions of a human using a machine. Machine vision systems may enable perception, recognition, and understanding of the real world through image acquisition, image processing, and image interpretation.
Edge computing is a computing model that provides near-end services in close proximity to the side of the object or data source, using an open platform with integrated network, computing, storage, and application core capabilities. The application program is initiated at the edge side, generates faster network service response, and meets the basic requirements of the industry in the aspects of real-time service, application intelligence, security, privacy protection and the like. Edge computation is between a physical entity and an industrial connection, or at the top of a physical entity. The cloud computing can still access the historical data of the edge computing.
Disclosure of Invention
The invention provides an edge computing method, device and storage medium for machine vision, which are used for solving the problems that the processing efficiency is low and the load of a cloud platform and the network bandwidth consumption are increased due to the fact that the traditional machine vision task processing depends on the cloud platform:
The invention provides an edge computing method for machine vision, which comprises the following steps:
s1: the method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network;
s2: the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks;
s3: performing real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results;
s4: the edge equipment sends the generated corresponding decision, control instruction and event trigger signal to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information;
s5: and the cloud platform transmits the feedback information back to the edge equipment, and the edge equipment takes corresponding operation or behavior according to the feedback information and transmits the operation or behavior to the execution equipment or system.
Further, the vision equipment comprises a camera and a sensor; the visual data includes image data and video data.
Further, the edge equipment preprocesses the received visual data and analyzes a machine visual task, and according to the characteristics of the machine visual task, the machine visual task is decomposed into a plurality of subtasks and the subtasks are subjected to feature extraction; the subtasks include: target detection task, image classification task, face recognition task and image segmentation task; the feature extraction of the subtasks comprises the following steps:
the target detection task determines the position of an object in an image by using a boundary box, classifies the detected object, determines the category of the object, and tracks the position and the motion trail of the object in continuous frames;
the image classification task extracts the features from the images through a convolutional neural network, designs and trains a classifier model, and matches the extracted features with predefined categories to obtain image classification results;
the face recognition task determines the position of a face by detecting the face in an image or a video, and performs alignment operation on the detected face; extracting feature vectors in the face image through a deep learning algorithm; matching the extracted feature vector with the face features in the database to determine the identity or perform verification/identification;
The image segmentation task obtains accurate segmentation results of a plurality of objects in an image by distributing each pixel in the image to different semantic categories and simultaneously segmenting and labeling different objects at the pixel level.
Further, the subtasks are extracted by a parallel processing method, and the method comprises the following steps:
decomposing the machine vision task into a plurality of subtasks;
according to task division, visual data are decomposed into a plurality of data blocks, and the size and the number of the data blocks are determined according to available parallel computing resources and task complexity;
equally distributing the data blocks to the parallel computing units according to the performance and task complexity of the computing units by using a task scheduling algorithm;
the parallel computing unit processes the distributed data blocks through a convolutional neural network and performs feature extraction;
after all the parallel computing units finish feature extraction, combining the generated features into integral features through a feature fusion algorithm;
and carrying out next processing on the integral characteristic according to specific task requirements, wherein the next processing comprises dimension reduction, normalization and calibration.
Further, the subtasks after feature extraction are analyzed and processed in real time through a machine vision model on the edge equipment, and corresponding decision making, control instructions and event triggering signals are generated according to real-time analysis and processing results; comprising the following steps:
S31: inputting the subtasks for feature extraction into corresponding subtask models, and analyzing and processing the subtasks for feature extraction in real time through a machine learning algorithm;
s32: the edge device can generate corresponding decisions according to predefined rules or policies;
s33: according to the generated decision, the edge equipment generates a corresponding control instruction, wherein the control instruction is used for controlling related equipment or executing specific operation;
s34: based on the occurrence of a specific event or the detection of a specific target, the edge device generates a corresponding event trigger signal.
Further, the edge equipment sends corresponding decision making, control instruction and event triggering signals to a cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information; comprising the following steps:
s41: the cloud platform receives decision, control instructions and event trigger signals from the edge equipment and stores the decision, control instructions and event trigger signals in a data structure;
s42: the cloud platform analyzes the received data and extracts relevant information according to a preset format and protocol; analyzing and predicting decision results through a machine learning algorithm according to different data types;
s43: based on the results of the data parsing and processing, the cloud platform may generate corresponding feedback information.
Further, the feedback information comprises state update, control instruction confirmation, event trigger signal confirmation, decision result feedback and data analysis report.
Further, the cloud platform receives decision, control instructions and event trigger signals from the edge device and stores the decision, control instructions and event trigger signals in a data structure; comprising the following steps:
s411: determining a time period for storing the data segment, and acquiring a time stamp of the data from the received decision, control instruction and event trigger signal;
s412: creating a storage structure for each time period, wherein the storage structure comprises a database table, a folder and a file;
s413: the received data is distributed into the corresponding time period according to the time stamp by comparing the time stamp with the starting time and the ending time of the time period;
s414: data categorized into respective time periods is stored into corresponding storage structures.
The invention provides an edge computing device for machine vision, which comprises:
and a data acquisition module: the method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network;
and the feature extraction module is used for: the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks;
And an analysis and processing module: performing real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results;
an information generation module: the edge equipment sends the generated corresponding decision, control instruction and event trigger signal to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information;
the instruction transfer module: and the cloud platform transmits the feedback information back to the edge equipment, and the edge equipment takes corresponding operation or behavior according to the feedback information and transmits the operation or behavior to the execution equipment or system.
The present invention proposes a storage medium having stored thereon a computer program to be executed by a processor to implement an edge calculation method for machine vision as described in any of the above.
The invention has the beneficial effects that: edge computing pushes the processing of machine vision tasks to edge devices closer to the data source. Thus, the data transmission delay can be reduced, and the real-time data processing and response can be realized. By calculating and deducing on the edge equipment, the visual data can be rapidly analyzed and processed, so that the requirements of real-time application scenes are met; the edge computing method moves data processing and storage to the edge equipment, so that the requirement for transmitting sensitive data to a cloud platform is reduced; the privacy risk in the data transmission process is reduced, and the safety of the data is improved. By performing the calculation locally, the privacy and the integrity of the data can be better protected; and the edge calculation transfers a part of calculation tasks from the cloud to the edge equipment for processing, so that the load of the cloud platform and the network pressure of data transmission are reduced. Only key data to be transmitted and stored are sent to the cloud, so that the data transmission quantity and the occupation of cloud computing resources are effectively reduced, and the efficiency of the whole system is improved; the edge computing method allows the edge equipment to still perform partial machine vision task processing when the network is disconnected or the edge equipment cannot be connected to the cloud platform; providing the ability to work offline and increasing the fault tolerance of the system. By realizing a certain degree of intelligence on the edge equipment, the dependence on network connection can be reduced, and the stability and reliability of the system are ensured; edge computing may reduce the need for cloud computing resources, thereby reducing costs. And the data processing and storage are moved to the edge equipment, so that the cost of cloud computing service can be saved, and better cost performance is realized through more efficient local computing. In addition, edge computation may also reduce the bandwidth consumption and network communication costs required for data transmission.
Drawings
Fig. 1 is a step diagram of an edge computing method for machine vision according to the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will be more clearly understood, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, and the described embodiments are merely some, rather than all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
According to one embodiment of the present invention, an edge computing method for machine vision is provided, the method comprising:
S1: the method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network; the vision equipment comprises a camera and a sensor; the visual data includes image data and video data. The network transmission time calculation formula is as follows:
wherein Y is the rated transmission rate of the visual equipment, C is the total data transmission rate of the visual equipment on each sub-channel, L is the data transmission rate of the visual equipment on the sub-channel j during data transmission, n is the number of sub-channels, t is the operation time period of the edge equipment, and d is the gain of the sub-channel.
S2: the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks; the preprocessing comprises image data preprocessing and video data preprocessing; the received image data is preprocessed. Preprocessing may include denoising, image enhancement, resizing, etc. operations to improve the accuracy and effectiveness of subsequent tasks. The received video data is preprocessed. Preprocessing may include frame rate control, video compression, key frame extraction, etc. to meet real-time processing and resource conservation requirements. The characteristics include diversity, complexity, real-time, large data volume, unstructured data, and robustness.
S3: performing real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results;
s4: the edge equipment sends the generated corresponding decision, control instruction and event trigger signal to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information;
s5: and the cloud platform transmits the feedback information back to the edge equipment, and the edge equipment takes corresponding operation or behavior according to the feedback information and transmits the operation or behavior to the execution equipment or system. The operations or behaviors include device control and task adjustment; the edge device can execute the control operation of the device according to the feedback information provided by the cloud platform. For example, the edge device may turn on, turn off, or adjust camera parameters and sensor settings in accordance with the control instructions acknowledge. Task adjustment: according to decision result feedback and state update of the cloud platform, the edge device can adjust the execution mode and priority of the current task. For example, the edge device may adjust the execution policy of the image classification task based on the results of the object detection task.
The working principle of the technical scheme is as follows: the vision equipment (such as a camera and a sensor) is responsible for collecting image data and video data and transmitting the data to the edge equipment through network transmission; the edge device pre-processes the received visual data. And preprocessing the image data, such as denoising, image enhancement, size adjustment and the like, so as to improve the accuracy and effect of subsequent tasks. Preprocessing video data, such as frame rate control, video compression, key frame extraction and the like, so as to meet the requirements of real-time processing and resource saving; and extracting the characteristics of the preprocessed data by using a machine vision model on the edge equipment. According to the characteristics of the machine vision task, decomposing the task into a plurality of subtasks, and extracting the characteristics of each subtask; the method comprises various tasks such as target detection, classification, segmentation and the like; and analyzing and processing the subtasks in real time through a machine vision model. The model may further calculate and infer the extracted features to obtain specific analysis results. Including target recognition, object tracking, anomaly detection, behavior recognition, etc.; and generating corresponding decision, control instruction and event trigger signals by the edge equipment according to the real-time analysis and processing results. This information may be used for device control, task adjustment, or interaction with other systems. The edge equipment sends the generated information to the cloud platform; and the cloud platform receives the information sent by the edge equipment, and processes and analyzes the information. Through further analysis of the data, the cloud platform generates corresponding feedback information such as equipment control instructions, task optimization suggestions and the like; the edge equipment receives feedback information from the cloud platform and executes corresponding operation or behavior according to the instruction and the suggestion in the feedback information; including device control, such as adjusting camera parameters or sensor settings, and task adjustment, such as adjusting task execution policies based on target detection results.
The technical scheme has the effects that: the real-time machine vision processing can be realized by putting the vision data acquisition, preprocessing and analysis tasks on the edge equipment to finish; data transmission delay is reduced, and analysis and decision making of visual data can be responded more quickly; by preprocessing visual data, extracting features and partially analyzing tasks on the edge equipment, the computing resource and bandwidth requirements on the cloud platform can be reduced. Only key information and decision results need to be transmitted to the cloud platform, so that a large amount of data transmission and calculation cost is reduced; since the visual data processing is mainly performed on the edge device, the dependence on network connection is reduced. Even if the network is unstable or disconnected, the edge equipment can still perform visual analysis and decision to a certain extent, so that the robustness of the system is improved; by performing data processing and decision making on the edge device, the transmission of sensitive data can be reduced, thereby improving the privacy preserving capability. Important data can be processed locally on the equipment, and only necessary results are transmitted to the cloud platform, so that data leakage and privacy risks are reduced; the feedback information generated by the cloud platform can help the edge device to control the device and adjust the task. According to the decision result and state update of the cloud platform, the edge device can automatically adjust the execution mode and priority of the current task, so that more flexible and intelligent system operation is realized. The network transmission time can be calculated according to the transmission rate, the data transmission rate, the number of sub-channels, the gain and other parameters of the visual equipment through the formula. Therefore, the time of data transmission can be optimized, the instantaneity and the response speed are improved, and the transmission time is ensured to be within the available time range of the edge equipment. Meanwhile, the formula has scalability and adaptability, can be adjusted and customized according to specific requirements, and is suitable for different network environments and equipment requirements. Meanwhile, by calculating the respective parameters in the formula, the network transmission time T may be calculated according to the rated transmission rate Y of the vision device, the total data transmission rate C on each sub-channel, and the data transmission rate L on each sub-channel. Therefore, the time of data transmission can be optimized, and the real-time performance and response speed of the data are improved; the presence of the number of sub-channels n and the sub-channel gain d in the formula allows a more accurate calculation of the transmission time. The number and gain of the sub-channels are important factors of the network structure and environment, and by considering the parameters, the transmission strategy and the resource allocation can be better adjusted to achieve the optimal transmission time; the limit condition T of the operation time period T of the edge equipment in the formula is less than or equal to T-nd, so that the transmission time is ensured not to exceed the available time of the edge equipment. Therefore, the data transmission can be ensured to be completed within a limited time, and the task execution delay or failure caused by overlong transmission time is avoided; the parameters in the formula can be adjusted according to specific conditions, such as the number of the subchannels and the setting of the gains, and the formula has good scalability and adaptability. The system can be customized according to different network environments and equipment requirements so as to meet the transmission requirements in different scenes.
According to one embodiment of the invention, the edge equipment preprocesses the received visual data and analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks; the subtasks include: target detection task, image classification task, face recognition task and image segmentation task; the feature extraction of the subtasks comprises the following steps:
the target detection task determines the position of an object in an image by using a boundary box, classifies the detected object, determines the category of the object, and tracks the position and the motion trail of the object in continuous frames;
the image classification task extracts features from the images through a convolutional neural network, designs and trains a classifier model, and matches the extracted features with predefined categories to obtain image classification results;
the face recognition task determines the position of a face by detecting the face in an image or a video, and performs alignment operation on the detected face, so that different faces have consistent spatial relationship; extracting feature vectors in the face image through a deep learning algorithm; matching the extracted feature vector with the face features in the database to determine the identity or perform verification/identification;
The image segmentation task obtains accurate segmentation results of a plurality of objects in an image by distributing each pixel in the image to different semantic categories and simultaneously segmenting and labeling different objects at the pixel level.
The working principle of the technical scheme is as follows: after the edge device receives the visual data, the data is preprocessed. This includes denoising, enhancement and resizing of image data, as well as frame rate control, compression and key frame extraction of video data. Then, according to the characteristics of the machine vision task, decomposing the task into subtasks such as target detection, image classification, face recognition, image segmentation and the like; for the object detection task, the edge device uses a corresponding algorithm and model, such as a Convolutional Neural Network (CNN) based object detection model, to determine the position of the object in the image through a bounding box, and to classify the detected object to determine the class of the object. Meanwhile, tracking the position and the motion trail of the object in continuous frames through a tracking algorithm; the image classification task uses convolutional neural networks to extract representative features from the images. The edge equipment designs and trains a classifier model suitable for image classification, and the model matches the extracted features with predefined categories so as to obtain the classification result of the image; face recognition tasks involve face detection, alignment, and feature extraction. The edge device determines the face position in the image by using a face detection algorithm and performs an alignment operation so that different faces have a consistent spatial relationship. Then, feature vectors in the face image are extracted through a deep learning algorithm. Finally, matching the extracted feature vector with the face features in the database, thereby determining the identity or performing verification/identification; the image segmentation task aims at assigning each pixel in the image to a different semantic category and at the same time accurately segmenting and labeling different objects. The edge device performs a pixel-level segmentation operation on the image using a segmentation algorithm, such as a Convolutional Neural Network (CNN) -based segmentation model, so as to obtain accurate segmentation results for a plurality of objects in the image.
The technical scheme has the effects that: by decomposing the machine vision task into a plurality of subtasks, the edge device can process different types of tasks such as object detection, image classification, face recognition, image segmentation and the like at the same time. The edge equipment can meet diversified application requirements, so that richer and comprehensive machine vision functions are realized; for each subtask, the edge device uses a corresponding algorithm and model for feature extraction. The object detection task determines the object position and classification through a boundary box, the image classification task extracts representative image features through a convolutional neural network, the face recognition task extracts face feature vectors through an alignment and deep learning algorithm, and the image segmentation task performs object segmentation and labeling on a pixel level. The accuracy and the effect of the task can be improved through the feature extraction process; by preprocessing and task decomposition on the edge device, the time and resource consumption of data transmission to the cloud platform are reduced. The edge equipment can conduct feature extraction and task analysis in real time, so that the machine vision application can respond and process visual data at a higher speed, delay is reduced, and instantaneity is improved; the edge equipment performs preprocessing and partial task analysis on the visual data, and only sends key information and decision results to the cloud platform; the data transmission and calculation resource requirements on the cloud platform are reduced, and the pressure and cost of the cloud platform are reduced; by performing partial data processing and task analysis on the edge device, sensitive visual data can be processed locally without transmission to the cloud platform; the privacy and data security of the user can be protected.
In one embodiment of the invention, the subtasks are extracted by a parallel processing method, and the method comprises the following steps:
decomposing the machine vision task into a plurality of subtasks;
according to task division, visual data are decomposed into a plurality of data blocks, and the size and the number of the data blocks are determined according to available parallel computing resources and task complexity; the number calculation formula of the data blocks is as follows: the calculation formula of the number S of the data blocks is as follows:
the data record number is H, the average record size is E, the available computing resource is K, the maximum parallel line number is P, and the reserved memory quantity is F.
Equally distributing the data blocks to the parallel computing units according to the performance and task complexity of the computing units by using a task scheduling algorithm;
the parallel computing unit processes the distributed data blocks through a convolutional neural network and performs feature extraction;
after all the parallel computing units finish feature extraction, combining the generated features into integral features through a feature fusion algorithm;
and carrying out next processing on the integral characteristic according to specific task requirements, wherein the next processing comprises dimension reduction, normalization and calibration.
The working principle of the technical scheme is as follows: first, a machine vision task is decomposed into a plurality of subtasks. Then, the inputted visual data is decomposed into a plurality of data blocks according to the task division. The size and number of data blocks is determined based on the available parallel computing resources and the complexity of the task. The specific determination method comprises the following steps:
Parallel computing resource limitations: if the edge device has multiple CPU cores or GPUs, the size and number of data blocks may be determined based on available computing resources. For example, if 4 CPU cores are available, the data set may be divided into 4 blocks, each of which is processed by one CPU core.
Task complexity: the complexity of the task also has an impact on the determination of the data block size and number. If the task is very complex, processing a large block of data may result in a long computation time, resulting in other computing units being idle. In this case, the data block may be subdivided into smaller sub-blocks to better utilize parallel computing resources and improve overall processing efficiency of the task.
Data generation speed: if the data is generated at a high speed and real-time processing is required, the size and number of data blocks should be set to a level that can be processed quickly. This ensures that new data blocks arrive before the computation is completed, ensuring real-time. For example, in an image classification task, it is assumed that an edge device has 8 CPU cores available, and the task is to process 1000 images. If the processing time for each image is uniform, the data set may be divided into 8 data blocks, each containing 125 images. Thus, each CPU core is responsible for processing one data block, 8 images can be processed simultaneously, and the processing speed is improved. However, if some images in a task are more complex and require longer processing time than others, the data blocks may be further subdivided according to the complexity of the images. For example, the data set may be divided into 16 data blocks, each containing 62.5 images. In this way, computing resources can be subdivided for faster processing when processing longer time images.
The data blocks are equally distributed to the available parallel computing units using a task scheduling algorithm. Each parallel computing unit processes the allocated data blocks by using a convolutional neural network or other algorithm, and performs feature extraction operation. The specific task scheduling algorithm comprises the following steps:
round robin scheduling algorithm: the data blocks are allocated to different computing units one by one in the order of the computing units. This ensures that each computing unit receives the data blocks on average, thereby achieving load balancing. For example, assuming that there are 4 CPU cores and 16 data blocks, the round robin scheduling algorithm would allocate the first 4 data blocks to the 1 st, 2 nd, 3 rd and 4 th CPU cores, respectively, and then continue to allocate the remaining data blocks in a round robin fashion.
Shortest job first scheduling algorithm: according to the processing time of each data block, the data block is allocated to the calculation unit with the shortest processing time. This minimizes the overall processing time of the task. For example, assuming that there are 4 CPU cores and 8 data blocks, each data block has a processing time of 2, 4, 6, 8, 10, 12, 14, and 16 time units, respectively, the shortest job priority scheduling algorithm will assign the first 4 data blocks to the 4 CPU cores with the shortest processing times, respectively.
Dynamic scheduling algorithm: the allocation of the data blocks is dynamically adjusted according to the real-time performance and task complexity of the computing unit. For example, a computational unit with a lower load may be allocated to a data block with a higher complexity based on the real-time load situation and task complexity of the computational unit to fully utilize the computational resources. The scheduling algorithm can improve the overall parallel computing efficiency.
After all the parallel computing units finish feature extraction, the generated features need to be fused. And combining the features generated by each computing unit into integral features through a feature fusion algorithm. This allows the feature information of each data block to be preserved and the results of multiple parallel computing units to be taken into account comprehensively.
And (3) further processing: the overall characteristics are further processed according to specific task requirements. This may include a dimension reduction operation, reducing the dimension of the feature; normalizing, namely mapping the characteristic value to a fixed range; and a calibration operation, wherein the characteristics are adjusted to meet specific task requirements.
The technical scheme has the effects that: and by a parallel processing method, the characteristics of a plurality of data blocks are extracted at the same time, and the available computing resources are fully utilized. Therefore, the feature extraction efficiency can be improved, and the task processing speed can be increased; the computing tasks are distributed evenly to the available parallel computing units through task partitioning and data partitioning. According to the performance and task complexity of the computing unit, reasonably distributing the data blocks, thereby maximally utilizing parallel computing resources and improving the processing capacity of the whole system; and combining the features generated by the plurality of parallel computing units into integral features through a feature fusion algorithm. The results of the plurality of computing units can be comprehensively considered, and more accurate and comprehensive characteristic representation can be obtained while the respective characteristic information is maintained; the overall characteristics are further processed according to specific task requirements. This includes a dimension reduction operation, reducing the dimension of the feature; normalizing, namely mapping the characteristic value to a fixed range; and a calibration operation, wherein the characteristics are adjusted to meet specific task requirements. Therefore, flexible characteristic processing can be carried out according to specific requirements of the task, and the execution effect of the task is optimized. Via the formula, the visual data can be reasonably divided into a plurality of data blocks according to the available parallel computing resources and task complexity, and the size and the number of the data blocks can be determined. Therefore, the computing resource can be utilized to the greatest extent, the parallelism of the tasks is optimized, the processing efficiency and the performance of the system are improved, the balance and the locality of data are ensured, and the execution effect of the tasks is improved. Meanwhile, the number of the data blocks can be determined according to the hardware configuration and the resource limitation of the system through the available computing resource K and the maximum parallel number P in the computing formula; reasonable utilization of computing resources can be ensured, and processing efficiency and performance are improved; by determining the size and number of data blocks, the parallelism of a task can be optimized according to the task complexity and the available parallel computing resources. According to parameter adjustment in the formula, the number of data blocks and the parallelism of tasks can be balanced in a reasonable range so as to obtain the optimal processing effect; the amount of memory F set forth in the formula may be provided to the system for other operations and buffer management, taking into account computational resource and memory constraints. By reasonably setting the value of F, the performance degradation and errors of the system caused by insufficient resources can be prevented; according to the data record number H and the average record size E in the formula, the data can be balanced divided and managed; the method can ensure the moderate size of the data block so as to meet the requirement of task processing, fully utilize the locality of the data and reduce the expenditure of data access.
According to one embodiment of the invention, the subtasks after feature extraction are analyzed and processed in real time through a machine vision model on the edge equipment, and corresponding decision making, control instructions and event triggering signals are generated according to real-time analysis and processing results; comprising the following steps:
s31: inputting the subtasks for feature extraction into corresponding subtask models, and analyzing and processing the subtasks for feature extraction in real time through a machine learning algorithm;
s32: the edge device can generate corresponding decisions according to predefined rules or policies;
s33: according to the generated decision, the edge equipment generates a corresponding control instruction, wherein the control instruction is used for controlling related equipment or executing specific operation;
s34: based on the occurrence of a specific event or the detection of a specific target, the edge device generates a corresponding event trigger signal.
The working principle of the technical scheme is as follows: the subtasks after preprocessing and feature extraction are input into the corresponding machine vision model. These models use machine learning algorithms to analyze and process features in real time. The model may be a trained deep learning model, such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN), for tasks such as object detection, image classification, face recognition, etc. Calculating and deducing the characteristics, and generating a real-time analysis result by the model; based on the real-time analysis and processing results, the edge device may generate corresponding decisions according to predefined rules or policies. These rules or policies may be based on specific application requirements or task goals. For example, in a target detection task, the edge device may determine whether further action or decision needs to be taken based on the detected object type and location; and generating a corresponding control instruction by the edge equipment according to the generated decision. The control instructions may be used to control related devices or to perform specific operations. For example, in a monitoring system, when abnormal behavior is detected, the edge device may generate control instructions that require the camera to focus or track a particular target; based on the occurrence of a specific event or the detection of a specific target, the edge device generates a corresponding event trigger signal. These signals may be used to trigger subsequent processes or to inform other systems to perform the corresponding operations. For example, in a security system, when intrusion is detected, an edge device may generate an event trigger signal informing an alarm system to respond.
The technical scheme has the effects that: and performing real-time analysis and processing on the subtasks subjected to the feature extraction by running a machine vision model on the edge equipment. This can greatly reduce response time and enable the system to react in time to perceived visual data; the edge device may generate a corresponding decision according to predefined rules or policies. Therefore, under the condition that cloud participation is not needed, a fast and accurate decision can be made according to the real-time analysis result. Meanwhile, the definition of rules and strategies can be customized according to specific application scenes so as to meet different requirements; based on the generated decisions, the edge devices may generate corresponding control instructions for controlling the associated devices or performing specific operations. The edge equipment has real-time feedback and control capability on the surrounding environment and equipment, and the system behavior can be actively adjusted and controlled according to the needs; based on the occurrence of a particular event or the detection of a particular target, the edge device may generate a corresponding event trigger signal. These signals may be used to trigger subsequent processing flows, to notify other systems to perform corresponding operations, or to trigger alarm and emergency response mechanisms. This improves the real-time and event handling capabilities of the system.
According to one embodiment of the invention, the edge equipment sends corresponding decision making, control instruction and event triggering signals to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information; comprising the following steps:
s41: the cloud platform receives decision, control instructions and event trigger signals from the edge equipment and stores the decision, control instructions and event trigger signals in a data structure;
s42: the cloud platform analyzes the received data and extracts relevant information according to a preset format and protocol; analyzing and predicting decision results through a machine learning algorithm according to different data types;
s43: based on the results of data analysis and processing, the cloud platform can generate corresponding feedback information; the feedback information comprises state update, control instruction confirmation, event trigger signal confirmation, decision result feedback and data analysis report. Specific:
and (5) updating the state: according to the data analysis result, the cloud platform can generate state update information to inform the edge equipment of task execution conditions, progress and results. For example, the edge device is informed whether a particular target has been successfully identified or detected.
Control instruction confirmation: the cloud platform may reply with a confirmation to the control instruction sent by the edge device to ensure that the instruction has been received and begins execution. The edge device can get feedback and adjust accordingly according to the confirmation information.
Event trigger signal confirmation: if the edge device informs the cloud platform of a specific event through the event trigger signal, the cloud platform can perform confirmation reply, which indicates that the signal is received and further processing is performed according to the event. For example, in a security system, an edge device sends an intrusion alert signal, and the cloud platform may acknowledge receipt of the signal and issue an alert to notify the relevant personnel.
And (3) decision result feedback: based on the analysis results, the cloud platform may generate decision result feedback, including interpretation, evaluation, and advice of the analysis results. For example, in an intelligent transportation system, the cloud platform may provide advice to adjust the timing strategy of the signal lamp according to the analysis result.
Data analysis report: the cloud platform may further analyze the data generated on the edge device and generate detailed data analysis reports. These reports may include statistics, trend analysis, anomaly detection, etc., providing a more comprehensive and thorough analysis of visual data for the decision maker.
The working principle of the technical scheme is as follows: the cloud platform receives the decision, control instruction and event trigger signal from the edge device and stores them in the corresponding data structure. Thus, the reliability and the integrity of the data can be ensured, and a foundation is provided for subsequent processing and analysis; the cloud platform analyzes the received data and extracts relevant information according to a preset format and protocol. For example, specific operational instructions are extracted from the decisions, event types and related parameters are extracted from the event trigger signals. According to different data types, the cloud platform analyzes and predicts the decision result by using a machine learning algorithm so as to better understand the meaning and influence of the data; based on the results of the data parsing and processing, the cloud platform may generate corresponding feedback information. Such feedback information may include status updates for reporting task performance to the edge device; control instruction confirmation, which confirms that the corresponding control instruction is received and executed; the event triggering signal confirms that the corresponding event processing is received and triggered; the decision result feedback is carried out, and the cloud analysis and prediction results are sent back to the edge equipment for reference; data analysis reports provide detailed reports on the results of data analysis and processing.
The technical scheme has the effects that: the decision, control instruction and event trigger signal generated by the edge equipment are sent to the cloud platform for processing and analysis, so that the complex decision task can be completed by utilizing powerful calculation and storage resources of the cloud platform. The cloud platform can analyze and predict the decision result by using a more complex and accurate machine learning algorithm, so that more accurate and intelligent decision is provided; the cloud platform can receive and analyze data sent by the edge equipment, and relevant information is extracted according to a predefined format and protocol. Analysis of the data by machine learning algorithms can provide deeper, comprehensive data analysis results. The cloud platform can generate various feedback information such as state update, control instruction confirmation, event trigger signal confirmation, decision result feedback and data analysis report so as to meet the requirements of different application scenes; by giving decision processing and data analysis tasks to the cloud platform, cloud computing and storage resources can be fully utilized, and the burden of edge equipment is reduced. The edge device can concentrate on data acquisition, preprocessing and preliminary analysis, while the cloud platform is responsible for more complex and computationally intensive task processing, realizing the cooperative work of the edge and the cloud. Thus, the overall performance of the system can be improved, and the energy consumption and the resource consumption of the edge equipment can be reduced; the cloud platform can receive decision making, control instructions and event triggering signals sent by the edge equipment in real time and generate corresponding feedback information. This enables the system to respond to and process requests from edge devices in a timely manner and supports remote management and monitoring. The cloud platform can provide real-time state update, control instruction confirmation and decision result feedback for the edge equipment, so that a user can know the running condition of the system in time and make corresponding adjustment and decision.
In one embodiment of the invention, the cloud platform receives decision, control instructions and event trigger signals from the edge device and stores them in a data structure; comprising the following steps:
s411: determining a time period for storing the data segment, and acquiring a time stamp of the data from the received decision, control instruction and event trigger signal;
s412: creating a storage structure for each time period, wherein the storage structure comprises a database table, a folder and a file;
s413: the received data is distributed into the corresponding time period according to the time stamp by comparing the time stamp with the starting time and the ending time of the time period;
s414: data categorized into respective time periods is stored into corresponding storage structures.
The working principle of the technical scheme is as follows: the cloud platform determines a time period for which the data is to be stored in segments and obtains a timestamp of the data from the received decisions, control instructions, and event trigger signals. This may be determined according to specific needs and application scenarios, e.g. stored on an hourly, daily, weekly, etc. time period; a corresponding storage structure is created per time period, which may include database tables, folders, files, and the like. The creation of the storage structure can be customized according to different data types and storage requirements so as to ensure the integrity and easy management of the data; by comparing the time stamp of the received data with the start time and the end time of the time period, the data is allocated to the corresponding time period according to the time stamp thereof. Therefore, the data can be classified and stored according to the time period, and subsequent retrieval and management are convenient; data categorized into respective time periods is stored into corresponding storage structures. The decision, control instructions, event trigger signals, etc. data may be stored in a database table or files may be stored in corresponding folders and files.
The technical scheme has the effects that: by storing the received decisions, control instructions, and event trigger signals in a data structure, the cloud platform is able to structure and organize the data. Each time period corresponds to a storage structure, and the time periods can be stored in the forms of database tables, folders, files and the like. Thus, the data can be conveniently managed and inquired, and the maintainability of the system and the efficiency of data analysis are improved; by storing and classifying the data according to time periods, the cloud platform can realize efficient data retrieval and backtracking. Corresponding data can be quickly located and acquired according to a specific time range without traversing the whole data set. The method is very useful for tasks such as data analysis, fault detection, system monitoring and the like, and improves the availability of data and the response speed of the system; storing the data structure in time slots facilitates management and backup of the data. The data in different time periods can be stored independently, so that the data backup and recovery operations are convenient. Meanwhile, the creation of the data structure can be combined with a backup strategy, so that the safety and reliability of the data are ensured; by storing the data in a suitable data structure, the performance and extensibility of the system can be improved. And a proper storage mode such as a database, a folder or a file is selected according to actual requirements, so that the reading and writing operation of data is more efficient and flexible. Meanwhile, according to the division of time periods, the storage structure can be horizontally expanded so as to meet the requirements of large-scale data storage and processing.
In one embodiment of the invention, an edge computing device for machine vision, the device comprises:
and a data acquisition module: the method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network; the vision equipment comprises a camera and a sensor; the visual data includes image data and video data. The network transmission time calculation formula is as follows:
wherein Y is the rated transmission rate of the visual equipment, C is the total data transmission rate of the visual equipment on each sub-channel, L is the data transmission rate of the visual equipment on the sub-channel j during data transmission, n is the number of sub-channels, t is the operation time period of the edge equipment, and d is the gain of the sub-channel.
And the feature extraction module is used for: the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks; the preprocessing comprises image data preprocessing and video data preprocessing; the received image data is preprocessed. Preprocessing may include denoising, image enhancement, resizing, etc. operations to improve the accuracy and effectiveness of subsequent tasks. The received video data is preprocessed. Preprocessing may include frame rate control, video compression, key frame extraction, etc. to meet real-time processing and resource conservation requirements. The characteristics include diversity, complexity, real-time, large data volume, unstructured data, and robustness.
And an analysis and processing module: performing real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results;
an information generation module: the edge equipment sends the generated corresponding decision, control instruction and event trigger signal to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information; the feedback information comprises state update, control instruction confirmation, event trigger signal confirmation, decision result feedback and data analysis report.
The instruction transfer module: and the cloud platform transmits the feedback information back to the edge equipment, and the edge equipment takes corresponding operation or behavior according to the feedback information and transmits the operation or behavior to the execution equipment or system.
The working principle of the technical scheme is as follows: the vision equipment (such as a camera and a sensor) is responsible for collecting image data and video data and transmitting the data to the edge equipment through network transmission; the edge device pre-processes the received visual data. And preprocessing the image data, such as denoising, image enhancement, size adjustment and the like, so as to improve the accuracy and effect of subsequent tasks. Preprocessing video data, such as frame rate control, video compression, key frame extraction and the like, so as to meet the requirements of real-time processing and resource saving; and extracting the characteristics of the preprocessed data by using a machine vision model on the edge equipment. According to the characteristics of the machine vision task, decomposing the task into a plurality of subtasks, and extracting the characteristics of each subtask; the method comprises various tasks such as target detection, classification, segmentation and the like; and analyzing and processing the subtasks in real time through a machine vision model. The model may further calculate and infer the extracted features to obtain specific analysis results. Including target recognition, object tracking, anomaly detection, behavior recognition, etc.; and generating corresponding decision, control instruction and event trigger signals by the edge equipment according to the real-time analysis and processing results. This information may be used for device control, task adjustment, or interaction with other systems. The edge equipment sends the generated information to the cloud platform; and the cloud platform receives the information sent by the edge equipment, and processes and analyzes the information. Through further analysis of the data, the cloud platform generates corresponding feedback information such as equipment control instructions, task optimization suggestions and the like; the edge equipment receives feedback information from the cloud platform and executes corresponding operation or behavior according to the instruction and the suggestion in the feedback information; including device control, such as adjusting camera parameters or sensor settings, and task adjustment, such as adjusting task execution policies based on target detection results.
The technical scheme has the effects that: the real-time machine vision processing can be realized by putting the vision data acquisition, preprocessing and analysis tasks on the edge equipment to finish; data transmission delay is reduced, and analysis and decision making of visual data can be responded more quickly; by preprocessing visual data, extracting features and partially analyzing tasks on the edge equipment, the computing resource and bandwidth requirements on the cloud platform can be reduced. Only key information and decision results need to be transmitted to the cloud platform, so that a large amount of data transmission and calculation cost is reduced; since the visual data processing is mainly performed on the edge device, the dependence on network connection is reduced. Even if the network is unstable or disconnected, the edge equipment can still perform visual analysis and decision to a certain extent, so that the robustness of the system is improved; by performing data processing and decision making on the edge device, the transmission of sensitive data can be reduced, thereby improving the privacy preserving capability. Important data can be processed locally on the equipment, and only necessary results are transmitted to the cloud platform, so that data leakage and privacy risks are reduced; the feedback information generated by the cloud platform can help the edge device to control the device and adjust the task. According to the decision result and state update of the cloud platform, the edge device can automatically adjust the execution mode and priority of the current task, so that more flexible and intelligent system operation is realized. The network transmission time can be calculated according to the transmission rate, the data transmission rate, the number of sub-channels, the gain and other parameters of the visual equipment through the formula. Therefore, the time of data transmission can be optimized, the instantaneity and the response speed are improved, and the transmission time is ensured to be within the available time range of the edge equipment. Meanwhile, the formula has scalability and adaptability, can be adjusted and customized according to specific requirements, and is suitable for different network environments and equipment requirements. Meanwhile, by calculating the respective parameters in the formula, the network transmission time T may be calculated according to the rated transmission rate Y of the vision device, the total data transmission rate C on each sub-channel, and the data transmission rate L on each sub-channel. Therefore, the time of data transmission can be optimized, and the real-time performance and response speed of the data are improved; the presence of the number of sub-channels n and the sub-channel gain d in the formula allows a more accurate calculation of the transmission time. The number and gain of the sub-channels are important factors of the network structure and environment, and by considering the parameters, the transmission strategy and the resource allocation can be better adjusted to achieve the optimal transmission time; the limit condition T of the operation time period T of the edge equipment in the formula is less than or equal to T-nd, so that the transmission time is ensured not to exceed the available time of the edge equipment. Therefore, the data transmission can be ensured to be completed within a limited time, and the task execution delay or failure caused by overlong transmission time is avoided; the parameters in the formula can be adjusted according to specific conditions, such as the number of the subchannels and the setting of the gains, and the formula has good scalability and adaptability. The system can be customized according to different network environments and equipment requirements so as to meet the transmission requirements in different scenes.
An embodiment of the invention is a storage medium having stored thereon a computer program for execution by a processor to implement an edge calculation method for machine vision as described in any of the above.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. An edge computing method for machine vision, the method comprising:
the method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network;
the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks;
performing real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results;
The edge equipment sends the generated corresponding decision, control instruction and event trigger signal to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information;
and the cloud platform transmits the feedback information back to the edge equipment, and the edge equipment takes corresponding operation or behavior according to the feedback information and transmits the operation or behavior to the execution equipment or system.
2. The edge computing method for machine vision as described in claim 1, wherein said vision equipment includes a camera and a sensor; the visual data includes image data and video data.
3. The edge computing method for machine vision according to claim 1, wherein the edge device preprocesses the received vision data and analyzes a machine vision task, decomposes the machine vision task into a plurality of subtasks according to the characteristics of the machine vision task, and performs feature extraction on the subtasks; the subtasks include: target detection task, image classification task, face recognition task and image segmentation task; the feature extraction of the subtasks comprises the following steps:
the target detection task determines the position of an object in an image by using a boundary box, classifies the detected object, determines the category of the object, and tracks the position and the motion trail of the object in continuous frames;
The image classification task extracts features from the images through a convolutional neural network, designs and trains a classifier model, and matches the extracted features with predefined categories to obtain image classification results;
the face recognition task determines the position of a face by detecting the face in an image or a video, and performs alignment operation on the detected face; extracting feature vectors in the face image through a deep learning algorithm; matching the extracted feature vector with the face features in the database to determine the identity or perform verification/identification;
the image segmentation task obtains accurate segmentation results of a plurality of objects in an image by distributing each pixel in the image to different semantic categories and simultaneously segmenting and labeling different objects at the pixel level.
4. A method of edge computation for machine vision according to claim 3, characterized in that the subtasks are feature extracted by parallel processing method, the method comprising:
decomposing the machine vision task into a plurality of subtasks;
according to task division, visual data are decomposed into a plurality of data blocks, and the size and the number of the data blocks are determined according to available parallel computing resources and task complexity;
Equally distributing the data blocks to the parallel computing units according to the performance and task complexity of the computing units by using a task scheduling algorithm;
the parallel computing unit processes the distributed data blocks through a convolutional neural network and performs feature extraction;
after all the parallel computing units finish feature extraction, combining the generated features into integral features through a feature fusion algorithm;
and carrying out next processing on the integral characteristic according to specific task requirements, wherein the next processing comprises dimension reduction, normalization and calibration.
5. The edge computing method for machine vision according to claim 1, wherein the subtasks after feature extraction are analyzed and processed in real time by a machine vision model on the edge device, and corresponding decision, control instruction and event trigger signals are generated according to the real-time analysis and processing results; comprising the following steps:
inputting the subtasks for feature extraction into corresponding subtask models, and analyzing and processing the subtasks for feature extraction in real time through a machine learning algorithm;
the edge device can generate corresponding decisions according to predefined rules or policies;
According to the generated decision, the edge equipment generates a corresponding control instruction, wherein the control instruction is used for controlling related equipment or executing specific operation;
based on the occurrence of a specific event or the detection of a specific target, the edge device generates a corresponding event trigger signal.
6. The edge computing method for machine vision according to claim 1, wherein the edge device sends a decision, a control command and an event trigger signal to a cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information; comprising the following steps:
the cloud platform receives decision, control instructions and event trigger signals from the edge equipment and stores the decision, control instructions and event trigger signals in a data structure;
the cloud platform analyzes the received data and extracts relevant information according to a preset format and protocol; analyzing and predicting decision results through a machine learning algorithm according to different data types;
based on the results of the data parsing and processing, the cloud platform may generate corresponding feedback information.
7. The edge computing method for machine vision as described in claim 6, wherein the feedback information includes status updates, control command acknowledgements, event trigger signal acknowledgements, decision result feedback, and data analysis reporting.
8. The edge computing method for machine vision as described in claim 6, wherein the cloud platform receives decision, control instructions, and event trigger signals from the edge device and stores them in a data structure; comprising the following steps:
determining a time period for storing the data segment, and acquiring a time stamp of the data from the received decision, control instruction and event trigger signal;
creating a storage structure for each time period, wherein the storage structure comprises a database table, a folder and a file;
the received data is distributed into the corresponding time period according to the time stamp by comparing the time stamp with the starting time and the ending time of the time period;
data categorized into respective time periods is stored into corresponding storage structures.
9. An edge computing device for machine vision, the device comprising:
and a data acquisition module: the method comprises the steps of collecting visual data through visual equipment, and transmitting the collected visual data to edge equipment through a network;
and the feature extraction module is used for: the edge equipment pre-processes the received visual data, analyzes a machine visual task, decomposes the machine visual task into a plurality of subtasks according to the characteristics of the machine visual task, and extracts the characteristics of the subtasks;
And an analysis and processing module: performing real-time analysis and processing on the subtasks subjected to feature extraction through a machine vision model on the edge equipment, and generating corresponding decision, control instructions and event trigger signals according to real-time analysis and processing results;
an information generation module: the edge equipment sends the generated corresponding decision, control instruction and event trigger signal to the cloud platform, and the cloud platform processes and analyzes the received information and generates corresponding feedback information;
the instruction transfer module: and the cloud platform transmits the feedback information back to the edge equipment, and the edge equipment takes corresponding operation or behavior according to the feedback information and transmits the operation or behavior to the execution equipment or system.
10. A storage medium having stored thereon a computer program, wherein the program is to be executed by a processor to implement the edge calculation method for machine vision according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311512133.2A CN117389742B (en) | 2023-11-10 | 2023-11-10 | Edge computing method, device and storage medium for machine vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311512133.2A CN117389742B (en) | 2023-11-10 | 2023-11-10 | Edge computing method, device and storage medium for machine vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117389742A true CN117389742A (en) | 2024-01-12 |
| CN117389742B CN117389742B (en) | 2024-05-31 |
Family
ID=89437349
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311512133.2A Active CN117389742B (en) | 2023-11-10 | 2023-11-10 | Edge computing method, device and storage medium for machine vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117389742B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117761444A (en) * | 2024-02-22 | 2024-03-26 | 深圳普泰电气有限公司 | Method and system for monitoring service life of surge protector |
| CN117788047A (en) * | 2024-02-27 | 2024-03-29 | 深圳市亚飞电子商务有限公司 | Commodity demand analysis method and system based on big data |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180284745A1 (en) * | 2016-05-09 | 2018-10-04 | StrongForce IoT Portfolio 2016, LLC | Methods and systems for self-organization of collected data using 3rd party data from a data marketplace in an industrial internet of things environment |
| US20190025813A1 (en) * | 2016-05-09 | 2019-01-24 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for intelligent collection and analysis of vehicle data |
| WO2019109771A1 (en) * | 2017-12-05 | 2019-06-13 | 南京南瑞信息通信科技有限公司 | Power artificial-intelligence visual-analysis system on basis of multi-core heterogeneous parallel computing |
| CN111586091A (en) * | 2020-03-25 | 2020-08-25 | 重庆特斯联智慧科技股份有限公司 | Edge computing gateway system for realizing computing power assembly |
| CN112804188A (en) * | 2020-12-08 | 2021-05-14 | 鹏城实验室 | Scalable vision computing system |
| CN113114758A (en) * | 2021-04-09 | 2021-07-13 | 北京邮电大学 | Method and device for scheduling tasks for server-free edge computing |
| CN113534832A (en) * | 2021-08-03 | 2021-10-22 | 国网江苏省电力有限公司泰州供电分公司 | Unmanned aerial vehicle inspection tracking distribution network line flying method based on edge calculation |
| CN113886094A (en) * | 2021-12-07 | 2022-01-04 | 浙江大云物联科技有限公司 | Resource scheduling method and device based on edge calculation |
-
2023
- 2023-11-10 CN CN202311512133.2A patent/CN117389742B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180284745A1 (en) * | 2016-05-09 | 2018-10-04 | StrongForce IoT Portfolio 2016, LLC | Methods and systems for self-organization of collected data using 3rd party data from a data marketplace in an industrial internet of things environment |
| US20190025813A1 (en) * | 2016-05-09 | 2019-01-24 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for intelligent collection and analysis of vehicle data |
| WO2019109771A1 (en) * | 2017-12-05 | 2019-06-13 | 南京南瑞信息通信科技有限公司 | Power artificial-intelligence visual-analysis system on basis of multi-core heterogeneous parallel computing |
| CN111586091A (en) * | 2020-03-25 | 2020-08-25 | 重庆特斯联智慧科技股份有限公司 | Edge computing gateway system for realizing computing power assembly |
| CN112804188A (en) * | 2020-12-08 | 2021-05-14 | 鹏城实验室 | Scalable vision computing system |
| CN113114758A (en) * | 2021-04-09 | 2021-07-13 | 北京邮电大学 | Method and device for scheduling tasks for server-free edge computing |
| CN113534832A (en) * | 2021-08-03 | 2021-10-22 | 国网江苏省电力有限公司泰州供电分公司 | Unmanned aerial vehicle inspection tracking distribution network line flying method based on edge calculation |
| CN113886094A (en) * | 2021-12-07 | 2022-01-04 | 浙江大云物联科技有限公司 | Resource scheduling method and device based on edge calculation |
Non-Patent Citations (1)
| Title |
|---|
| 卢海峰;顾春华;罗飞;丁炜超;杨婷;郑帅;: "基于深度强化学习的移动边缘计算任务卸载研究", 计算机研究与发展, no. 07, 7 July 2020 (2020-07-07) * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117761444A (en) * | 2024-02-22 | 2024-03-26 | 深圳普泰电气有限公司 | Method and system for monitoring service life of surge protector |
| CN117761444B (en) * | 2024-02-22 | 2024-04-30 | 深圳普泰电气有限公司 | Method and system for monitoring service life of surge protector |
| CN117788047A (en) * | 2024-02-27 | 2024-03-29 | 深圳市亚飞电子商务有限公司 | Commodity demand analysis method and system based on big data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117389742B (en) | 2024-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN117389742B (en) | Edge computing method, device and storage medium for machine vision | |
| US11023707B2 (en) | System and method for selecting a part of a video image for a face detection operation | |
| Zhang et al. | Blockchain-based collaborative edge intelligence for trustworthy and real-time video surveillance | |
| CN110192393A (en) | Large-scale real-time video analysis | |
| CN104254873A (en) | Alert volume normalization in a video surveillance system | |
| CN112950400A (en) | Data processing platform | |
| KR20220044828A (en) | Facial attribute recognition method, device, electronic device and storage medium | |
| CN104657205A (en) | Virtualization-based video content analyzing method and system | |
| CN111553213A (en) | Real-time distributed identity-aware pedestrian attribute identification method in mobile edge cloud | |
| CN113955594A (en) | Elevator control method and device, computer equipment and storage medium | |
| CN113822240A (en) | Method and device for extracting abnormal behaviors from power field operation video data | |
| CN112579824A (en) | Video data classification method and device, electronic equipment and storage medium | |
| JP6797956B2 (en) | Estimator, estimation method and program | |
| CN108596068B (en) | Method and device for recognizing actions | |
| CN112669353B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN114384866B (en) | Data partitioning method based on distributed deep neural network framework | |
| KR102477845B1 (en) | System and method of processing image data for batch processing of big data and computer program for the same | |
| Mokashi et al. | Video Analytics Technologies Available for Surveillance Systems | |
| Alghawli | Computer Vision Technology for Fault Detection Systems Using Image Processing. | |
| CN112311734B (en) | Image feature extraction method of multi-channel video, electronic equipment and storage medium | |
| Lu et al. | IoT-Enabled Streaming Image Analytics With Privacy-Aware Self-Adaptive and Reflective Designs | |
| SanMiguel et al. | A semantic-guided and self-configurable framework for video analysis | |
| JP7575205B2 (en) | Execution device, execution method, and execution program | |
| WO2022259575A1 (en) | Learning device, inference device, learning method, inference method, and program | |
| Sharma | Chapter-7 Edge Computing with Deep Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |