CN112364084B - Visual data processing method and system for deep customization algorithm integration - Google Patents
Visual data processing method and system for deep customization algorithm integration Download PDFInfo
- Publication number
- CN112364084B CN112364084B CN202011291725.2A CN202011291725A CN112364084B CN 112364084 B CN112364084 B CN 112364084B CN 202011291725 A CN202011291725 A CN 202011291725A CN 112364084 B CN112364084 B CN 112364084B
- Authority
- CN
- China
- Prior art keywords
- service
- algorithm
- data
- model
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 198
- 230000000007 visual effect Effects 0.000 title claims abstract description 33
- 238000003672 processing method Methods 0.000 title claims abstract description 8
- 230000010354 integration Effects 0.000 title abstract description 8
- 238000012545 processing Methods 0.000 claims abstract description 61
- 238000000034 method Methods 0.000 claims abstract description 45
- 238000004458 analytical method Methods 0.000 claims abstract description 22
- 238000012856 packing Methods 0.000 claims abstract description 4
- 238000007726 management method Methods 0.000 claims description 57
- 230000008569 process Effects 0.000 claims description 30
- 238000010276 construction Methods 0.000 claims description 26
- 238000012360 testing method Methods 0.000 claims description 23
- 238000013475 authorization Methods 0.000 claims description 20
- 238000013486 operation strategy Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000012217 deletion Methods 0.000 claims description 10
- 230000037430 deletion Effects 0.000 claims description 10
- 230000000977 initiatory effect Effects 0.000 claims description 8
- 239000000725 suspension Substances 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 238000007792 addition Methods 0.000 claims description 5
- 230000004048 modification Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 238000012800 visualization Methods 0.000 claims description 5
- 238000013075 data extraction Methods 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 230000009849 deactivation Effects 0.000 claims description 3
- 230000008676 import Effects 0.000 claims description 3
- 230000006872 improvement Effects 0.000 description 8
- 230000001276 controlling effect Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 238000012896 Statistical algorithm Methods 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data processing, in particular to a visual data processing method and system for deep custom algorithm integration. The method comprises the following steps: s1, packing and uploading a required algorithm through analysis algorithm library management, and configuring basic information of the algorithm; s2, constructing a data service model, processing data by adopting an algorithm, and arranging the data into data service; s3, after the service is arranged, the service is approved through a data service approval module; and S4, managing the service by using the data service management module after the approval is passed. The invention performs visual data processing arrangement by carrying out integrated management on the algorithm. The invention can generate service after arrangement, is convenient for users to use to the greatest extent, and solves the problems of high learning cost and complicated processing of data processing personnel simply and with low cost.
Description
Technical Field
The invention relates to the field of data processing, in particular to a visual data processing method and system for deep custom algorithm integration.
Background
At present, along with the progress of technology, we enter a data-driven era, every organization and enterprise want to find value in data, and the data has become an indispensable part of this new era. Most data processing analysts at present use traditional code writing, namely, data is processed through a python, java and other language custom algorithm or algorithms provided by Spark, flink and other big data processing tools; and SQL statement analysis, namely, processing data by writing SQL statements in a database or a hive data warehouse tool and processing the data by using a self-contained function. However, these data processing methods are either high in learning cost, require learning to write codes, SQL writing methods, and use data processing tools, are cumbersome to use, require copying and modifying processing logic written before when similar data is processed, or have poor reusability, and are quite cumbersome for others, and are not necessarily used by the other party. Generally, the technology adopted for data processing at present causes high cost, time consuming and financial resources due to high learning cost, complicated use, poor reusability and the like.
In the aspect of data processing, most of processing staff processes data by using a traditional code or SQL statement writing mode, and general users need to learn various software such as a hive data warehouse, spark offline calculation, ozzie scheduling and the like, or process the data by using a code writing mode, so that the learning cost and the use cost are high. In order to solve these problems of the data processor, a system capable of integrating various processing algorithms and being simply used by the data processor is required.
The existing solution is that a visual ETL tool Kettle is an ETL tool of a foreign open source, pure java is written and can run on Window, linux, unix, installation is not needed, data extraction is efficient and stable, you are allowed to manage data from different databases, a graphical user environment is provided to describe what you want to do, two script files are arranged in the Kettle, transformation and job are used for completing basic conversion of the data, and job is used for completing control of the whole workflow. However, kettle cannot effectively manage the owned algorithm, and the new algorithm is difficult; the data arranging and converting process is too complicated, and two parts of arrangement and conversion and work are needed; and the data cannot be issued as service after arrangement, so that the data can be conveniently docked. Therefore, the technical scheme has not been applied on a large scale at present.
In the aspect of data processing, no effective system or technical scheme is available to solve the problems of high learning cost, poor algorithm reusability, complicated algorithm use and the like. The processor needs to learn a lot of software and needs to master a lot of algorithms. Currently, the existing data processing software on the market is often insufficient in practicality, is excessively complicated to use, cannot effectively integrate various algorithms, and is less in number of users.
Disclosure of Invention
The invention provides a visual data processing method and a visual data processing system for deep custom algorithm integration, and aims to solve the problems of high learning cost, poor algorithm reusability, complicated algorithm use and the like in the aspect of current data processing.
The invention provides a visual data processing method for deep custom algorithm integration, which comprises the following steps:
s1, packing and uploading a required algorithm through analysis algorithm library management, and configuring basic information of the algorithm;
S2, constructing a data service model, processing data by adopting an algorithm, and arranging the data into data service;
s3, after the service is arranged, the service is approved through a data service approval module;
And S4, managing the service by using the data service management module after the approval is passed.
As a further improvement of the present invention, the step S1 specifically includes:
The analysis algorithm library management provides an algorithm library comprising basic service calculation formulas, algorithm models and general calculation modes, and simultaneously provides an editing entry of a newly added algorithm tool, a user packages the needed algorithm, performs algorithm registration on an algorithm management page, and configures basic information comprising algorithm attributes, algorithm detailed information and algorithm names.
As a further improvement of the present invention, the analysis algorithm library management in the step S1 includes the following sub-execution blocks:
s11, algorithm list management: displaying the currently added algorithm in a list form, and performing classified management;
s12, algorithm query: searching and inquiring according to the algorithm name and the creation time;
s13, a new algorithm: the user adds an algorithm, fills in detailed information, and imports a local algorithm package;
s14, modifying an algorithm: the user changes the detailed information of the algorithm and re-uploads the algorithm package;
S15, removing algorithm: deleting the unused algorithm, and reminding before deleting; meanwhile, prompts which are not allowed to be deleted are made for the occupied algorithm;
S16, enabling/disabling an algorithm: and (5) carrying out an activation and deactivation operation on the algorithm.
As a further improvement of the present invention, the step S2 specifically includes: after uploading the algorithm, the user processes the data through a data service model construction, the data service model construction provides a visual page, and the registered algorithm is introduced through a dragging mode to process the data; comprises the following substeps:
S21, pulling in a data extraction operator, and selecting a data source;
s22, adding a processing algorithm uploaded by a user, editing configuration attributes of the algorithm, and constructing a service model;
s23, performing operation test on the constructed service model, connecting a data source with a processing algorithm, and storing the data source;
S24, publishing the connection result.
As a further improvement of the present invention, the processing of the data by the data service model construction in the step S22 includes the following steps:
S22a, model basic information registration: the registration information comprises a service model name, a service model description, model authority setting, an applicable region range and an applicable time stage;
s22b, model label setting: using the label setting to serve a specified label to the data;
s22c, setting model parameters: intuitively displaying the model parameter configuration result by using a visualization technology, wherein the parameter setting operation comprises the addition, deletion and modification of parameters;
S22d, adding a business flow: providing a data service subject library API or forming an API sharing interface according to user requirements, and modularly processing the service flow by adding a child node; and simultaneously configuring the attribute of the node of the data processing flow.
As a further improvement of the present invention, the step S23 specifically includes the following substeps:
S23a, evaluating service model integrity: and testing the integrity of the service model, and testing information including the task allocation feedback condition, the flow sub-nodes, the data sources, the integrity of the logic rules and the continuity.
S23b, service model test operation: if the integrity evaluation of the service model is reasonable, performing test operation, connecting all the business processes, process sub-nodes, algorithms and logic rules, generating a new data form for storage, and forming computer readable information.
As a further improvement of the present invention, the step S24 specifically includes:
submitting the audited service model to service release management, storing the audited service model as shared service, and editing service information comprising basic information, service operation strategy and service authority configuration for release.
As a further improvement of the present invention, the process of performing approval by the data service approval module in step S3 specifically includes:
And issuing approval items according to the user permission display service, performing approval passing and approval failure operations according to the user application content, and filling in approval failure reasons when the approval result is failure.
As a further improvement of the present invention, the process of managing the service by the data service management module in the step S4 includes
S41, controlling the generated service, and specifically performing the following sub-execution plates:
S41. service information preview: all relevant information and authorization information of the preview service, wherein the information comprises service metadata and model metadata, such as input and output parameters, service addresses and calling authorities;
s41b. service initiation: initiating a service in the suspension and sharing of related authorization information;
S41c. service suspension: temporarily stopping sharing of the selected service and related authorization information, and reserving the service;
s41d, service under-frame: the selected service and related authorization information are put off the shelf, and the service version is reserved;
s41e, service deletion: deleting the selected service and related authorization information;
S42, managing an operation strategy, wherein the operation strategy comprises the following sub-execution plates:
S42a, periodically operating: setting service automatic running time, data source updating mode and updating frequency, carrying out service timing call and saving running results, and screening and providing result set service call according to user provided parameters;
S42b, calling operation: according to the data source and data logic stored in the service, automatically extracting data and calculating and returning the service result in real time according to parameters provided by the user in the process of calling the user.
The invention also provides a visual data processing system for deeply customizing the algorithm integration, which comprises an analysis algorithm library management module, a data service model construction module, a data service approval module and a data service management module; the analysis algorithm library management module, the data service model construction module, the data service approval module and the data service management module are sequentially connected;
the analysis algorithm library management module: providing basic information of basic general purpose, and simultaneously providing an editing inlet of a newly added algorithm tool;
The data service model construction module: providing a model construction inlet and maintaining the constructed model, wherein the model mainly comprises data service model construction, service model operation test and data service model release;
The data service approval module: approving the released model and judging whether the service can be used externally or not;
The data service management module: and controlling the generated service and managing the operation strategy.
The beneficial effects of the invention are as follows: the algorithm is integrated and managed, and the data processing arrangement is visualized. The invention can generate service after arrangement, is convenient for users to use to the greatest extent, and solves the problems of high learning cost and complicated processing of data processing personnel simply and with low cost.
Drawings
FIG. 1 is a flow chart of a method of processing visual data in the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent.
Embodiment one:
The technical scheme mainly carries out algorithm configuration through an algorithm management module, and carries out visual data processing according to business application scenes aiming at the data which are collected. The technical aspect adopts a B/S architecture, page design supports responsive adaptation mobile terminal browsing, front-end and back-end separation technical architecture is adopted, the front-end is based on a VUE.JS technical system, the server is developed based on SpringBoot frames, the server can be used as independent service and can be seamlessly accessed into a frame for Spring Cloud micro-service management, data storage adopts a distributed frame hadoop3.X, and storage based on hadoop3.X, hive query and Spark calculation are combined, so that more convenient expansion capacity and excellent performance are brought, service is not unavailable as the data volume increases, and data return in a tolerable time can be ensured.
As shown in fig. 1, the method for processing visual data integrated by deep customization algorithm of the invention comprises the following steps:
s1, packing and uploading a required algorithm through analysis algorithm library management, and configuring basic information of the algorithm;
S2, constructing a data service model, processing data by adopting an algorithm, and arranging the data into data service;
s3, after the service is arranged, the service is approved through a data service approval module;
And S4, managing the service by using the data service management module after the approval is passed.
The step S1 specifically includes:
the analysis algorithm library management provides an algorithm library comprising basic service calculation formulas, algorithm models, general calculation modes and the like, simultaneously provides an editing entry of a newly added algorithm tool, packages the needed algorithm, registers the algorithm on an algorithm management page, configures basic information comprising algorithm attributes, algorithm detailed information, algorithm names and the like, and finally provides the basic information for visual data modeling.
The analysis algorithm library management in step S1 includes the following sub-execution blocks:
s11, algorithm list management: displaying the currently added algorithm in a list form, and performing classified management;
s12, algorithm query: searching and inquiring according to the algorithm name and the creation time;
s13, a new algorithm: the user adds an algorithm, fills in detailed information, and imports a local algorithm package;
s14, modifying an algorithm: the user changes the detailed information of the algorithm and re-uploads the algorithm package;
S15, removing algorithm: deleting the unused algorithm, and reminding before deleting; meanwhile, prompts which are not allowed to be deleted are made for the occupied algorithm;
S16, enabling/disabling an algorithm: and (5) carrying out an activation and deactivation operation on the algorithm.
The step S2 specifically includes: after uploading the algorithm, the user processes the data through a data service model construction, the data service model construction provides a visual page, and the registered algorithm is introduced through a dragging mode to process the data; comprises the following substeps:
S21, pulling in a data extraction operator, and selecting a data source;
s22, adding a processing algorithm uploaded by a user, editing configuration attributes of the algorithm, and constructing a service model;
s23, performing operation test on the constructed service model, connecting a data source with a processing algorithm, and storing the data source;
S24, publishing the connection result.
In step S22, the data is processed by constructing a data service model for the uploaded algorithm, which specifically includes the following sub-steps:
S22a, model basic information registration: the registration information comprises a service model name, a service model description, model authority setting, an applicable region range and an applicable time stage;
s22b, model label setting: using the label setting to serve a specified label to the data;
s22c, setting model parameters: intuitively displaying the model parameter configuration result by using a visualization technology, wherein the parameter setting operation comprises the addition, deletion and modification of parameters;
S22d, adding a business flow: providing a data service subject library API or forming an API sharing interface according to user requirements, and modularly processing the service flow by adding a child node; and simultaneously configuring the attribute of the node of the data processing flow.
The step S23 specifically includes the following substeps:
s23a, evaluating service model integrity: testing the integrity of the service model, and testing information including the task distribution feedback condition, the flow sub-node, the data source, the integrity and the continuity of the logic rule;
s23b, service model test operation: if the integrity evaluation of the service model is reasonable, performing test operation, connecting all the business processes, process sub-nodes, algorithms and logic rules, generating a new data form for storage, and forming computer readable information.
The step S24 specifically includes:
submitting the audited service model to service release management, storing the audited service model as shared service, and editing service information comprising basic information, service operation strategy and service authority configuration for release.
The process of performing approval by the data service approval module in step S3 specifically includes:
And issuing approval items according to the user permission display service, performing approval passing and approval failure operations according to the user application content, and filling in approval failure reasons when the approval result is failure. The arranged data model can be issued as the service only after approval, and the authorized user is required to approve the data model, so that the reasonable use of the service is ensured.
Wherein, the process of managing the service by the data service management module in the step S4 comprises
S41, controlling the generated service, and specifically performing the following sub-execution plates:
S41. service information preview: all relevant information and authorization information of the preview service, wherein the information comprises service metadata and model metadata, such as input and output parameters, service addresses and calling authorities;
s41b. service initiation: initiating a service in the suspension and sharing of related authorization information;
S41c. service suspension: temporarily stopping sharing of the selected service and related authorization information, and reserving the service;
s41d, service under-frame: the selected service and related authorization information are put off the shelf, and the service version is reserved;
s41e, service deletion: the selected service and related authorization information are deleted.
S42, managing an operation strategy, wherein the operation strategy comprises the following sub-execution plates:
S42a, periodically operating: setting service automatic running time, data source updating mode and updating frequency, carrying out service timing call and saving running results, and screening and providing result set service call according to user provided parameters;
S42b, calling operation: according to the data source and data logic stored in the service, automatically extracting data and calculating and returning the service result in real time according to parameters provided by the user in the process of calling the user.
The approved service can be managed on a data service management page, the data service management mainly controls the generated service and manages the operation strategy, and the service can be freely started and stopped and the service usable time period can be regulated.
The method of the invention uses analysis algorithm library management as an entrance of algorithm integration, and can effectively manage and integrate the algorithm; through the visual process construction, the method is convenient for users to use, and the learning cost is reduced; the data service release can directly release the processed data as the service, and is convenient for users to use.
The invention provides a method for inheriting various algorithms, can visually arrange the data processor, finally forms data service, and has simple use and strong practicability. The algorithm is integrated and managed in a simple mode, visual data are processed and arranged, and the processed visual data are released as services, so that the use cost is greatly reduced.
Embodiment two:
The invention provides a visual data processing system integrated with deep customization algorithm, which comprises an analysis algorithm library management module, a data service model construction module, a data service approval module and a data service management module; the analysis algorithm library management module, the data service model construction module, the data service approval module and the data service management module are sequentially connected.
Analysis algorithm library management module: providing basic business calculation formulas, algorithm models, general calculation modes and the like which are basic and general, and simultaneously providing an editing inlet of a newly added algorithm tool. The algorithm comprises a Hive extraction algorithm, a count algorithm, a Hive query algorithm, a data summarization algorithm, a statistical algorithm, a calculation algorithm, a Spark algorithm, a space fusion calculation algorithm provided by a basic support platform algorithm set and the like. The specific functions are as follows:
1) Algorithm list management: and displaying the currently added algorithm in a list form, and performing classification management. The list presentation information includes an algorithm name, an algorithm description, creation time, current status (enabled/disabled), and the like. The list is arranged in reverse order according to the creation time and supports the page turning function.
2) Algorithm query: support for search queries based on algorithm names and creation time.
3) The new algorithm is as follows: and supporting a user to add an algorithm, filling detailed information, and supporting importing a local algorithm package.
4) Modifying an algorithm: and supporting the change of the detailed information of the algorithm by the user and the re-uploading of the algorithm package.
5) Removal algorithm: and supporting an algorithm for deleting useless, and reminding before deleting. And simultaneously, aiming at the occupied algorithm, a prompt that deletion is not allowed is made.
6) An enable/disable algorithm: and supporting the operation of enabling and disabling the algorithm. The start-stop of the algorithm directly affects the operation of the data service model.
The data service model building module: providing a model construction entrance and maintaining the constructed model, wherein the model mainly comprises data service model construction, service model operation test and data service model release.
Wherein the data service model construction comprises:
1) Model basic information registration: the registration information includes service model names, service model descriptions, model authority settings, applicable region ranges, applicable time periods, and the like.
2) Model label setting: with tag settings, a specific tag, like a descriptive message, is served to the data, allowing for additional tags and multi-tag selection.
3) Model parameter setting: and visually displaying the model parameter configuration result by using a visualization technology, wherein the main functions comprise addition, deletion and modification of parameters.
4) Adding a business flow: providing a data service subject library API or forming an API sharing interface according to the user requirement, and modularly processing the service flow by adding a sub-node (algorithm). While the attributes of the nodes of the data processing flow may be configured.
Service model operation test the service model operation test is carried out aiming at the constructed service model, and the service model operation test comprises the following functions:
1) Service model integrity assessment: and testing the integrity of the service model, wherein the integrity comprises information such as task distribution feedback conditions, flow sub-nodes, data sources, logic rule integrity, continuity and the like.
2) Service model commissioning: if the integrity evaluation of the service model is reasonable, the test operation can be performed, and the operation connects all the business processes, process sub-nodes, algorithms and logic rules, generates a new data form for storage, and forms computer readable information.
And (3) issuing a data service model: submitting the checked service model to service release management, editing information such as basic information, service operation strategy, service authority configuration and the like of the service for sharing the service, and releasing the information.
And the data service approval module: and the published model is approved, and whether the service can be used externally is judged by one step, so that the authorized user needs to be confirmed. The module displays all tasks to be and already done by the user. And displaying the service release approval matters according to the user rights. And carrying out approval passing and approval failure operation according to the application content of the user. And when the approval result is failure, filling in the reason of failure of the approval.
And the data service management module: the generated service controls and manages the operation strategy, and the specific content is as follows:
1) Service operation management:
service information preview: all relevant information and authorization information of the preview service, including service metadata, model metadata and the like, such as input and output parameters, service addresses, calling authorities and the like;
service initiation: initiating a service in the suspension and sharing of related authorization information;
service suspension: temporarily stopping sharing of the selected service and related authorization information, and reserving the service;
service is put down: the selected service and related authorization information are put off the shelf, and the service version is reserved;
service deletion: the selected service and related authorization information are deleted.
2) Service operation policy management: service operation policy management is to manage the service calling mechanism. The specific functions are as follows:
and (3) periodically operating: setting service automatic running time, data source updating mode and updating frequency, carrying out service timing call and saving running results, and screening and providing result set service call according to user provided parameters;
calling and running: according to the data source and data logic stored in the service, automatically extracting data and calculating and returning the service result in real time according to parameters provided by the user in the process of calling the user.
The system of the invention is a visual data processing tool which is specially customized for each data processing requirement and service. Through the customization of the bottom layer to the data processing algorithm, the ETL and the ETL process can be easily constructed without codes, the data can be easily introduced, moved, prepared, converted and processed, and the data modeling can be completed in an intuitive visual environment. The method is a powerful support for realizing multi-source data fusion and multi-element service fusion, and can improve quality and efficiency in the process of providing data service capability to the outside. Through the mode of customizing algorithm and visual processing data, the problems of high learning cost, complicated use, poor reusability and the like in the data processing process are well solved, convenience can be provided for data processing personnel, and the use cost is reduced.
The invention can directly generate data service, which is convenient for data butt joint; the invention has low learning cost, and the visual construction service can quickly and simply construct a processing flow; the analysis algorithm library management can effectively manage and integrate the algorithms.
The foregoing is a further detailed description of the invention in connection with the preferred embodiments, and it is not intended that the invention be limited to the specific embodiments described. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the invention, and these should be considered to be within the scope of the invention.
Claims (8)
1. The visual data processing method integrated by the deep customization algorithm is characterized by comprising the following steps of:
s1, packing and uploading a required algorithm through analysis algorithm library management, and configuring basic information of the algorithm;
S2, constructing a data service model, processing data by adopting an algorithm, and arranging the data into data service;
s3, after the service is arranged, the service is approved through a data service approval module;
s4, managing the service by using the data service management module after the approval is passed;
The step S2 specifically includes: after uploading the algorithm, the user processes the data through a data service model construction, the data service model construction provides a visual page, and the registered algorithm is introduced through a dragging mode to process the data; comprises the following substeps:
S21, pulling in a data extraction operator, and selecting a data source;
s22, adding a processing algorithm uploaded by a user, editing configuration attributes of the algorithm, and constructing a service model;
s23, performing operation test on the constructed service model, connecting a data source with a processing algorithm, and storing the data source;
S24, publishing the connection result;
in the step S22, the data is processed through the data service model construction for the uploaded algorithm, which specifically includes the following sub-steps:
S22a, model basic information registration: the registration information comprises a service model name, a service model description, model authority setting, an applicable region range and an applicable time stage;
s22b, model label setting: using the label setting to serve a specified label to the data;
S22c, setting model parameters: intuitively displaying the model parameter configuration result by using a visualization technology, wherein the parameter setting operation comprises the addition, deletion and modification of parameters;
S22d, adding a business flow: providing a data service subject library API or forming an API sharing interface according to user requirements, and modularly processing the service flow by adding a child node; and simultaneously configuring the attribute of the node of the data processing flow.
2. The method for processing visual data integrated with deep customization algorithm according to claim 1, wherein in step S1, the method specifically comprises:
The analysis algorithm library management provides an algorithm library comprising basic service calculation formulas, algorithm models and general calculation modes, and simultaneously provides an editing entry of a newly added algorithm tool, a user packages the needed algorithm, performs algorithm registration on an algorithm management page, and configures basic information comprising algorithm attributes, algorithm detailed information and algorithm names.
3. The method for processing visual data integrated with deep custom algorithm according to claim 2, wherein the analysis algorithm library management in step S1 comprises the following sub-execution blocks:
s11, algorithm list management: displaying the currently added algorithm in a list form, and performing classified management;
s12, algorithm query: searching and inquiring according to the algorithm name and the creation time;
s13, a new algorithm: the user adds an algorithm, fills in detailed information, and imports a local algorithm package;
s14, modifying an algorithm: the user changes the detailed information of the algorithm and re-uploads the algorithm package;
S15, removing algorithm: deleting the unused algorithm, and reminding before deleting; meanwhile, prompts which are not allowed to be deleted are made for the occupied algorithm;
S16, enabling/disabling an algorithm: and (5) carrying out an activation and deactivation operation on the algorithm.
4. The method for processing visual data integrated with deep custom algorithm according to claim 1, wherein said step S23 comprises the following steps:
s23a, evaluating service model integrity: testing the integrity of the service model, and testing information including the task distribution feedback condition, the flow sub-node, the data source, the integrity and the continuity of the logic rule;
s23b, service model test operation: if the integrity evaluation of the service model is reasonable, performing test operation, connecting all the business processes, process sub-nodes, algorithms and logic rules, generating a new data form for storage, and forming computer readable information.
5. The method for processing visual data integrated with deep customization algorithm according to claim 1, wherein the step S24 specifically includes:
submitting the audited service model to service release management, storing the audited service model as shared service, and editing service information comprising basic information, service operation strategy and service authority configuration for release.
6. The method for processing visual data integrated with deep customization algorithm according to claim 1, wherein the step S3 of the data service approval module executing the approval process specifically includes:
And issuing approval items according to the user permission display service, performing approval passing and approval failure operations according to the user application content, and filling in approval failure reasons when the approval result is failure.
7. The method for processing visual data integrated with deep custom algorithm according to claim 1, wherein the data service management module in step S4 manages services comprising
S41, controlling the generated service, and specifically performing the following sub-execution plates:
S41. service information preview: all relevant information and authorization information of the preview service, wherein the information comprises service metadata and model metadata, such as input and output parameters, service addresses and calling authorities;
s41b. service initiation: initiating a service in the suspension and sharing of related authorization information;
S41c. service suspension: temporarily stopping sharing of the selected service and related authorization information, and reserving the service;
s41d, service under-frame: the selected service and related authorization information are put off the shelf, and the service version is reserved;
s41e, service deletion: deleting the selected service and related authorization information;
S42, managing an operation strategy, wherein the operation strategy comprises the following sub-execution plates:
S42a, periodically operating: setting service automatic running time, data source updating mode and updating frequency, carrying out service timing call and saving running results, and screening and providing result set service call according to user provided parameters;
S42b, calling operation: according to the data source and data logic stored in the service, automatically extracting data and calculating and returning the service result in real time according to parameters provided by the user in the process of calling the user.
8. The visual data processing system integrated with the deep customization algorithm is characterized by comprising an analysis algorithm library management module, a data service model construction module, a data service approval module and a data service management module; the analysis algorithm library management module, the data service model construction module, the data service approval module and the data service management module are sequentially connected;
the analysis algorithm library management module: providing basic information of basic general purpose, and simultaneously providing an editing inlet of a newly added algorithm tool;
The data service model construction module: providing a model construction inlet and maintaining the constructed model, wherein the model mainly comprises data service model construction, service model operation test and data service model release;
The data service approval module: approving the released model and judging whether the service can be used externally or not;
The data service management module: controlling the generated service and managing the operation strategy;
wherein the data service model construction comprises:
1) Model basic information registration: the registration information comprises a service model name, a service model description, model authority setting, an applicable region range and an applicable time stage;
2) Model label setting: using tag setting to service a specific tag for data, similar to a description message, allowing for the selection of newly added tags and multiple tags;
3) Model parameter setting: visually displaying the model parameter configuration result by using a visualization technology, wherein the main functions comprise addition, deletion and modification of parameters;
4) Adding a business flow: providing a data service subject library API or forming an API sharing interface according to user requirements, and modularly processing a service flow by adding a sub-node (algorithm); and simultaneously configuring the attribute of the node of the data processing flow.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011291725.2A CN112364084B (en) | 2020-11-18 | 2020-11-18 | Visual data processing method and system for deep customization algorithm integration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011291725.2A CN112364084B (en) | 2020-11-18 | 2020-11-18 | Visual data processing method and system for deep customization algorithm integration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112364084A CN112364084A (en) | 2021-02-12 |
| CN112364084B true CN112364084B (en) | 2024-10-22 |
Family
ID=74532507
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011291725.2A Active CN112364084B (en) | 2020-11-18 | 2020-11-18 | Visual data processing method and system for deep customization algorithm integration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112364084B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538356A (en) * | 2021-07-08 | 2021-10-22 | 广东工业大学 | Online high-concurrency multifunctional material defect automatic marking system and method |
| CN114385136B (en) * | 2021-12-29 | 2022-11-22 | 武汉达梦数据库股份有限公司 | Flow decomposition method and device for running ETL (extract transform load) by Flink framework |
| CN114461348A (en) * | 2022-02-14 | 2022-05-10 | 中冶赛迪重庆信息技术有限公司 | Interactive management platform for algorithmic services |
| CN116028566B (en) * | 2023-02-21 | 2023-08-22 | 中国科学院空间应用工程与技术中心 | Spacecraft data analysis system, method, medium and equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036365A (en) * | 2014-06-26 | 2014-09-10 | 浪潮软件股份有限公司 | Method for constructing enterprise-level data service platform |
| CN106681882A (en) * | 2015-11-06 | 2017-05-17 | 上海瑞致软件有限公司 | IT-service concentrated monitoring and managing system based on Apriori algorithm |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8196047B2 (en) * | 2009-01-20 | 2012-06-05 | Microsoft Corporation | Flexible visualization for services |
| KR101326985B1 (en) * | 2011-09-27 | 2013-11-14 | 이메디정보기술(주) | Method and apparatus for developing, distributing and executing object-wise dynamic compileless programs |
| CN108764808B (en) * | 2018-03-29 | 2020-03-27 | 北京九章云极科技有限公司 | Data analysis processing system and online model deployment method thereof |
| CN111177227B (en) * | 2020-01-17 | 2022-11-08 | 国网福建省电力有限公司 | Power data self-service analysis system and decision application migration method |
-
2020
- 2020-11-18 CN CN202011291725.2A patent/CN112364084B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036365A (en) * | 2014-06-26 | 2014-09-10 | 浪潮软件股份有限公司 | Method for constructing enterprise-level data service platform |
| CN106681882A (en) * | 2015-11-06 | 2017-05-17 | 上海瑞致软件有限公司 | IT-service concentrated monitoring and managing system based on Apriori algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112364084A (en) | 2021-02-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112364084B (en) | Visual data processing method and system for deep customization algorithm integration | |
| CN107819824B (en) | Urban data opening and information service system and service method | |
| CN112668295B (en) | Management system and method based on custom form and active workflow | |
| US9800675B2 (en) | Methods for dynamically generating an application interface for a modeled entity and devices thereof | |
| US9449034B2 (en) | Generic ontology based semantic business policy engine | |
| JP4676199B2 (en) | Systems and methods for integrating, managing, and coordinating customer activities | |
| US7574379B2 (en) | Method and system of using artifacts to identify elements of a component business model | |
| US10296305B2 (en) | Method and device for the automated production and provision of at least one software application | |
| US7917344B2 (en) | Enterprise multi-program process development and integration process | |
| US20060143220A1 (en) | Software application framework using meta-data defined object definitions | |
| US20080270477A1 (en) | Workflow method, system, and data structure | |
| US11783254B2 (en) | Method and system for implementing an adaptive data governance system | |
| US20120246115A1 (en) | Folder structure and authorization mirroring from enterprise resource planning systems to document management systems | |
| JP2005525658A (en) | System and method for representing and editing multidimensional data | |
| WO2016118940A1 (en) | Systems and methods for automatically generating application software | |
| US9558215B2 (en) | Governing information | |
| US20130144880A1 (en) | Business partner grouping | |
| US20090094577A1 (en) | Generating data access operations based on a data model using a data services model | |
| Reschenhofer et al. | Lessons learned in aligning data and model evolution in collaborative information systems | |
| CN113919680A (en) | Method for constructing management information system based on general tasks | |
| US20240256576A1 (en) | Data processing system with manipulation of logical dataset groups | |
| US20150046286A1 (en) | Product content lifecycle and product features | |
| CN117216042A (en) | Construction method and device of data standardization platform | |
| US11526895B2 (en) | Method and system for implementing a CRM quote and order capture context service | |
| JPH1083420A (en) | Model-based operation support system and method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |