CN109857673B - Control identification method and device - Google Patents
Control identification method and device Download PDFInfo
- Publication number
- CN109857673B CN109857673B CN201910138832.2A CN201910138832A CN109857673B CN 109857673 B CN109857673 B CN 109857673B CN 201910138832 A CN201910138832 A CN 201910138832A CN 109857673 B CN109857673 B CN 109857673B
- Authority
- CN
- China
- Prior art keywords
- control
- identified
- icon
- information
- identifying
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a control identification method, which comprises the following steps: determining a target interface where a control to be identified is located; identifying the control to be identified based on the control information of the target interface; the control information comprises at least two of control tree, text information and icon information. By adopting the embodiment of the invention, the control identification accuracy can be improved.
Description
Technical Field
The invention relates to the field of terminals, in particular to a control identification method and device.
Background
Before an Application (APP) is released, functions and performances of the APP need to be tested, running conditions of the APP need to be tested on multiple terminals, whether the functions of the APP meet expectations or not is checked, and compatibility on different terminals is achieved.
In the application program test, a series of test script steps need to be executed, wherein the test script step is specifically to perform relevant operations such as clicking, sliding, long pressing and the like on the control corresponding to each test script step, and because the control is identified, the problem of low identification accuracy of the control exists, the application program test is further influenced.
Therefore, it is necessary to provide a control identification method capable of improving the control identification accuracy.
Disclosure of Invention
The embodiment of the invention provides a control identification method, which aims to solve the problem of low control identification accuracy.
In order to solve the technical problem, the invention is realized as follows:
in a first aspect, a control identification method is provided, where the method includes:
determining a target interface where a control to be identified is located;
identifying the control to be identified based on the control information of the target interface;
the control information comprises at least two of control tree, text information and icon information.
In a second aspect, an apparatus for identifying a control is provided, the apparatus comprising:
the determining module is used for determining a target interface where the control to be identified is located;
the identification module is used for identifying the control to be identified based on the control information of the target interface;
the control information comprises at least two of control tree, text information and icon information.
In a third aspect, a terminal device is provided, which includes: a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the method according to the first aspect.
In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method according to the first aspect.
In the embodiment of the invention, the control to be recognized is recognized based on the control information of the target interface where the control to be recognized is located, so that the control recognition accuracy can be improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a control identification method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a method for identifying a control based on a control tree according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a method for identifying a control based on text information according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for identifying a control based on icon information according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating a control identification method according to another embodiment of the present invention;
fig. 7 is a schematic structural diagram of a control identifying apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
An application scenario of the present invention is exemplarily illustrated with reference to fig. 1.
The application scene comprises the following steps: the test platform 100 and the terminal device 102 carrying the test object, a first control to be recognized 104 and a second control to be recognized 106 are displayed on a user interface on the terminal device 102, wherein:
in the automatic test process, the test platform 100 analyzes the test script and determines the identification of the target control as the test object; identifying the target control from the user interface based on the control tree, the text information and the icon information; the control tree comprises an identifier or a control to be identified in a text form, the text information comprises a control to be identified in a text form (such as the second control 106 to be identified), and the icon information comprises a control to be identified in an icon form (such as the first control 104 to be identified).
Therefore, the target control can be continuously identified in the automatic test process, and the script step corresponding to the target control is executed, so that the aim of executing the whole automatic test script is fulfilled.
The automatic test generally refers to the automation of software test, the software test is to run a system or an application program under a preset condition, and evaluate a running result, wherein the preset condition comprises a normal condition and an abnormal condition; the automatic test script is a series of instructions for specifying automatic test, and each instruction corresponds to each test script step; controls refer to the encapsulation of data and methods; the control tree can visually represent the hierarchical relationship among the controls; the text information is all texts in the current user interface; the icon information is all icons in the current user interface.
Fig. 2 is a schematic flowchart of an icon recognition method according to an embodiment of the present invention, where the method may be executed by a terminal device in an application scenario corresponding to fig. 1, and referring to fig. 2, the method may specifically include the following steps:
step 202: and determining a target interface where the control to be recognized is located.
The target interface is a user interface where a control to be identified is located on the terminal device, and the test platform can determine the user interface where the control to be identified is located.
Step 204: identifying the control to be identified based on the control information of the target interface;
the control information comprises at least two of control tree, text information and icon information.
It should be noted that, when the control information includes a control tree and text information, one implementation manner of step 204 may be:
step S1: identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
the identifier may be a preset ID (identifier), the control to be recognized in the form of the identifier may be "124", etc., the control to be recognized in the form of text may be "Close", etc., the control tree may further include the control to be recognized in the form of an icon, and the control to be recognized in the form of an icon may be "<' or
And the like.
Further, referring to fig. 3, one implementation of step S1 may be:
step S102: and searching a target control in the control tree of the target interface according to a preset control tree searching strategy.
The target control is a control corresponding to a certain test step in the automatic test process, and the preset control tree search strategy may specifically be exemplified as follows:
determining the preferred attribute of the target control from a plurality of attributes of the target control; searching a target control according to the preferred attribute; the priority attribute may be specified by a tester preselection, and the attribute of the target control may be a control type, an identifier of the control, and the like.
Step S104: it is determined whether a control to be identified is found.
If yes, go to step S106; if not, executing step S108;
it should be noted that, because the attributes of the controls may be missing, or different controls have the same attribute, when the target control is searched in the control tree according to the preferred attribute of the target control, situations may occur in which the control cannot be found, one control is found, or multiple controls are found. For ease of description, the one or more controls found may be referred to as controls to be identified.
Step S104 may specifically be exemplified as:
the target control may be a "search button," the identifier of which may be "1234," and the icon may be
The control type may be a type A control, and the text may be "search"; the target control can be searched in the control tree of the target interface according to the identifier of the target control, and as the identifier "1234" of the target control is possibly lost, the target control cannot be searched through the identifier of the target control; and searching a target control in the control tree of the target interface according to the control type of the target control, wherein one or more controls to be identified can be found because the A-type controls in the control tree can be one or more, and identifying the controls to be identified when the one or more controls to be identified are found.
Step S106: determining whether the control to be identified is in the form of an identifier;
if yes, go to step S110; if not, go to step S114;
wherein the identifier form is a display form of the control on the control tree, the display form of the control can also be a text form or an icon form, for example, the return button can be displayed as "223" in the identifier form, or "Close" in the text form, or "□" in the icon form.
Step S110: it is determined whether the control to be recognized in the form of the identifier matches the target control in the form of the identifier.
If yes, go to step S112; if not, go to step S114;
step S110 may specifically be exemplified by:
determining whether the matching degree of the control to be identified in the identifier form and the target control in the identifier form is not lower than a first preset matching threshold, and if not, matching; and if the matching is lower than the first preset matching threshold value, the matching is not performed.
Step S112: identifying the control to be identified in the identifier form as a target control;
step S114: determining whether the control to be recognized is in a text form;
if yes, go to step S116; if not, executing step S120;
step S116: determining whether the control to be recognized in the text form is matched with the target control in the text form;
if yes, go to step S118; if not, executing step S120;
step S116 may specifically be exemplified by:
determining whether the matching degree of the control to be recognized in the text form and the target control in the text form is not lower than a second preset matching threshold, and if not, matching; and if the matching threshold is lower than the second preset matching threshold, the matching is not performed.
Step S118: identifying the control to be identified in the text form as a target control;
step S120: determining whether the control to be identified in the icon form is matched with the target control in the icon form;
if yes, go to step S122;
step S120 may specifically be exemplified by:
determining whether the matching degree of the control to be identified in the icon form and the target control in the icon form is not lower than a third preset matching threshold, and if not, matching; and if the matching threshold value is lower than the third preset matching threshold value, the matching is not performed.
Step S122: identifying the control to be identified in the icon form as a target control;
step S108: determining whether a control to be identified matched with a target control in an icon form exists in a plurality of controls to be identified in the icon form;
if yes, go to step S124;
step S108 may specifically be exemplified by:
when the plurality of controls to be recognized are found, the matching degree of each control to be recognized in the controls to be recognized in the icon form and the target control in the icon form is determined, if the matching degree is not lower than a third preset matching threshold value, the control to be recognized matched with the target control exists, and otherwise, the control to be recognized matched with the target control does not exist.
Step S124: and identifying the control to be identified in the icon form with the highest matching degree in the matched controls to be identified as the target control.
Step S124 may specifically be exemplified by:
if the third preset matching threshold is 90%, the matching degrees of the matched controls to be recognized are respectively 92%, 95% and 98%, and the controls to be recognized with the matching degree of 98% are the target controls.
Step S2: and if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form.
Further, referring to fig. 4, one implementation of step S2 may be:
step S202: in the text information, identifying a control to be identified by adopting an OCR (Optical Character Recognition) mode, wherein the text information comprises the control to be identified in a text form;
step S204: determining whether the control to be recognized in the text form is matched with the target file in the text form;
if yes, go to step S206;
step S204 may specifically be exemplified as:
determining whether the matching degree of the control to be recognized in the text form and the target control in the text form is not lower than a second preset matching threshold, and if not, matching; and if the matching threshold is lower than the second preset matching threshold, the matching is not performed.
Step S206: and identifying the control to be identified in the text form as a target control.
In one embodiment, before step S202, the method further includes:
step S201: determining whether the screenshot of the target interface is matched with the screenshot of the reference interface;
if yes, go to step S202;
the method comprises the following steps of testing two states of script recording and execution, wherein a target interface is an interface where a control to be identified is located when a current script step is executed; the reference interface is the interface where the target control is located when the current script is recorded;
step S201 may specifically be exemplified by:
determining whether the matching degree of the screenshot of the target interface and the screenshot of the reference interface is lower than a fourth preset matching threshold value; if the current value is not lower than the fourth preset matching threshold value, matching is carried out; and if the matching threshold value is lower than a fourth preset matching threshold value, the matching is not performed.
Based on the implementation manner of the step 204, the control to be recognized can be recognized in a manner of combining the control tree of the target interface and the text information, so that the accuracy, diversity and reliability of control recognition are improved.
It should be noted that, when the control information includes control tree and icon information, another implementation manner of step 204 may be:
step S1': identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
step S2': and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Based on the implementation manner of the step 204, the control to be recognized can be recognized in a manner of combining the control tree and the icon information of the target interface, so that the accuracy, diversity and reliability of control recognition are improved.
Further, referring to fig. 5, one implementation of step S2' may be:
step S202': identifying a control to be identified in the icon information by adopting an image identification mode, wherein the icon information comprises the control to be identified in an icon form;
step S204': determining whether the control to be identified in the icon form is matched with the target control in the icon form;
if yes, go to step S206';
the step S204' may be specifically exemplified as:
determining whether the matching degree of the control to be identified in the icon form and the target control in the icon form is not lower than a third preset matching threshold, and if not, matching; and if the matching threshold value is lower than the third preset matching threshold value, the matching is not performed.
Step S206': and identifying the identification control in the form of the icon to be identified as a target control.
In one embodiment, before step S202', the method further includes:
step S201': determining whether the screenshot of the target interface is matched with the screenshot of the reference interface;
if yes, go to step S202';
step S201 may specifically be exemplified by:
determining whether the matching degree of the screenshot of the target interface and the screenshot of the reference interface is lower than a fourth preset matching threshold value; if the current value is not lower than the fourth preset matching threshold value, matching is carried out; and if the matching threshold value is lower than a fourth preset matching threshold value, the matching is not performed.
It should be noted that, when the control information includes text information and icon information, another implementation manner of step 204 may be:
identifying the control to be identified based on the text information, wherein the text information comprises the control to be identified in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Based on the implementation manner of the step 204, the control to be recognized can be recognized in a manner of combining the text information and the icon information of the target interface, so that the accuracy, diversity and reliability of control recognition are improved.
Further, one implementation manner of step S2 or step S2' may be:
determining whether the complexity of the control to be identified in the form of an icon is lower than a complexity threshold;
if so, identifying a father control of the control to be identified based on the icon information so as to identify the control to be identified, wherein the icon information comprises the father control in an icon form;
and if not, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in the form of an icon.
The implementation manner may specifically be exemplified as follows:
if the complexity of the control to be identified in the icon form is lower than the complexity threshold, namely the control to be identified in the icon form is too simple, determining whether the complexity of a parent control of the control to be identified in the icon form is lower than the complexity threshold;
if the complexity of the parent control of the control to be identified in the icon form is not lower than the complexity threshold, identifying the parent control of the control to be identified in an image identification mode, and determining whether the parent control of the control to be identified is matched with the parent control of the target control; if the parent control of the control to be identified is matched with the target control, identifying the parent control of the control to be identified as the parent control of the target control, and determining the control to be identified as the target control;
if the complexity of the control to be recognized in the icon form is not lower than the complexity threshold, recognizing the control to be recognized in an image recognition mode, and determining whether the control to be recognized is matched with the target control; and if the control to be identified is matched with the target control, identifying the control to be identified as the target control.
Based on the method, the control to be identified is identified by identifying the father control of the control to be identified, so that the reliability and flexibility of control identification are further improved.
It should be noted that, another implementation manner of step S2 or step S2' may be:
if the complexity of the parent control of the control to be identified in the icon form is lower than the complexity threshold, determining whether the complexity of the parent control of the control to be identified (namely the parent control of the control to be identified) is lower than the complexity threshold;
if not, identifying the father control of the control to be identified by adopting an image identification mode, and determining whether the father control of the control to be identified is matched with the father control of the target control; and if the parent control of the control to be identified is matched with the target control, identifying that the parent control of the control to be identified is the parent control of the target control, and determining that the control to be identified is the target control.
Based on the method, the control to be identified is identified by identifying the father control of the control to be identified, so that the reliability and flexibility of control identification are further improved.
It should be noted that when the control information includes a control tree, text information, and icon information, another implementation manner of step 204 may be:
step S1 ″: identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
step S2 ″: if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
step S3 ″: and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Based on the implementation manner of the step 204, the control to be recognized can be recognized in a manner of combining the control tree, the text information and the icon information of the target interface, so that the accuracy, diversity and reliability of control recognition are improved.
After step S3 ″, step 204 further includes:
if the control to be identified cannot be identified, determining whether a first preset identification period is exceeded;
if not, identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Based on this, through setting up the default identification cycle, increased the number of times of controlling part discernment, improved the reliability and the efficiency of controlling part discernment.
Further, after step S3 ″, step 204 further includes:
if the first preset identification period is exceeded, determining whether a second preset identification period is exceeded;
if not, identifying the control to be identified based on the text information, wherein the text information comprises the control to be identified in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Based on the control identification method, the preset identification period is set, the number of times of control identification is increased, and when the time is not sufficient, the control is identified preferentially based on one or more control information, so that the flexibility, reliability and efficiency of control identification are improved.
For the embodiment corresponding to fig. 2, the control to be recognized is recognized based on the control information of the target interface where the control to be recognized is located, so that the control recognition accuracy can be improved.
Fig. 6 is a flowchart illustrating an icon recognition method according to another embodiment of the present invention, where the method may be executed by a terminal device in an application scenario corresponding to fig. 1, and referring to fig. 6, the method may specifically include the following steps:
step 602: when the control information comprises a control tree, text information and icon information, identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
step 604: if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
step 606: if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
step 608: if the control to be identified cannot be identified, determining whether a first preset identification period is exceeded;
if not, go to step 602; if yes, go to step 610;
step 610: determining whether a second preset identification period is exceeded;
if not, go to step 604; if yes, the process is ended.
For the embodiment corresponding to fig. 6, the control is identified based on at least 2 of the control tree, the text information and the target information of the target interface, so that the accuracy, reliability and flexibility of control identification are improved; by setting the preset identification period, the efficiency of control identification is improved.
In addition, for simplicity of explanation, the above-described method embodiments are described as a series of acts or combinations, but it should be understood by those skilled in the art that the present invention is not limited by the order of acts or steps described, as some steps may be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
Fig. 7 is a schematic structural diagram of a control identification apparatus according to an embodiment of the present invention, and referring to fig. 7, the apparatus may specifically include: a determination module 702 and an identification module 704, wherein:
a determining module 702, configured to determine a target interface where a control to be identified is located;
an identifying module 704, configured to identify the control to be identified based on the control information of the target interface;
the control information comprises at least two of control tree, text information and icon information.
For the embodiment corresponding to fig. 7, the control to be recognized is recognized based on the control information of the target interface where the control to be recognized is located, so that the accuracy of control recognition can be improved.
Optionally, when the control information includes a control tree and text information, the identifying module includes:
the first identification submodule is used for identifying the control to be identified based on the control tree, and the control tree comprises the control to be identified in an identifier or text form;
and if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form.
Optionally, when the control information includes control tree and icon information, the identifying module further includes:
the second identification submodule is used for identifying the control to be identified based on the control tree, and the control tree comprises the control to be identified in an identifier or text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Optionally, when the control information includes text information and icon information, the identifying module further includes:
the third identification submodule is used for identifying the control to be identified based on the text information, and the text information comprises the control to be identified in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Optionally, the second identification sub-module or the third identification sub-module further includes:
the complexity determining unit is used for determining whether the complexity of the control to be identified in the icon form is lower than a complexity threshold value;
if so, identifying a father control of the control to be identified based on the icon information so as to identify the control to be identified, wherein the icon information comprises the father control in an icon form;
and if not, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in the form of an icon.
Optionally, when the control information includes a control tree, text information, and icon information, the identification module further includes:
the fourth identification submodule is used for identifying the control to be identified based on the control tree, and the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Optionally, the fourth identification sub-module further includes:
the first period determining unit is used for determining whether a first preset identification period is exceeded or not if the control to be identified cannot be identified;
if not, identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
Optionally, the first period determining unit further includes:
a second period determining subunit, configured to determine whether the first preset identification period is exceeded or not if the first preset identification period is exceeded;
if not, identifying the control to be identified based on the text information, wherein the text information comprises the control to be identified in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
The device provided by the embodiment of the present invention can implement each process implemented by the device in the method embodiments of fig. 2 to fig. 6, and is not described herein again to avoid repetition. Further, it should be noted that, among the respective components of the apparatus of the present invention, the components thereof are logically divided according to the functions to be realized, but the present invention is not limited thereto, and the respective components may be newly divided or combined as necessary.
Figure 8 is a schematic diagram of a hardware structure of a terminal device implementing various embodiments of the present invention,
the terminal device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, a processor 810, and a power supply 811. Those skilled in the art will appreciate that the terminal device configuration shown in fig. 8 does not constitute a limitation of the terminal device, and that the terminal device may include more or fewer components than shown, or combine certain components, or a different arrangement of components. In the embodiment of the present invention, the terminal device includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted terminal, a wearable device, a pedometer, and the like.
The processor 810 is configured to determine a target interface where the control to be identified is located;
identifying the control to be identified based on the control information of the target interface;
the control information comprises at least two of control tree, text information and icon information.
The control to be identified is identified based on the control information of the target interface where the control to be identified is located, so that the control identification accuracy can be improved.
It should be understood that, in the embodiment of the present invention, the radio frequency unit 801 may be used for receiving and sending signals during a message sending and receiving process or a call process, and specifically, receives downlink data from a base station and then processes the received downlink data to the processor 810; in addition, the uplink data is transmitted to the base station. In general, radio frequency unit 801 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. Further, the radio frequency unit 801 can also communicate with a network and other devices through a wireless communication system.
The terminal device provides wireless broadband internet access to the user through the network module 802, such as helping the user send and receive e-mails, browse webpages, access streaming media, and the like.
The audio output unit 803 may convert audio data received by the radio frequency unit 801 or the network module 802 or stored in the memory 809 into an audio signal and output as sound. Also, the audio output unit 803 may also provide audio output related to a specific function performed by the terminal apparatus 800 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 803 includes a speaker, a buzzer, a receiver, and the like.
The input unit 804 is used for receiving an audio or video signal. The input Unit 804 may include a Graphics Processing Unit (GPU) 8041 and a microphone 8042, and the Graphics processor 8041 processes image data of a still picture or video obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the display unit 806. The image frames processed by the graphics processor 8041 may be stored in the memory 809 (or other storage medium) or transmitted via the radio frequency unit 801 or the network module 802. The microphone 8042 can receive sound, and can process such sound into audio data. The processed audio data may be converted into a format output transmittable to a mobile communication base station via the radio frequency unit 801 in case of a phone call mode.
The terminal device 800 also includes at least one sensor 805, such as light sensors, motion sensors, and other sensors. Specifically, the light sensor includes an ambient light sensor that can adjust the brightness of the display panel 8061 according to the brightness of ambient light, and a proximity sensor that can turn off the display panel 8061 and/or the backlight when the terminal device 800 moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), detect the magnitude and direction of gravity when stationary, and can be used to identify the terminal device posture (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration identification related functions (such as pedometer, tapping), and the like; the sensors 805 may also include fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which are not described in detail herein.
The display unit 806 is used to display information input by the user or information provided to the user. The Display unit 806 may include a Display panel 8061, and the Display panel 8061 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.
The user input unit 807 is operable to receive input numeric or character information and generate key signal inputs related to user settings and function control of the terminal device. Specifically, the user input unit 807 includes a touch panel 8071 and other input devices 8072. The touch panel 8071, also referred to as a touch screen, may collect touch operations by a user on or near the touch panel 8071 (e.g., operations by a user on or near the touch panel 8071 using a finger, a stylus, or any other suitable object or accessory). The touch panel 8071 may include two portions of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 810, receives a command from the processor 810, and executes the command. In addition, the touch panel 8071 can be implemented by various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. In addition to the touch panel 8071, the user input unit 807 can include other input devices 8072. In particular, other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein.
Further, the touch panel 8071 can be overlaid on the display panel 8061, and when the touch panel 8071 detects a touch operation on or near the touch panel 8071, the touch operation is transmitted to the processor 810 to determine the type of the touch event, and then the processor 810 provides a corresponding visual output on the display panel 8061 according to the type of the touch event. Although in fig. 8, the touch panel 8071 and the display panel 8061 are two independent components to implement the input and output functions of the terminal device, in some embodiments, the touch panel 8071 and the display panel 8061 may be integrated to implement the input and output functions of the terminal device, and this is not limited herein.
The interface unit 808 is an interface for connecting an external device to the terminal apparatus 800. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 808 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the terminal apparatus 800 or may be used to transmit data between the terminal apparatus 800 and an external device.
The memory 809 may be used to store software programs as well as various data. The memory 809 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 809 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The processor 810 is a control center of the terminal device, connects various parts of the whole terminal device by using various interfaces and lines, and performs various functions of the terminal device and processes data by running or executing software programs and/or modules stored in the memory 809 and calling data stored in the memory 809, thereby performing overall monitoring of the terminal device. Processor 810 may include one or more processing units; preferably, the processor 810 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 810.
In addition, the terminal device 800 includes some functional modules that are not shown, and are not described in detail here.
Preferably, an embodiment of the present invention further provides a terminal device, which includes a processor 810, a memory 809, and a computer program stored in the memory 809 and capable of running on the processor 810, where the computer program, when executed by the processor 810, implements each process of the above-mentioned control identification method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not described here again.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the above-mentioned control identification method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (6)
1. A control identification method is characterized by comprising the following steps:
determining a target interface where a control to be identified is located;
identifying the control to be identified based on the control information of the target interface;
the control information comprises at least two of a control tree, text information and icon information;
when the control information comprises a control tree and text information, identifying the control to be identified based on the control information of the target interface, including:
identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
when the control information comprises a control tree and icon information, identifying the control to be identified based on the control information of the target interface, including:
identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
when the control information comprises text information and icon information, identifying the control to be identified based on the control information of the target interface, including:
identifying the control to be identified based on the text information, wherein the text information comprises the control to be identified in a text form;
if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
based on the icon information, identifying the control to be identified comprises:
determining whether the complexity of the control to be identified in the form of an icon is lower than a complexity threshold;
if so, identifying a father control of the control to be identified based on the icon information so as to identify the control to be identified, wherein the icon information comprises the father control in an icon form;
if not, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
if the complexity of the control to be identified in the icon form is lower than the complexity threshold, namely the control to be identified in the icon form is too simple, determining whether the complexity of a parent control of the control to be identified in the icon form is lower than the complexity threshold;
if the complexity of the parent control of the control to be identified in the icon form is not lower than the complexity threshold, identifying the parent control of the control to be identified in an image identification mode, and determining whether the parent control of the control to be identified is matched with the parent control of the target control; if the parent control of the control to be identified is matched with the target control, identifying the parent control of the control to be identified as the parent control of the target control, and determining the control to be identified as the target control;
if the complexity of the control to be recognized in the icon form is not lower than the complexity threshold, recognizing the control to be recognized in an image recognition mode, and determining whether the control to be recognized is matched with the target control; and if the control to be identified is matched with the target control, identifying the control to be identified as the target control.
2. The method according to claim 1, wherein when the control information includes a control tree, text information, and icon information, identifying the control to be identified based on the control information of the target interface includes:
identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
3. The method of claim 2, wherein after identifying the control to be identified based on the icon information, the method further comprises:
if the control to be identified cannot be identified, determining whether a first preset identification period is exceeded;
if not, identifying the control to be identified based on the control tree, wherein the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
4. The method according to claim 3, wherein if the first predetermined identification period is exceeded, determining whether a second predetermined identification period is exceeded;
if not, identifying the control to be identified based on the text information, wherein the text information comprises the control to be identified in a text form;
and if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form.
5. A control recognition apparatus, comprising:
the determining module is used for determining a target interface where the control to be identified is located;
the identification module is used for identifying the control to be identified based on the control information of the target interface;
the control information comprises at least two of a control tree, text information and icon information;
when the control information comprises a control tree and text information, the identification module comprises:
the first identification submodule is used for identifying the control to be identified based on the control tree, and the control tree comprises the control to be identified in an identifier or text form;
if the control to be recognized cannot be recognized, recognizing the control to be recognized based on the text information, wherein the text information comprises the control to be recognized in a text form;
when the control information comprises control tree and icon information, the identification module further comprises:
the second identification submodule is used for identifying the control to be identified based on the control tree, and the control tree comprises the control to be identified in an identifier or text form;
if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
when the control information comprises text information and icon information, the identification module further comprises:
the third identification submodule is used for identifying the control to be identified based on the text information, and the text information comprises the control to be identified in a text form;
if the control to be identified cannot be identified, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
the second identification submodule or the third identification submodule further includes:
the complexity determining unit is used for determining whether the complexity of the control to be identified in the icon form is lower than a complexity threshold value;
if so, identifying a father control of the control to be identified based on the icon information so as to identify the control to be identified, wherein the icon information comprises the father control in an icon form;
if not, identifying the control to be identified based on the icon information, wherein the icon information comprises the control to be identified in an icon form;
if the complexity of the control to be identified in the icon form is lower than the complexity threshold, namely the control to be identified in the icon form is too simple, determining whether the complexity of a parent control of the control to be identified in the icon form is lower than the complexity threshold;
if the complexity of the parent control of the control to be identified in the icon form is not lower than the complexity threshold, identifying the parent control of the control to be identified in an image identification mode, and determining whether the parent control of the control to be identified is matched with the parent control of the target control; if the parent control of the control to be identified is matched with the target control, identifying the parent control of the control to be identified as the parent control of the target control, and determining the control to be identified as the target control;
if the complexity of the control to be recognized in the icon form is not lower than the complexity threshold, recognizing the control to be recognized in an image recognition mode, and determining whether the control to be recognized is matched with the target control; and if the control to be identified is matched with the target control, identifying the control to be identified as the target control.
6. A terminal device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910138832.2A CN109857673B (en) | 2019-02-25 | 2019-02-25 | Control identification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910138832.2A CN109857673B (en) | 2019-02-25 | 2019-02-25 | Control identification method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109857673A CN109857673A (en) | 2019-06-07 |
| CN109857673B true CN109857673B (en) | 2022-02-15 |
Family
ID=66898921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910138832.2A Active CN109857673B (en) | 2019-02-25 | 2019-02-25 | Control identification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109857673B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114116441A (en) * | 2020-12-30 | 2022-03-01 | 京东数科海益信息科技有限公司 | UI (user interface) testing method and device, electronic equipment and storage medium |
| CN113076257B (en) * | 2021-04-19 | 2024-08-09 | 西安闻泰信息技术有限公司 | Test script recording method, device, medium and computer equipment |
| CN114511696A (en) * | 2021-12-22 | 2022-05-17 | 北京云测信息技术有限公司 | Control positioning method and device, electronic equipment and readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309797A (en) * | 2012-03-09 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Automatic testing method and device of user interface |
| CN104391797A (en) * | 2014-12-09 | 2015-03-04 | 北京奇虎科技有限公司 | GUI (graphical user interface) widget identification method and device |
| CN108399124A (en) * | 2018-02-07 | 2018-08-14 | 深圳壹账通智能科技有限公司 | Application testing method, device, computer equipment and storage medium |
| CN109002759A (en) * | 2018-06-07 | 2018-12-14 | Oppo广东移动通信有限公司 | text recognition method, device, mobile terminal and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9483304B2 (en) * | 2011-07-29 | 2016-11-01 | National Instruments Corporation | Interface wires for a measurement system diagram |
| US9274742B2 (en) * | 2012-10-25 | 2016-03-01 | PixiOnCloud, Inc. | Visual-symbolic control of remote devices having display-based user interfaces |
| CN105843494B (en) * | 2015-01-15 | 2020-06-09 | 中兴通讯股份有限公司 | Method, device and terminal for realizing area screen capture |
| CN108363599B (en) * | 2018-01-12 | 2019-07-19 | 深圳壹账通智能科技有限公司 | User interface shows recognition methods and terminal device |
| CN108595965B (en) * | 2018-04-28 | 2019-12-20 | 北京字节跳动网络技术有限公司 | Method and device for preventing attribute information of interface control from being stolen |
-
2019
- 2019-02-25 CN CN201910138832.2A patent/CN109857673B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309797A (en) * | 2012-03-09 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Automatic testing method and device of user interface |
| CN104391797A (en) * | 2014-12-09 | 2015-03-04 | 北京奇虎科技有限公司 | GUI (graphical user interface) widget identification method and device |
| CN108399124A (en) * | 2018-02-07 | 2018-08-14 | 深圳壹账通智能科技有限公司 | Application testing method, device, computer equipment and storage medium |
| CN109002759A (en) * | 2018-06-07 | 2018-12-14 | Oppo广东移动通信有限公司 | text recognition method, device, mobile terminal and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109857673A (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108845853B (en) | Application program starting method and mobile terminal | |
| CN109947650B (en) | Script step processing method, device and system | |
| CN109814786B (en) | Image storage method and terminal equipment | |
| CN108710458B (en) | Split screen control method and terminal equipment | |
| CN109639863B (en) | Voice processing method and device | |
| CN110913067A (en) | Information sending method and electronic equipment | |
| WO2021083091A1 (en) | Screenshot capturing method and terminal device | |
| CN107835495B (en) | Message prompting method and device and mobile terminal | |
| CN110096203B (en) | Screenshot method and mobile terminal | |
| CN109857673B (en) | Control identification method and device | |
| CN108920040B (en) | Application icon sorting method and mobile terminal | |
| CN110069407B (en) | Function test method and device for application program | |
| CN109982273B (en) | Information reply method and mobile terminal | |
| CN108628534B (en) | Character display method and mobile terminal | |
| CN110784394A (en) | Prompting method and electronic equipment | |
| CN109947323B (en) | Terminal display method and terminal | |
| CN109491572B (en) | Screen capturing method of mobile terminal and mobile terminal | |
| CN109521916B (en) | File processing method based on flexible screen, mobile terminal and storage medium | |
| CN108897467B (en) | Display control method and terminal equipment | |
| CN110851042A (en) | Interface display method and electronic equipment | |
| CN110888572A (en) | Message display method and terminal equipment | |
| CN108132823B (en) | Notification message prompting method and mobile terminal | |
| CN111443968A (en) | Screenshot method and electronic equipment | |
| CN109783379B (en) | Script execution exception determining method and device | |
| CN111614837B (en) | Cross-application data calling method and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210721 Address after: 100016 no.4301, No.52 Jiuxianqiao hospital, Chaoyang District, Beijing Applicant after: BEIJING TESTIN INFORMATION TECHNOLOGY Co.,Ltd. Address before: Room 2016, building 2, No.8, Fenghuang Third Road, Zhongxin Guangzhou Knowledge City, Guangzhou 510260, Guangdong Province Applicant before: GUANGZHOU TESTIN INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |