JP7443014B2 - robot arm testing equipment - Google Patents

Description

The present invention relates to a robot arm testing device.

Recently, mixed reality (hereinafter also referred to as MR) using a head-mounted display (hereinafter also referred to as HMD) is being developed, as described in, for example, Japanese Patent Application Publication No. 2015-125641 and 2004-355131. ) Technology is developing. To quote from Japanese Patent Application Laid-Open No. 2015-125641, MR technology superimposes an image of a virtual world (virtual space) generated by a computer on an image of the real world (real space) captured by an HMD. This is a technology that displays images on the screen. That is, the real space image and the virtual space image (three-dimensional model) are displayed on the HMD in a superimposed manner. By using MR technology, it is possible to check the size of a finished product, the workability of removing (installing) parts, etc. without actually manufacturing a model. A mixed reality device using MR technology includes, for example, an HMD and an information processing unit that presents mixed reality, which is a superimposed image, to a user via the HMD.

Furthermore, in order to present mixed reality with an HMD, it is necessary to determine the position and orientation of the HMD in the target space (real space). For example, as described in Japanese Unexamined Patent Publication No. 2015-215191, in order to use a mixed reality device, it is necessary to determine the relative position and orientation between a reference coordinate system defined in real space and a camera coordinate system. It is essential to measure the Here, for example, Japanese Patent Application Laid-Open No. 2015-215191 and Japanese Patent Application Publication No. 2008-70267 describe estimating the position and orientation of a camera from information of a target marker imaged by a camera provided in an HMD. Coordinate information is set in the target marker. When a camera captures an image of a target marker installed in a work space (real space), the position and orientation of the camera can be estimated using a known calculation method.

Japanese Patent Application Publication No. 2015-125641 Japanese Patent Application Publication No. 2004-355131 Japanese Patent Application Publication No. 2015-215191 Japanese Patent Application Publication No. 2008-70267

By the way, when introducing a robot arm into a factory, detailed simulations are required in advance to determine whether or not the robot arm's operation will interfere with other equipment. For example, by placing and operating a robot arm to be introduced at an actual factory site, it is possible to understand the extent of interference. However, when a robot arm is actually placed in a factory and an operation test is performed, the hand parts attached to the tip of the robot arm (e.g., gripping hand, welding torch, polishing hand, vacuum hand, etc.) are connected to other equipment. There is a risk of damage due to contact.

Further, if the robot arm is assembled in a factory and it is determined that the arrangement or type of the robot arm or hand parts is not appropriate, the installation location or hand parts must be changed and reassembled. As described above, in the conventional method, a great deal of effort was required in testing when the robot arm was introduced.

An object of the present invention is to provide a robot arm in which damage to the hand parts of the robot arm can be prevented and operation and interference tests of the hand parts can be easily performed.

The robot arm testing device of the present invention includes a display, a camera that images the outside, imaging data of a first target marker that is a target marker in a first coordinate system that is imaged by the camera, and information on the first target marker. a position and orientation measurement unit that measures the position and orientation of the camera based on coordinate information set for the camera; and a position and orientation measurement unit that measures the position and orientation of the camera based on coordinate information set for the camera; an information processing unit that presents mixed reality, which is a superimposed image of real space and a three-dimensional model; A robot arm to which a second target marker, which is a target marker of a second coordinate system independent of the system, is attached, and a three-dimensional hand model representing the hand component set corresponding to the second target marker are stored. a storage unit , the three-dimensional model is actual machine layout data, the first coordinate system is a world coordinate system, and the second coordinate system has an origin on the first coordinate system. It is a sub-coordinate system that moves as the second target marker moves, and the information processing unit displays the three-dimensional model on the display based on the imaging data of the first target marker, and displays the three-dimensional model on the display based on the imaging data of the first target marker. The three-dimensional hand model is displayed on the first coordinate system based on the imaging data of the marker, and the imaging data of the arm is displayed as the real space superimposed on the three-dimensional model and the three-dimensional hand model.

A robot arm testing device according to a second aspect of the present invention includes a display, a camera that images the outside, image data of a target marker imaged by the camera, and coordinate information set for the target marker. a position and orientation measurement unit that measures the position and orientation of the camera; and a display that allows the user to see the real space and the three-dimensional model through the display based on the position and orientation of the camera measured by the position and orientation measurement unit. an information processing unit that presents a mixed reality that is a superimposed image; a robot arm having the target marker attached to the tip of the arm instead of a hand component; and the hand component configured to correspond to the target marker. a storage unit that stores the three-dimensional model representing the target marker, and the information processing unit displays the three-dimensional model on the display based on the image data of the target marker along with the real space.

According to the first aspect of the present invention, the display includes a three-dimensional model defined in the first coordinate system, a three-dimensional hand model representing hand parts defined in the second coordinate system, and imaged data showing the arm of the actual machine. The real space represented as it is is displayed in a superimposed manner. When the arm that is being imaged by the camera is actually activated, the imaging data of the arm is activated on the three-dimensional model (first coordinates) on the display. When the tip of the arm moves, the second target marker (origin of the second coordinates) moves accordingly, and the three-dimensional hand model moves on the first coordinates. Every time the second target marker shakes its head and changes its position or orientation, the position or orientation of the three-dimensional hand model in the second coordinate system changes.

According to this configuration, even if the three-dimensional hand model or the arm contacts the three-dimensional model, the hand components do not actually come into contact with anything else, so damage to the hand components is prevented. Furthermore, by setting the factory's actual machine layout data as a three-dimensional model, motion and interference tests can be conducted in advance at a different location without actually placing the robot arm in the factory. Further, by simply changing the data of the three-dimensional hand model, it is possible to test another hand component. As described above, according to the first aspect, damage to the hand components of the robot arm can be prevented, and operation/interference tests of the hand components can be easily performed.

According to the reference aspect, the hand parts are displayed as a three-dimensional model on the display on the image data (real space) actually captured by the camera. The position and orientation of the target marker changes in accordance with the actual movement of the arm, and the position and orientation of the three-dimensional model on the display also change. In this way, the position and orientation of the three-dimensional model of the hand component that appears in the real space on the display changes due to a change in the image data of the target marker. The second embodiment below corresponds to a reference aspect.

According to this configuration, when the robot arm is actually placed in a factory and tested, the target marker may come into contact with other devices, but the actual hand component may not come into contact with other devices. Therefore, damage to the hand parts is prevented. Furthermore, it is possible to perform operation/interference tests on multiple types of hand parts simply by changing the data of the displayed three-dimensional model without replacing the actual hand parts. As described above, according to the second aspect, damage to the hand components of the robot arm can be prevented, and operation/interference tests of the hand components can be easily performed.

FIG. 1 is a configuration diagram of a robot arm testing device according to a first embodiment. FIG. 3 is an explanatory diagram for explaining the display state of the display of the first embodiment. This is an example of setting the origin of the coordinate system in the first embodiment. FIG. 7 is an explanatory diagram for explaining another example display state of the display of the first embodiment. It is a block diagram of the robot arm test device of 2nd Embodiment.

Embodiments of the present invention will be described below based on the drawings. Note that in each of the following embodiments, parts that are the same or equivalent to each other are given the same reference numerals in the drawings. Moreover, each figure used for explanation is a conceptual diagram. The description and drawings of each embodiment may be referred to.

<First embodiment>
As shown in FIG. 1, the robot arm testing device 1 of the first embodiment includes a display 2, a camera 3, a position and orientation measurement section 4, an information processing section 5, a robot arm 6, a storage section 7, It is equipped with The camera 3 is a device that images the outside. In the example of the first embodiment, the robot arm testing device 1 includes a tablet terminal 100 including a CPU and a memory, and a robot arm 6.

The tablet terminal 100 includes a display 2, a camera 3, a position/orientation measuring section 4, an information processing section 5, and a storage section 7. The tablet terminal 100 is a computer loaded with software related to MR. The display 2 is provided on the front surface of the tablet terminal 100, and the camera 3 is provided on the back surface of the tablet terminal 100. Each of the functional units 4, 5, and 7 is arranged inside the tablet terminal 100.

The position and orientation measurement unit 4 uses the camera 3 to determine the position and orientation of the first target marker 91, which is a target marker in the first coordinate system, captured by the camera 3, based on the captured data and the coordinate information set for the first target marker 91. Measure the position and posture of 3. The first coordinate system is a three-dimensional so-called world coordinate system, and is defined by an origin position and three axes. The position and orientation measuring unit 4 calculates the position and orientation of the camera 3 based on the size and shape of the first target marker 91 in the imaging data.

The position/orientation measurement unit 4 reads coordinate information set according to the pattern of the first target marker 91 and estimates from which position the camera 3 is facing in which direction. A plurality of first target markers 91 are displayed on a wall surface, an installation surface, or a display placed behind the robot arm 6.

Based on the position and orientation of the camera 3 measured by the position and orientation measurement unit 4, the information processing unit 5 displays mixed reality (MR), which is a superimposed image of the real space and the three-dimensional model 81, to the user via the display 2. present. The three-dimensional model 81 is, for example, point cloud data obtained by scanning a factory or image data based on the point cloud data.

The three-dimensional model 81 displayed on the display 2 as MR corresponds to the coordinate information of the first target marker 91, and its appearance changes according to changes in the position of the first target marker 91 imaged by the camera 3, etc. For example, based on the position, size, pattern, and shape (imaging angle) of the first target marker 91, the position and orientation of the camera 3 in the target space can be estimated. For example, as described in JP-A-2015-215191 and JP-A-2008-70267, a mixed reality device (MR device) uses a known The position and orientation of the camera 3 can be measured using the following method. The mixed reality device (MR device) is a well-known device, and detailed description thereof will be omitted.

The robot arm 6 is a robot arm in which a second target marker 92, which is a target marker of a second coordinate system independent of the first coordinate system, is attached to the tip of the arm 61 instead of a hand component. In other words, the actual hand component is not attached to the robot arm 6, but instead, the second target marker 92 is attached to the location where the hand component is scheduled to be placed. The second coordinate system is a sub-coordinate system. That is, the origin of the second coordinate system moves along the movement of the arm 61 on the first coordinate without affecting the origin position of the first coordinate.

The arm 61 has, for example, one or more joints, and is capable of rotation and operation of each small arm. A plurality of second target markers 92 are installed at the tip of the arm 61. A plurality of second target markers 92 are arranged, for example, on the surface of a member that is a combination of a plurality of hexahedrons. A plurality of second target markers 92 are arranged three-dimensionally (so that perpendicular lines intersect).

The storage unit 7 is a device that stores a plurality of three-dimensional hand models 82 representing hand parts of the robot arm 6, which are set to correspond to the second target marker 92. The three-dimensional hand model is, for example, CG (computer graphics) such as a gripping hand, a welding torch, a polishing hand, or a vacuum hand. The storage unit 7 stores a plurality of three-dimensional hand models 82 having different shapes.

Here, as shown in FIG. 2, the information processing unit 5 displays a three-dimensional model on the display 2 based on the imaging data of the first target marker 91, and displays a three-dimensional model on the display 2 based on the imaging data of the second target marker 92. The dimensional hand model 82 is displayed, and the imaging data of the arm 61 is displayed as real space, superimposed on the 3-dimensional model and the 3-dimensional hand model.

The information processing unit 5 is set so that a specific color portion of the image data captured by the camera 3 is displayed as a real space in a superimposed image. The color of the surface of the arm 61 (mostly the same color) is set as this specific color. Therefore, since the information processing unit 5 displays the color of the surface of the arm 61 on the display 2 as a real space, the image data of the arm 61 of the actual device is displayed on the display 2 as is. When the arm 61 moves within the imaging range of the camera 3, the arm 61, which is also displayed without CG on the display 2, moves on the first coordinate. Thus, in the first embodiment, a three-dimensional model of the arm 61 is not necessary.

When the robot arm 6 is operated to move and change the posture of only the second target marker 92, only the three-dimensional hand model 82 moves and changes its posture on the display 2. The position and orientation measurement unit 4 determines movement of the origin of the second coordinate on the first coordinate, change in orientation, etc. from the appearance of the second target marker 92. Since the position of the arm 61 on the first coordinates is determined by the origin of the second coordinates, the position and orientation measurement unit 4 can also grasp the relative positional relationship between the second target marker 92 and the arm 61. Therefore, on the display 2, the three-dimensional hand model 82 appears in front of the arm 61 or is hidden behind the arm 61 depending on the front-back relationship (front/back) between the three-dimensional hand model 82 and the arm 61. Note that FIG. 3 is an example of setting the origin position.

In the first embodiment, the arm 61 is colored so that at least part of the color is a specific color different from the color of the background of the arm 61 in the imaging data of the camera 3. Then, the information processing unit 5 displays the specific color in front of the three-dimensional model 81. This allows the arm 61 to be displayed more clearly.

When the three-dimensional model 81 and the three-dimensional hand model 82 come into contact (become the same coordinates), for example, the three-dimensional hand model 82 at the contact portion is no longer displayed. Alternatively, it is also possible to display only the touched portion in a different manner (color, blinking, etc.). In this manner, the information processing unit 5 may perform special display processing including at least one of color change processing and blinking processing on the contact portion between the three-dimensional model 81 and the three-dimensional hand model 82.

(Effects of the first embodiment)
According to the first embodiment, the display 2 displays a three-dimensional model 81 defined in the first coordinate system, a three-dimensional hand model 82 representing hand parts defined in the second coordinate system, and the arm 61 of the actual machine. The real space (displayed in a specific color) expressed as imaged data is displayed in a superimposed manner. When the arm 61 that is photographing with the camera 3 is actually operated, the image data of the arm 61 is displayed on the display 2 on the three-dimensional model 81 (first coordinates). When the tip of the arm 61 moves, the second target marker 92 (the origin of the second coordinates) moves accordingly, and the three-dimensional hand model 82 moves on the first coordinates. Every time the second target marker 92 shakes its head and changes its position or orientation, the position or orientation of the three-dimensional hand model 82 in the second coordinate system changes.

According to this configuration, even if the three-dimensional hand model 82 or the arm 61 contacts the three-dimensional model, the hand components do not actually come into contact with anything else, so damage to the hand components is prevented. Furthermore, by setting the actual machine layout data of the factory as the three-dimensional model 81, a motion/interference test can be conducted in advance at another location without actually arranging the robot arm 6 in the factory. Further, by simply changing the data of the three-dimensional hand model 82, it is possible to test another hand component. As described above, according to the first embodiment, damage to the hand components of the robot arm 6 can be prevented, and operation/interference tests of the hand components can be easily performed.

(another example)
In the example shown in FIG. 4, the camera 3 and the display 2 are separate bodies (connected by wire). Further, a plurality of second target markers 92 are attached to one solid (formed by a plurality of cubes). In this scene, the camera 3 images the second target marker 92 from the front so that the arm 61 is positioned behind the second target marker 92. As a result, the part of the arm 61 hidden behind the second target marker 92 (solid) is displayed on the display 2 in a blank white state because the specific color is not detected. In this way, the tablet terminal 100 displays the object on the display 2 after grasping the context (depth) from the coordinates.

<Second embodiment>
The robot arm testing device 1A of the second embodiment includes a tablet terminal 100 and a robot arm 6, as shown in FIG. The tablet terminal 100 includes a display 2, a camera 3 for capturing an image of the outside, a position and orientation measurement section 4, an information processing section 5, and a storage section 7.

The position and orientation measurement unit 4 measures the position and orientation of the camera 3 based on image data of the target marker 93 captured by the camera 3 and coordinate information set for the target marker 93. The information processing unit 5 presents mixed reality, which is a superimposed image of the real space and the three-dimensional model, to the user via the display 2 based on the position and orientation of the camera 3 measured by the position and orientation measurement unit 4. The robot arm 6 is a robot arm in which a target marker 93 is attached to the tip of the arm 61 instead of a hand component. The storage unit 7 is a device that stores a three-dimensional model representing a hand component set corresponding to the target marker 93. The storage unit 7 stores a plurality of three-dimensional models having different shapes.

The information processing unit 5 of the second embodiment displays a three-dimensional model on the display 2 based on the imaging data of the target marker 93 as well as the real space. That is, in the second embodiment, unlike the first embodiment, only the three-dimensional model representing the hand component is displayed in CG, and the other image data is displayed as it is in real space. The screen displayed on the display 2 in the second embodiment corresponds to, for example, a state in which the portion other than the three-dimensional hand model 82 in FIG. 4 is actual imaging data. In the second embodiment, only the sub-coordinate system exists as a coordinate system.

(Effects of the second embodiment)
According to the second embodiment, hand parts are displayed as a three-dimensional model on the display 2 on the image data (real space) actually captured by the camera 3. The position and orientation of the target marker 93 changes in accordance with the actual movement of the arm 61, and the position and orientation of the three-dimensional model on the display 2 also change. In this way, the position and orientation of the three-dimensional model of the hand component appearing in the real space on the display 2 changes due to a change in the image data of the target marker 93.

According to this configuration, when the robot arm 6 is actually placed in a factory and tested, the target marker 93 may come into contact with other devices, but the actual hand parts may not come into contact with other devices. do not have. Therefore, damage to the hand parts is prevented. Furthermore, it is possible to perform operation/interference tests on multiple types of hand parts simply by changing the data of the displayed three-dimensional model without replacing the actual hand parts. As described above, according to the second embodiment, damage to the hand components of the robot arm 6 can be prevented, and operation/interference tests of the hand components can be easily performed.

<Others>
The present invention is not limited to the above embodiments. For example, the display 2 and the camera 3 may be arranged integrally, such as in a head-mounted display or a tablet terminal, or they may be communicably connected by wire or wirelessly so that they can move relative to each other. Good too. According to the tablet terminal 100, MR images can be easily checked by multiple people. Moreover, each function may be realized not only by the tablet terminal 100 but also by another computer.

Further, as a reference, the above test can also be performed by arranging a plurality of cameras and using an optical sensor that detects the position of the spherical reflective marker. However, as in the above embodiment, an apparatus using a flat target marker allows for simpler equipment and allows for easier testing.

According to the first embodiment, unlike AR, two coordinate systems are set separately, allowing the three-dimensional hand model 82 to move independently. Furthermore, in the first embodiment, a three-dimensional virtual factory can be reproduced in a virtual space, and it is possible to integrate a real machine and a virtual factory space.

Moreover, according to the robot arm testing apparatuses 1 and 1A, the sense of distance between the CG and the actual machine can be accurately reflected and displayed on the display 2. Moreover, the robot arm testing devices 1 and 1A can be used, for example, for training workers, for robot teaching, or as a training machine. The robot arm testing device 1A enables virtual preliminary interference confirmation with obstacles in the installed factory space. According to the robot arm testing devices 1 and 1A, robot motion can be easily confirmed in a factory or virtual factory space. The robot arm testing device can also be called a virtual robot hand simulation system. The hand parts can also be called a robot hand.

DESCRIPTION OF SYMBOLS 1, 1A...Robot arm test device, 2...Display, 3...Camera, 4...Position and orientation measurement unit, 5...Information processing unit, 6...Robot arm, 61...Arm, 81...Three-dimensional model (for example, based on point cloud data) background image), 82... three-dimensional hand model, 91... first target marker, 92... second target marker, 93... target marker.

Claims (5)

display and
A camera that captures images of the outside,
The position and orientation of the camera are measured based on imaging data of a first target marker that is a target marker in a first coordinate system captured by the camera and coordinate information set for the first target marker. a position and orientation measurement unit;
an information processing unit that presents mixed reality, which is a superimposed image of real space and a three-dimensional model, to the user via the display based on the position and orientation of the camera measured by the position and orientation measurement unit;
a robot arm in which a second target marker, which is a target marker of a second coordinate system independent from the first coordinate system, is attached, instead of the hand component, at a location at the tip of the arm where the hand component is scheduled to be placed; ,
a storage unit that stores a three-dimensional hand model representing the hand component, which is set corresponding to the second target marker;
Equipped with
The three-dimensional model is actual machine layout data,
The first coordinate system is a world coordinate system,
The second coordinate system is a sub-coordinate system whose origin moves as the second target marker moves on the first coordinate system,
The information processing unit includes:
displaying the three-dimensional model on the display based on the imaging data of the first target marker; displaying the three-dimensional hand model on the first coordinate system based on the imaging data of the second target marker; A robot arm testing device that superimposes and displays imaging data of the arm on the three-dimensional model and the three-dimensional hand model as the real space. A three-dimensional object is attached to a location at the tip of the arm where the hand component is scheduled to be placed,
The second target marker has a flat plate shape,
a plurality of the second target markers are three-dimensionally arranged on the surface of the three-dimensional object;
The robot arm testing device according to claim 1. A plurality of the first target markers are displayed on a wall surface, an installation surface, or a display disposed behind the robot arm with respect to the camera,
The robot arm testing device according to claim 1 or 2. The arm is colored so that at least part of the color is a specific color different from a background color of the arm in the imaging data of the camera,
The robot arm testing device according to claim 1 , wherein the information processing section displays the specific color in front of the three-dimensional model. 5. The information processing unit according to claim 1, wherein the information processing unit executes special display processing including at least one of a color change processing and a blinking processing on a contact portion between the three-dimensional model and the three-dimensional hand model. Robotic arm test equipment as described in Section .

Images