JPWO2020044491A1 - robot - Google Patents

Abstract

The robot (1) has a pedestal (11) having a surface in contact with the installation surface (5), a support column (12) vertically attached to the pedestal (11) and rotatable around a vertical axis, and the support column (12). ), And an arm (15, 16) connected to the tip of the arm and capable of changing the direction by a single or a plurality of joints (21, 22, 23), and changing the direction with the arm. A plurality of first cameras (14L, 14R) provided at a position fixed to the support column and capable of photographing the pedestal itself and the installation surface around the pedestal. ) And the first image is acquired from the plurality of first cameras, the partial plan view image is generated by coordinate-transforming the first image, and the plurality of the partial plan views are aligned and combined. A control unit (40) that generates a plan view image including an image of the pedestal and controls the direction of the arm in order to move the tip portion to a predetermined position based on the plan view image. , Equipped with.

Description

The present invention relates to a technique for acquiring and processing an image in order to realize automation of robot movement.

Automation of work in factories and laboratories using industrial robots (equipped with arms, installed in the work area and usually not moving by themselves; hereinafter, "robot" is defined by this definition). Is progressing. When a robot works, it is necessary to acquire information on the work target and its surroundings. Conventionally, a robot for acquiring an image using a camera and processing the image to obtain information such as the shape and position of an obstacle such as a work target and its surroundings and perform a desired work. There are known ways to calculate behavior.

In the system described in Patent Document 1 (FIG. 1 and the like), the fixed camera 5 is installed in a state of being fixed above the tray 7 which is the place where the robot works, and the tray 7 is photographed. The fixed camera coordinate system F is a known coordinate system. Then, the image taken by the fixed camera 5 is used to acquire the amount of misalignment of the work object, and the amount of misalignment is used for controlling the robot.

In the technique described in Patent Document 2, the robot has two systems of imaging means, a two-dimensional vision system 60 and a three-dimensional vision system 70, for processing an electric wire group. Then, according to FIG. 3 (flow chart) and the like, first, the two-dimensional vision system 60 is used to acquire the first image data D1 which is an image of the range including the entire electric wire group, and then based on the first image data D1. The determined region (R2) is imaged by the three-dimensional vision system 70 to obtain the second image data D2. Then, based on the second image data D2, the position of the processing target and the like are recognized, and an instruction to the processing robot is given.

In the technique described in Patent Document 3, a laser sensor 26 and a vision sensor 27 are provided at the tip of the robot 24 for sorting articles (FIG. 5 and the like). Then, according to FIG. 12 (flow chart) and the like, first, the distance to the upper surface of the load is acquired by using the laser sensor 26, and the load whose upper surface is at the highest position is specified. Then, after that, the external shape information and the like of the specified baggage are acquired by using the vision sensor 27.

In the technique described in Patent Document 4, a mark object is photographed using a fixed camera, and the position data is calibrated from the existing coordinate system to the newly set coordinate system based on the mark object. ..

In the techniques described in Patent Document 1 and Patent Document 4, the position information of the work target is acquired from the image taken by the fixed camera in the work environment of the robot.
Also in the technique described in Patent Document 2, it is expected that the camera 62 for acquiring the first image data D1 in the first stage is supported by the camera support member 64, and the electric wire group is arranged. The region R1 which is the entire region is arranged so as to be able to image (paragraph 0036).
In the technique described in Patent Document 3, both the laser sensor 26 and the vision sensor 27 are provided at the tip portion of the robot, and their positions are variable according to the movement of the arm.

In this way, the conventional technology uses a fixed camera, and it is necessary to install it at a position sufficiently higher than the robot in order to overlook the movable range of the robot, and a solid structure for that purpose is installed. I had to do it. In addition, since the camera is installed in a structure independent of the robot body, the positional relationship between the robot body and the installation surface and the positional relationship between the structure in which the camera is installed and the installation surface are grasped in advance, and the position is grasped. It was necessary to operate the robot on the premise of the relationship. During the operation of the robot, it was assumed that both the robot body and the structure on which the camera was installed were used in a state of being firmly fixed to the installation surface so that the positional relationship would not change.
In addition, in the conventional technology using a fixed camera, it is necessary to shoot an object from a fixed position, so the camera is fixed by devising the arrangement and angle so that there is no blind spot due to the robot itself or a structure. I needed it.
Furthermore, in the conventional technology using a fixed camera, the accuracy of the lens is good and the resolution of the image sensor is also high so that the shape and three-dimensional structure of the object can be recognized in addition to the place where the object is placed from a high position. It was necessary to use a high-precision, high-precision camera module.
On the other hand, for example, a relatively small robot is not a robot that is permanently or semi-permanently fixedly installed in a work environment, but is a robot that can be flexibly carried and used anywhere. Instead of being fixedly installed in a specific place, such a small robot can move the robot on the spot by simply placing the pedestal on the floor or work table, etc., and pick up or sort objects. It can be used for such purposes.
Alternatively, in the case of a mobile robot equipped with moving rails and wheels, it moves autonomously over a wide range, picks up objects, sorts them, and puts them on different lines or stores them in remote locations. It is possible to do it.

Such a portable robot can easily perform work at the placed position, but it is difficult to grasp the position with high accuracy with respect to the work environment (work place, etc.) as in Patent Documents 1, 2 and 4. I had a problem. It is conceivable to place a fixed camera or the like outside the portable robot to acquire position information, but it is necessary to firmly fix the portable robot body and the fixed camera while the robot is in operation. Since the placement of the portable robot changes each time it is carried and installed, it is necessary to readjust the placement and angle of the fixed camera. Therefore, when a fixed camera or the like must be installed, the mobility of the portable robot is sacrificed.
Further, in the case of a mobile robot, since a fixed camera overlooks the movable range of the robot, it is necessary to install a large number of fixed cameras at regular intervals according to the length of the moving rail, for example. A plurality of high-precision camera modules are required for one robot, the scale of the structure is large, and the cost is high.
Alternatively, when the robot moves in a free trajectory, there are too many degrees of freedom in the place where it can move, and it is not realistic to install a fixed camera.

JP-A-2017-071033 Japanese Unexamined Patent Publication No. 2016-192135 Japanese Unexamined Patent Publication No. 2013-086915 Japanese Unexamined Patent Publication No. 2000-076441

The problem to be solved by the present invention is to be able to acquire highly accurate position information about the work environment necessary for a portable robot or a mobile robot to work without installing a fixed camera or the like. It is to be.

[1] In order to solve the above problems, the robot according to one aspect is connected to a pedestal having a point in contact with the installation surface, a support column vertically attached to the pedestal and rotatable around a vertical axis, and the support column. An arm that can be turned by a single or multiple joints, a tip that is connected to the tip of the arm and can be turned with the arm, and fixed to the strut. A plurality of first cameras provided at positions and capable of photographing the pedestal itself and the installation surface around the pedestal, and first images are acquired from the plurality of first cameras, and the first images are coordinate-converted. By doing so, a partial plan view image is generated, and by aligning and synthesizing the plurality of the partial plan views, a plan view image including the image of the pedestal is generated, and the above-mentioned A control unit for controlling the direction of the arm in order to move the tip portion to a predetermined position is provided.

[2] Further, in one aspect, the robot further includes an irradiation unit which is fixedly provided to the tip portion and irradiates light having a predetermined pattern in the direction of the installation surface.

[3] Further, in one aspect, the robot further includes a second camera provided at the tip end portion, and the control unit further includes the arm based on a second image acquired from the second camera. Control to change the direction of.

[4] Further, in one aspect, the robot further includes an autonomous moving unit for autonomously moving on the installation surface.

[5] In one aspect, in the robot, the first camera and the second camera simultaneously photograph an object, and the control unit simultaneously photographs the first camera and the second camera, respectively. It is controlled to change the direction of the arm based on the image.

[6] In one aspect, in the robot, the first camera and the second camera photograph an object in different phases, and the control unit uses the first camera for each phase. It is controlled to change the direction of the arm based on the captured first image, and is controlled to change the direction of the arm based on the second image captured by the second camera.

[7] Further, in one aspect, the robot includes a plurality of the irradiation units, and at least a part of the irradiation units emits the light in a direction different from that of the other irradiation units. Irradiate.

[8] Further, in one aspect, in the robot, the control unit moves the tip portion so that the irradiation unit sequentially irradiates the light in the directions of a plurality of predetermined positions on the installation surface. While controlling, the control unit generates a plurality of the plan view images corresponding to each of the timings at which the irradiation unit irradiates the light in the direction of the plurality of predetermined positions, and the control unit generates a plurality of the plan view images. The step or unevenness of the installation surface is calculated based on the position of the light of the predetermined pattern included in the plurality of plan view images.

Since the robot of the present invention attaches the camera to the robot body, a structure for attaching the camera other than the robot is not required. Therefore, the robot can be operated by grasping only the positional relationship between the robot body and the installation surface. Further, since the pedestal of the robot can be photographed together with the installation surface around the robot, the positional relationship between the robot body and the installation surface can be automatically grasped. Therefore, even after moving the robot, it is not necessary to perform an operation for setting the position.

It is a side view which shows the schematic structure of the robot by 1st Embodiment. In the first embodiment, it is a schematic diagram which shows the synthesized image generated based on the image taken by the 1st camera. In the first embodiment, it is a schematic diagram which shows the synthesized image including a plurality of points which irradiate the installation surface by a laser irradiation part. It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 1st Embodiment (the 1). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 1st Embodiment (the 2). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 1st Embodiment (the 3). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 1st Embodiment (the 4). It is the schematic which shows the relationship between the arrangement of the laser irradiation part which is a characteristic part of 2nd Embodiment, and the surface which is irradiated with a laser. It is a schematic diagram which shows the 1st example of the pattern irradiated by the laser irradiation part by 2nd Embodiment. It is a schematic diagram which shows the 2nd example of the pattern irradiated by the laser irradiation part by 2nd Embodiment. It is the schematic explaining the principle for calculating the distance from the laser irradiation part to the irradiation surface by the robot by 2nd Embodiment. It is a schematic diagram explaining the principle for calculating the distance from the laser irradiation part to the irradiation surface by the robot by the modification of 2nd Embodiment. It is a schematic diagram which shows the moving operation of the robot by 3rd Embodiment. It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 3rd Embodiment (the 1). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 3rd Embodiment (the 2). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 3rd Embodiment (the 3). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 3rd Embodiment (the 4). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 4th Embodiment (the 1). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 4th Embodiment (the 2). It is a flowchart which shows the procedure of the process for controlling the operation which the robot grasps a target and carries it to a place position by 4th Embodiment (the 3). It is a perspective view which shows the operating environment of the robot when the installation surface has a step in 5th Embodiment. It is a schematic diagram which shows the synthesized image generated based on the image taken by the 1st camera in 5th Embodiment. It is an image showing a point on the installation surface irradiated by the laser irradiation unit 32 in the fifth embodiment, and is a composite image obtained by superimposing all the images taken by the first camera. It is an image showing a point on the installation surface irradiated by the laser irradiation unit 32 in the fifth embodiment, and is a composite image when there is a step on the installation surface. FIG. 5 is a schematic view (side view) for explaining a principle for calculating the distance from the laser irradiation unit to the irradiation surface having a step by the robot according to the fifth embodiment. It is a plan view corresponding to FIG. It is the schematic which extracted the geometric relation of each point in the side view of FIG.

Hereinafter, embodiments will be described with reference to the drawings.

[First Embodiment]
[1. Robot configuration]
FIG. 1 is a side view showing a schematic configuration of a robot according to the first embodiment. As shown in the figure, the robot 1 includes a pedestal 11, a support column 12, first cameras 14L and 14R, a first arm 15, a second arm 16, a first joint 21, a second joint 22, and a second. It includes three joints 23, a tip (hand) 31, a laser irradiation unit 32, a finger 33, a second camera 34, and a control unit 40. The left side of the figure is the front side of the robot 1, and the right side is the back side of the robot 1. The robot 1 is a relatively small device having a size that can be carried by a person. The robot 1 can be used by placing it on the installation surface 5. The bottom surface of the pedestal 11 is in contact with the installation surface 5. Unless an external force acts, the pedestal 11 is fixed at a predetermined position on the installation surface 5 and does not move. The support column 12 is a part of the arm, and its position with respect to the pedestal 11 is fixed. However, the support column 12 can rotate about a rotation axis perpendicular to the installation surface 5.

The first cameras 14L and 14R may be referred to as "stereo cameras". Further, the second camera 34 may be referred to as a "advanced camera".

The pedestal 11 supports the entire robot 1. The bottom surface of the pedestal 11 is flat and can be placed on the installation surface 5. That is, the pedestal 11 has a point or surface in contact with the installation surface 5.
The support column 12 is fixed to the pedestal 11 and supports the tip of the first arm 15.

The first cameras 14L and 14R are cameras provided on the left side and the right side of the support column 12, respectively. In this embodiment, two first cameras 14L and 14R are fixedly provided on the support column 12. The first cameras 14L and 14R are provided so as to photograph the front side (left side in FIG. 1) of the main body of the robot 1. Further, the main axes of the lenses of the first cameras 14L and 14R are provided so as to face slightly downward on the front side of the main body of the robot 1. Therefore, the images captured by the first cameras 14L and 14R include many installation surfaces 5. A part of the robot 1 itself (for example, a part of the pedestal 11, a part of the support column 12, etc.) may be reflected in the images captured by the first cameras 14L and 14R. That is, the first camera is provided at a position fixed to the support column 12 so that the pedestal 11 itself and the installation surface 5 around the pedestal can be photographed.

The number of first cameras may be three or more. As will be described later, the first camera is used for the purpose of covering the entire circumference of the main body of the robot 1 as a shooting range. Therefore, a plurality of first cameras are provided in order to prevent blind spots from occurring. Further, at least a part of the plurality of first cameras may be fixedly provided on the pedestal 11 instead of the support column 12. In any of these cases, each of the plurality of first cameras acquires an image having a fixed angle of view with respect to the installation surface 5.
A plurality of first cameras may be collectively referred to as "first camera 14".

The first arm 15 and the second arm 16 are robot arms. The first arm 15 and the second arm 16 are collectively referred to simply as "arms". The arm is directly or indirectly connected to the pedestal 11 and can be turned by a single or multiple joints.
The first joint 21 is a joint that connects the support column 12 and the first arm 15.
The second joint 22 is a joint that connects the first arm 15 and the second arm 16.
The third joint 23 is a joint that connects the second arm 16 and the tip portion 31 (also referred to as a “hand”).
Each of the first joint 21, the second joint 22, and the third joint 23 enables rotation about a rotation axis parallel to the installation surface 5 or a rotation axis perpendicular to the installation surface 5. Any joint may rotate on multiple axes.
By the action of these arms and joints, the robot 1 can move the tip portion 31 to an arbitrary position within the movable range. Further, the robot 1 can turn the tip portion 31 in any direction within the movable range.
In the robot 1, the state of movable parts such as joints (for example, displacement rotation angle, etc.) can always be grasped by the control unit 40.

Here, the robot 1 having two arms and three joints is configured, but the number of arms and the number of joints are not limited to those illustrated here, and are arbitrary. Is. An appropriate number of joints and an appropriate number of arms may be used depending on the design and the like.

The tip portion (hand) 31 has a function for the robot 1 to perform work. Specifically, a laser irradiation unit 32, a finger 33, and a second camera 34 are attached to the tip portion 31. As described above, the tip portion 31 can approach the object or the like (target) to be worked by the arm and the joint. Due to the function of the tip portion 31, the robot 1 can acquire an image of the target, grab the target, and carry the grabbed target. That is, the tip portion 31 is connected to the tip of the arm via a joint or the like. The orientation between the tip 31 and the second arm 16 can be changed.

The laser irradiation unit 32 (irradiation unit) irradiates the laser beam mainly in the direction toward the installation surface 5 (perpendicularly or diagonally to the installation surface 5). Although FIG. 1 shows only one laser irradiation unit 32, the laser irradiation unit 32 may be provided at a plurality of locations. Further, one laser irradiation unit 32 may irradiate the laser beam in a plurality of directions at the same time or in a time division manner. As the laser irradiation unit 32, for example, a cross laser is used. The cross laser irradiates a cross-shaped cross pattern.
That is, the laser irradiation unit 32 is fixedly provided to the tip portion 31 and irradiates light having a predetermined pattern in the (almost) installation surface direction.

The finger 33 is a member having a mechanism capable of grasping an object to be worked on. The finger 33 can not only grab the object but also push and move the object placed on the installation surface 5, for example. Further, the finger 33 may be provided with a mechanism for adsorbing an object.

The second camera 34 is a camera fixedly attached to the tip portion 31. The first camera described above acquires an image at a height position and an angle of view fixed with respect to the installation surface 5, whereas the second camera 34 acquires the image at the tip portion 31 as the tip portion 31 moves. It is possible to approach various places and acquire images within the constraints of the movable range of. The second camera 34 normally captures an image in a downward direction (a direction substantially facing the installation surface 5).
The image (second image) taken by the second camera 34 is used for grasping the position of the object and grasping the state of the finger 33 (whether or not the finger 33 is grasping the object). Be done. Based on these grasps, the movement and work of the arm of the robot 1 are planned.
Although only one second camera 34 is shown in FIG. 1, a plurality of second cameras 34 may be provided at the tip portion 31.

As described above, the robot 1 acquires visual information from the images taken by the first camera 14 and the second camera 34. The robot 1 may acquire information other than vision by having various sensors and the like.

The control unit 40 has a function for controlling the entire robot 1. The control unit 40 is realized by, for example, an electronic circuit. The control unit 40 may be realized by using a computer (microcomputer chip or the like). When a computer is used, a function for controlling the robot 1 can be described by a program. It is also possible to store the program in a storage means (for example, a semiconductor memory or the like) included in the control unit 40 so that the computer executes the program. The position where the control unit 40 is provided is arbitrary, but for example, the function of the control unit 40 can be stored inside the pedestal 11 as shown in the figure. Further, the control unit 40 may be composed of a plurality of devices. For example, a hardware board specialized for arm control of the robot 1 is stored inside the pedestal 11, and image processing from the camera and the control method of the arm are realized by software by a computer outside the robot 1, and the robot The configuration may be such that 1 and an external computer communicate with each other to realize the processing of the control unit 40.

When the robot 1 operates, the control unit 40 acquires images taken by the first camera 14 and the second camera 34, and acquires signals from various sensors provided in various parts of the robot 1. The control unit 40 makes a necessary determination based on the information or the signal, and as a result, outputs a control signal for operating each member of the robot 1. The control unit 40 outputs, for example, an instruction signal for bringing the first joint 21, the second joint 22, and the third joint 23 into a desired state. Further, the control unit 40 outputs an instruction signal so that the laser irradiation unit 32 and the finger 33 of the tip portion 31 perform a predetermined operation.

The control unit 40 acquires the first image from the plurality of first cameras, generates a partial plan view image by coordinate-transforming the first image, aligns the plurality of partial plan views, and synthesizes the images. Generate a plan image that includes an image of the pedestal. Further, the control unit 40 controls to change the direction of the arm in order to move the tip portion 31 to a predetermined position based on the plan view image.
Further, the control unit 40 controls to change the direction of the arm based on the second image acquired from the second camera 34.

[2. Generation of floor plan image Understanding the situation around the robot]
The control unit 40 generates an image corresponding to one plan view based on the images acquired by the plurality of first cameras 14L and 14R. Therefore, the control unit 40 first performs coordinate transformation (affine transformation) on each of the images obtained from the first cameras 14L and 14R. Then, the control unit 40 then generates one large image by synthesizing (pasting) each coordinate-transformed image.

FIG. 2 is a schematic view showing one plan view image generated by the control unit 40 based on the images taken by the first cameras 14L and 14R, respectively. In the figure, 101L is a partial plan view obtained by coordinate-transforming an image taken by the first camera 14L. Further, 101R is a partial plan view obtained by converting the coordinates of the image taken by the first camera 14L. Each of the first cameras 14L and 14R shoots with a lens directed diagonally downward, but by using the coordinate transformation which is an existing technique, the image of each camera can be used as a part of the plan view. Such images are referred to as coordinate-transformed images 101L and 101R, respectively. Each of the coordinate-transformed images 101L and 101R corresponds to an image in which the installation surface 5 is viewed in a plan view from above. The control unit 40 aligns the coordinate-transformed images 101L and 101R at appropriate positions, appropriately superimposes the overlapping areas of both images, or cuts off the composited image 101 as necessary. To get. A part of the pedestal 11 is reflected in the composited image 101. In addition, the installation surface 5 is shown in most of the composited image 101. Further, when objects or the like are placed within the photographing range on the installation surface 5, an image of those objects viewed in a plan view is also included in the synthesized image 101.

In the present embodiment, the first cameras 14L and 14R are arranged on the left and right sides of the support column 12 of the robot 1, respectively, but the method of arranging the plurality of first cameras 14 is not limited to the left and right sides. Further, the number of the first cameras 14 to be used is not limited to two, and for example, three or more first cameras may be used. As an example, a total of four first cameras are provided on the front, back, left, and right sides of the robot, and each of these cameras photographs the front, rear, left, and right, and after coordinate-converting the four images, 1 It may be combined with a sheet of plan view images. Further, the synthesized image 101 may cover the entire circumference of the robot 1 (360 degrees in a plane), or may cover only a part of the angle range thereof. The composite image 101 illustrated in FIG. 2 is a plan view image mainly showing the front side (including a part of the left side and the right side). The composited image 101 may be an image showing only the range related to the work of the robot 1.

Parameters (coefficients of the transformation matrix, etc.) when the control unit 40 performs coordinate transformation (affine transformation) may be set in advance for each individual robot 1, for example. Alternatively, a sign indicating a reference position is placed on the installation surface 5 to be photographed, and the parameters of coordinate conversion are dynamically determined based on the coordinate position of the sign included in the photographed image in the image. You may. As the sign, for example, a bar code, a two-dimensional code, or the like is used. The same applies to the subsequent recognition of a specific object or place using a sign.

The process in which the control unit 40 generates (a partial image) of the coordinate conversion process plan view image based on the images taken by the plurality of first cameras may be referred to as "around view process".

In this way, the control unit 40 creates a plan view image based on the images taken by the plurality of first cameras 14. As a result, the robot 1 can acquire information on the plan view of a relatively wide range around the robot 1 (for example, a range covering the movable area of the tip portion 31) without a blind spot. The information on the plan view obtained in this way includes information for planning the work to be executed by the robot 1, so that the robot 1 can smoothly proceed with the work.

The support column 12 is rotatable about an axis perpendicular to the installation surface, but since the first camera 14 captures a range including the pedestal of the robot 1, the control unit 40 is generated on a flat surface. It is also possible to grasp the rotation direction based on the figure image.

Further, the control unit 40 grasps the orientation of the pedestal 11, the orientation of each arm (first arm 15 and the second arm 16), and the orientation of the tip portion 31 based on the generated plan view image. Specifically, the control unit 40 grasps the orientation of the pedestal 11 shown in the generated plan view image in the surrounding environment. In other words, the control unit 40 has a position of a peripheral object or the like (may include a sign or the like indicating a known position) shown in the generated plan view image and a position of the pedestal 11 of the own robot 1. Understand the relationship. Then, the control unit 40 grasps the direction of the first arm 15 from the position of the pedestal 11 and the state of the first joint 21 (displacement angle about the rotation axis of the joint). Then, the control unit 40 grasps the direction of the second arm 16 from the direction of the first arm 15 and the state of the second joint 22 (displacement angle about the rotation axis of the joint). Then, the control unit 40 grasps the direction of the tip portion 31 from the direction of the second arm 16 and the state of the third joint 23 (displacement angle centered on the rotation axis of the joint). Since the lengths of the first arm 15 and the second arm 16 are fixed, the control unit 40 can also grasp the positions of the second joint 22 and the third joint 23, respectively. Further, the control unit 40 can grasp the position of the tip portion 31 based on the position of the third joint 23.
The control unit 40 can appropriately store the direction of each arm, the position of each joint, and the position and direction of the tip portion 31 obtained by the above method.

Further, the control unit 40 can grasp the position of the target (object, place, etc. to be worked) based on the position of the pedestal 11 of the own robot 1 based on the generated plan view image. The control unit 40 can grasp the position of the target in the generated plan view image by learning the characteristics of the target as an image in advance. Alternatively, the GUI (graphical user interface) may allow the user to specify the position of the target. When the GUI is used, the control unit 40 displays the generated plan view image on the display of the terminal device operated by the user, and the control unit 40 indicates the position of the target on the plan view image of the target. Grasp the position. By these methods, the control unit 40 can grasp the position of the target with respect to the own robot 1 with a predetermined accuracy. That is, the control unit 40 can make a plan to move, for example, the tip portion 31 closer to the target based on the grasped position of the target.

[3. Estimating the structure of the target by irradiating the laser and the action of grabbing the target]
The laser irradiation unit 32 provided at the tip portion 31 irradiates the laser. The second camera 34 provided at the tip portion 31 can also take a picture in a state of being irradiated with a laser. The laser irradiation unit 32 may irradiate a laser having a predetermined pattern (a laser that irradiates the projection surface with a predetermined shape). Further, the laser irradiation unit 32 may irradiate only one type of laser, or may irradiate the laser from a plurality of positions or at a plurality of angles. Further, only one second camera 34 may be provided, or a plurality of second cameras 34 may be provided. In either case, the control unit 40 estimates the distance to the target object and the direction of the target object based on the laser beam projected on the target object (target). At this time, the control unit 40 may refer to the information regarding the shape of the target object acquired in advance.

If the distance or orientation to the target object cannot be estimated, the control unit 40 slightly changes the position or orientation of the tip portion 31 by moving the arm by a predetermined amount. The control unit 40 re-estimates the distance and direction to the target object based on changes in the image (including changes in the shape or size of the laser beam projected on the target object) before and after moving the tip portion 31. It becomes possible.

The control unit 40 calculates the amount of movement of the arm to bring the tip portion 31 closer to the target object based on the information such as the distance and the direction to the target object obtained by analyzing the image of the second camera 34. .. Further, when grasping the target object, the control unit 40 calculates the amount of movement of the third joint 23 based on the information of the appropriate orientation of the tip portion 31 for that purpose. Further, if necessary, the control unit 40 calculates the amount of movement of the finger 33 for grasping the target object. Then, the control unit 40 grasps the target object by actually controlling the movement of each joint or finger based on these calculated values.

After performing the grasping operation, the control unit 40 further instructs the second camera 34 to take a picture. Then, the control unit 40 acquires the image taken by the second camera 34. When a plurality of second cameras 34 are provided, images may be acquired from each of the plurality of cameras. By analyzing the image, the control unit 40 can determine whether or not the finger 33 has been able to grab the target object. In addition, instead of making a judgment by the second camera, it may be judged by the first camera. Further, a limit switch or a tactile sensor may be mounted on the finger 33, and whether or not the target object can be grasped may be determined by the output signal.

[4. Operation to carry and store the grabbed object to the place position]
The control for the operation of carrying the object from the state where the finger 33 of the robot 1 is grasping the object to the place position is as follows. Here, the "place" is a place designated for placing an object or the like, or a container or the like for storing the object or the like.
Since the position where the finger 33 grabs the target object is substantially the same height as the installation surface, the arm is raised to a certain height position so as not to interfere with the installation surface or the surroundings when the target object moves.
The first cameras 14L and 14R perform photographing. The control unit 40 acquires images from the first cameras 14L and 14R, and generates a plan view image using those images. The method for generating the plan image is as described above.
Next, the control unit 40 grasps the place position based on the generated plan view image. The control unit 40 has learned the characteristics of the image at the place position in advance, and automatically recognizes the position from the plan view image. Alternatively, the GUI may allow the user to specify the position of the place. When the GUI is used, the control unit 40 displays the plan view image on the display of the terminal device and acquires the position information on the plan view image instructed by the user.

Next, the control unit 40 makes a plan to move the tip portion 31 from the current position to the upper part of the place position. The process of creating this movement plan is the same as the above-mentioned process of planning to move the tip portion 31 to the upper part of the target object. The control unit 40 moves the tip portion 31 to the upper part of the place position by controlling the movement of the arm based on the created plan. This movement is parallel to the xy plane.
Next, the control unit 40 acquires an image taken by the second camera and analyzes the image to confirm whether or not the tip portion 31 is above the place position. If it can be confirmed, the control unit 40 controls to store the object in the place position. That is, the control unit 40 controls the finger 33 so as to store the object at that position (that is, release the grasped object) after controlling the tip portion 31 to be lowered to the place position.

The robot 1 may not only use the fingers 33 to grab and release an object, but may also perform an operation such as pushing and moving an placed object. Further, the robot 1 may suck an object by using the suction mechanism of the finger 33, or may move the object in the sucked state. Also in this case, the control unit 40 appropriately moves the tip portion 31 and appropriately operates the finger 33.

[5. Laser irradiation]
By irradiating the laser from the laser irradiation unit 32 provided on the tip portion 31, the height of the installation surface (step and unevenness on the surface) can be measured. The explanation will be given below.

(1) Utilization of a plurality of laser irradiation units A plurality of laser irradiation units 32 can be provided on the tip portion 31.
FIG. 3 is a schematic view showing a state of irradiation by the laser irradiation unit 32. The figure shows a situation in which the two laser irradiation units 32 irradiate the installation surface 5 with the two irradiation lights P1 and P2, respectively. P1 and P2 are laser beams irradiated in dots or substantially dots. Here, let d be the distance between the points P1 and P2 on the installation surface 5. The first cameras 14L and 14R can take an image including these points P1 and P2 irradiated on the installation surface 5, and the control unit 40 can acquire the image. Note that FIG. 3 is a composited image obtained by synthesizing those images.

(2) Use of Lasers with Different Mounting Angles When a plurality of laser irradiation units 32 are provided on the tip portion 31, at least one of the plurality of laser irradiation units 32 is a laser in a direction different from that of the other laser irradiation units 32. It is possible to adjust to irradiate. In this case, if the height of the tip portion 31 (distance from the installation surface 5) changes, the distance between the two laser beams irradiated on the installation surface (distance d in FIG. 3) also changes according to the change. .. Utilizing this, the height from the tip portion 31 to the installation surface can be calculated based on the distance d between the laser beams emitted on the installation surface 5. The distance d on the installation surface 5 is obtained by performing an appropriate scale processing on the image acquired by the control unit 40.
As a result, the control unit 40 can obtain the amount of movement in the height direction when the tip portion 31 is moved, regardless of whether the finger 33 is holding the object or not. can.

(3) Use of a laser that irradiates the installation surface vertically
By controlling the robot, the control unit 40 sequentially moves the tip portion 31 to a plurality of preset set positions on the installation surface 5. The control unit 40 controls the irradiation by the laser irradiation unit 32 provided so as to irradiate the laser light directly downward (perpendicular to the installation surface 5) on each set position. The control unit 40 acquires images taken by the first cameras 14L and 14R each time. A composited image can be obtained by synthesizing the captured image by performing coordinate transformation and alignment. In the synthesized image obtained by performing the above operation at each set position, the spot of the irradiation light should appear at the same position. However, if there is a step or unevenness on the installation surface 5, the position of the point of the irradiation light shifts. The control unit can detect a deviation in the height of the installation surface 5 by performing a process of superimposing the combined images obtained at each set position.

By calculating the deviation of the irradiation points on the installation surface 5, the unevenness of the installation surface 5 can be known. The control unit 40 can determine the height at which the target is grabbed by this calculation.

[6. Processing procedure]
4, FIG. 5, FIG. 6, and FIG. 7 are flowcharts showing a processing procedure for controlling an operation in which the robot 1 grabs a target object and carries it to a place position. 4, FIG. 5, FIG. 6, and FIG. 7 are one flowchart connected by a coupler. Hereinafter, the processing procedure will be described with reference to this flowchart.

First, in step S1 of FIG. 4, the control unit 40 acquires the images taken by the first cameras 14L and 14R.
Next, in step S2, the control unit 40 performs an around view process (coordinate conversion process) of each image acquired in step S1 and performs a process of synthesizing those images. As a result, the control unit 40 generates a plan view image of the periphery of the robot 1.
Next, in step S3, the control unit 40 recognizes the pedestal 11 included in the plan view image generated in step S2. In order to recognize the pedestal 11, the control unit 40 learns the characteristics of the pedestal 11 as an image in advance. Alternatively, the control unit 40 recognizes the pedestal 11 by recognizing a sign (characteristic image) attached to a predetermined portion of the pedestal 11. Alternatively, the control unit 40 acquires in advance the approximate coordinate positions of the pedestal 11 in the images taken by the respective first cameras (14L and 14R). The control unit 40 may use a plurality of the methods illustrated here in combination in order to recognize the pedestal 11.

Next, in step S4, the control unit 40 uses the method described above to determine the orientation and angle of each arm (first arm 15, second arm 16) of the robot 1 and the orientation of the support column 12 (centered on the rotation axis). The direction of rotation) and the position and orientation of the tip 31 are calculated and stored. The control unit 40 calculates the orientation of each arm and the position and orientation of the tip portion 31 as a relative orientation or position with reference to the pedestal 11. Since the pedestal 11 in the plan view image is recognized in step S3, the control unit 40 can grasp the orientation of each arm and the position and orientation of the tip portion 31 with reference to the coordinates of the plan view image.
Next, in step S5, the control unit 40 recognizes the target (object or place) to be operated in the plan view image. As described above, the control unit 40 can recognize the target in the plan view image by using the characteristics of the target acquired in advance. Further, the control unit 40 may recognize the target by the designation by the user (designation from the GUI).

Next, in step S6, the control unit 40 plans the movement of the arm and the tip portion 31. The control unit 40 plans the movement of the arm and the tip portion 31 based on the current position of the arm and the tip portion 31 already grasped and the position of the target. For example, the control unit 40 plans the movement of the arm so that the tip portion 31 is moved in parallel with the xy plane (installation surface 5) to bring it to the upper part of the target position. At this time, the control unit 40 may perform a calculation using Cartesian coordinates (x-axis and y-axis), or may perform a calculation using polar coordinates with a predetermined position of the pedestal 11 as a pole. The "upper part of the target position" does not necessarily have to be the position directly above the target (the position moved upward from the target position in the xy plane perpendicular to the xy plane). For example, a position where the second camera 34 attached to the tip portion 31 can easily recognize the target or a position where the finger 33 can easily operate the target may be defined as the “upper part of the target position”.
Next, in step S7, the control unit 40 controls to move the arm based on the plan established in step S6. As a result, the tip portion 31 moves above the target position unless some unexpected situation (for example, the robot 1 itself is lifted and its position is changed) occurs.

Next, in step S8, the control unit 40 acquires an image taken by the second camera 34.
Next, in step S9, the control unit 40 analyzes the image acquired in step S8 and recognizes the target in the image. In addition, the control unit 40 grasps the position of the target in the image. The purpose of recognizing the target in this step is to check whether or not the tip portion 31 actually comes to the upper part of the target position targeted in the movement plan established in step S6.

Next, in step S10, the control unit 40 has an appropriate current position of the tip portion 31 with respect to the target based on the position of the target (position in the image captured by the second camera 34) grasped in step S9. Judge whether or not. If the position of the tip portion 31 (position of the arm) is appropriate (step S10: YES), the process proceeds to step S12 of FIG. If the position of the tip 31 is not appropriate (step S10: NO), the process proceeds to S11 to correct the position.
When the process proceeds to step S11, the control unit 40 determines whether or not the target is within the field of view of the second camera. In other words, the control unit 40 determines whether or not the target exists in the image acquired from the second camera 34 in step S8. If the target is within the field of view of the second camera (step S11: YES), the process proceeds to step S6 in order to correct the position by removing the arm. If the target is not within the field of view of the second camera (step S11: NO), the process returns to step S1 in order to start over from capturing the target by the first camera.

Next, the process shown in FIG. 5 will be described.
When the process proceeds to step S12, the control unit 40 controls the laser irradiation unit 32 to irradiate the laser.
Next, in step S13, the second camera 34 takes a picture while the laser is being irradiated. The control unit 40 acquires the image from the second camera 34.
Next, in step S14, the control unit 40 performs a process of estimating the three-dimensional structure of the target based on the image acquired in step S13. Specifically, as described above, the control unit 40 performs a process of estimating the distance to the target object, the direction of the object, and the like.

Next, in step S15, the control unit 40 determines whether or not the three-dimensional structure of the target can be estimated by the process of step S14. If the three-dimensional structure can be estimated (step S15: YES), the process proceeds to step S17. If the three-dimensional structure cannot be estimated (step S15: NO), the process proceeds to step S16 in order to change the position of the tip portion 31 and perform image acquisition again.
Next, when the process proceeds to step S16, the control unit 40 controls the arm to be slightly lowered. In other words, the control unit 40 controls the movement of the joint so that the position of the tip portion 31 is slightly (predetermined amount) lower than before. As a result, the tip portion 31 comes closer to the target object, and it is expected that a better image (image taken by the second camera 34) can be acquired. After step S16, the process returns to step S12.

Next, in step S17, the control unit 40 calculates the height and angle for grasping the target object based on the result estimated in the process in step S14. Based on this calculation result, the control unit 40 obtains the displacement amount of each joint for moving the tip portion 31 to the position where the target object is grasped.
Next, in step S18, the control unit 40 controls the movement of each joint based on the result obtained in step S17, and moves the tip portion 31 to a desired position by moving the arm. Then, the control unit 40 controls the finger 33 to perform an operation of grasping the target object.

Next, in step S19, the control unit 40 acquires an image taken by the second camera 34 and analyzes the image. Specifically, the control unit 40 analyzes whether or not the finger 33 has successfully grasped the target object.
Next, in step S20, the control unit 40 determines whether or not the target has been successfully grasped based on the analysis result in step S19. If the target is successfully grabbed (step S20: YES), the process proceeds to step S22. If the target has not been successfully grabbed (step S20: NO), the process proceeds to step S21.
When the process proceeds to step S21, the control unit 40 moves the arm so as to raise the tip portion 31 (away from the target object), and then performs a series of steps for grasping the target again in step S12. Return to the processing of.

When the process proceeds to step S22, the control unit 40 controls to move the arm so as to raise the tip portion 31 while the finger 33 grabs the target object.

The process from step S23 in FIG. 6 is a control for carrying the object grasped by the robot 1 to the place position and placing it.
In step S23, the control unit 40 acquires an image taken by the first camera 14.
Next, in step S24, the control unit 40 performs coordinate conversion processing (around view processing) and composition processing based on the image acquired in step S23 to generate a plan view image.
Next, in step S25, the control unit 40 recognizes the place position for placing the object being grasped on the plan view image generated in step S24. Specifically, the control unit 40 automatically recognizes the place position by using the feature of the place position acquired in advance. Alternatively, the control unit 40 grasps the information of the place position designated by the user by the GUI. Alternatively, the control unit 40 may be given the coordinate information of the place position with respect to the pedestal 11 in advance.

Next, in step S26, the control unit 40 creates a plan for moving the tip portion 31 from the current position to the upper part of the place position. The movement here is to move the tip portion 31 in a direction parallel to the xy plane. Specifically, the control unit 40 obtains an amount of displacement of each joint.
Next, in step S27, the control unit 40 controls to move the tip portion 31 based on the plan created in step S26.

Next, in step S28, the control unit 40 acquires an image taken by the second camera 34.
Next, in step S29, the control unit 40 performs a process of recognizing the position of the place based on the image acquired in step S28. Here, too, the control unit 40 automatically recognizes the place position by using the feature of the place position acquired in advance.
Next, in step S30, the control unit 40 determines whether or not the positions of the arm and the tip portion 31 are appropriate for the position of the place recognized in step S29. If the positions of the arm and the tip portion 31 are appropriate (step S30: YES), the process proceeds to step S32 of FIG. If the positions of the arm and the tip portion 31 are not appropriate (step S30: NO), the process proceeds to step S31.
When the process proceeds to step S31, the control unit 40 determines whether or not the position of the place is within the field of view of the second camera as a result of the recognition process in step S29. In other words, the control unit 40 determines whether or not the place is included in the image acquired from the second camera 34 in step S28.
When the position of the place is within the field of view of the second camera (step S31: YES), the process proceeds to step S26 in order to move the tip portion 31 again.
If the position of the place is not within the field of view of the second camera (step S31: NO), the process proceeds to step S23 in order to start over from the process of retaking a wide range of images with the first camera 14.

Then, when the process proceeds to step S32 in FIG. 7, the control unit 40 lowers the arm so that the tip portion 31 approaches the position of the place.
Then, in step S33, the control unit 40 stores the target object (target) in the place by controlling the finger 33 to release the target object.
In step S33, the entire control of a series of movements (flow charts from FIG. 4 to FIG. 7) until the robot 1 grabs the target and stores it in the place is completed.

As described above, in the procedure shown in this flowchart, the first camera and the second camera photograph the object in different phases, and the control unit captures the object by the first camera in each of the phases. It is controlled to change the direction of the arm based on one image, and is controlled to change the direction of the arm based on the second image taken by the second camera.

As described above, according to the present embodiment, the plurality of first cameras 14 capture a wide area image. The control unit 40 transforms the coordinates of the image taken by the first camera into a partial image of the plan view (around view processing). Further, the control unit 40 combines the plurality of partial images to generate one plan view image. By synthesizing the plan view images from a plurality of images, it is possible to acquire a plan view image without a blind spot by the robot 1 itself (pedestal 11 or the like). That is, the control unit 40 can grasp the situation around the robot 1. Further, the control unit 40 recognizes the pedestal 11 in the plan view image. As a result, the control unit 40 can grasp the surrounding situation, the position and orientation of the pedestal of the robot 1 itself, the orientation of each arm, and the position and orientation of the tip portion 31 in association with each other. That is, the control unit 40 can accurately grasp the position information of the robot 1 and its surroundings. That is, for example, even when the user carries or changes the orientation of the robot 1, the control unit 40 of the robot 1 automatically positions itself without relying on a camera or the like fixedly installed in the surrounding area (work environment). Can be grasped. That is, even if the position of the robot 1 is changed, it is not necessary to perform the readjustment operation for that purpose.

Further, according to the present embodiment, the control unit 40 recognizes the operation target object, the place, and the like by using the image taken by the second camera fixed to the tip portion 31. That is, it is not necessary to perform an operation for setting the positional relationship between the robot and the camera.

Further, according to the present embodiment, the robot 1 uses a first camera that captures a relatively wide area and a second camera that is fixedly installed at the tip end portion 31 of the arm. As a result, the robot 1 can finely recognize the object to be operated at the tip while looking over the entire periphery and grasping it. That is, the robot 1 can perform accurate work. In addition, the resolution and accuracy of the first camera 14 may be such that the position where the object exists can be recognized from a long distance. Further, the resolution and accuracy of the second camera 34 may be such that the shape of the object can be recognized from a short distance. By properly using these two types of cameras, the robot 1 can perform accurate work without necessarily using a camera having particularly high accuracy and resolution. The degree of freedom in selecting the camera module is expanded.
It is a contradictory requirement to get close to the object to be operated and grasp the detailed situation and to have a wide view of the whole, but the robot 1 of the present embodiment has both of them without installing a fixed camera. It becomes possible to meet the demand.
Further, even if the target object goes out of the field of view of the camera and the robot 1 loses sight of the target object, by moving the arm, the camera moves together with the arm and can catch the target object again. Even if a blind spot occurs, it may be eliminated by the movement of the arm, so there is a degree of freedom in the arrangement and angle when fixing the camera.
As described above, according to the present embodiment, it is possible to realize the automation of the robot operation without impairing the portability and maneuverability of the relatively small robot.

[Second Embodiment]
Next, the second embodiment will be described. The feature of the second embodiment is that a plurality of laser irradiation portions are provided at the tip portion 31, and at least one of the plurality of laser irradiation portions irradiates the laser in a direction different from that of the other laser irradiation portions. It is to be.

FIG. 8 is a schematic view (side view) showing a detailed configuration of the tip portion 31b, which is a characteristic portion according to the present embodiment. Further, the figure shows the relationship between the arrangement of the plurality of laser irradiation units in the present embodiment and the surface irradiated with the laser. As shown in the figure, the tip portion 31b includes a plurality of laser irradiation portions. Specifically, the tip portion 31b has laser irradiation portions 32-1 and 32-2. Further, the tip portion 31b includes a second camera 34. The finger 33 provided on the tip portion 31b is omitted in this figure.

As shown in the figure, the direction of irradiation by the laser irradiation unit 32-1 (indicated by the broken line A) and the direction of irradiation by the laser irradiation unit 32-2 (indicated by the broken line B) are different. That is, at least one of the plurality of laser irradiation units irradiates the laser in a direction different from that of the other laser irradiation units. These two laser irradiation units 32-1 and 32-2 are projected onto the installation surface 5. At this time, the distance on the installation surface between the points irradiated by the two lasers differs depending on the distance between the tip portion 31b and the installation surface 5.

9 and 10 are schematic views showing an example of a pattern in which the laser irradiation unit shown in FIG. 8 projects onto the installation surface 5, respectively. FIG. 9 shows a pattern projected on the installation surface 5 when the laser irradiation unit and the installation surface are separated by a certain distance. FIG. 10 shows a pattern projected on the installation surface 5 when the laser irradiation unit is closer to the installation surface than in FIG. 9.
In FIG. 9, the upper cross shape (diagonal cross shape) is a pattern irradiated by the laser irradiation unit 32-2. The lower cross shape (cross shape composed of vertical and horizontal lines) is a pattern irradiated by the laser irradiation unit 32-1.
In FIG. 10, the cross shape shown by the broken line indicates the position where the upper cross shape is projected in FIG. 9. Of the two solid crosses, the upper cross (diagonal cross) is the pattern irradiated by the laser irradiation unit 32-2. The lower cross shape (cross shape composed of vertical and horizontal lines) is a pattern irradiated by the laser irradiation unit 32-1.
As described above, the distance between the projected patterns is different between the case of FIG. 9 and the case of FIG. In other words, the distance from the position of the laser irradiation unit to the position of the projection surface can be estimated by measuring the distance between the two in the projected pattern.

FIG. 11 is a schematic view (side view) showing a detailed configuration of the tip portion 31b. Further, the figure shows the relationship between the arrangement of the plurality of laser irradiation units in the present embodiment and the surface irradiated with the laser. Next, a method of measuring the distance between the tip portion 31b and the installation surface 5 will be described with reference to this figure.

As shown in the figure, the tip portion 31b is provided with two laser irradiation portions 32-1 and 32-2. The distance between the light source of the laser irradiation unit 32-1 and the laser irradiation unit 32-2 is x. The laser beam (ray A) is emitted directly below (perpendicular to the installation surface 5) from the laser irradiation unit 32-1. Further, the laser beam (ray B) is irradiated from the laser irradiation unit 32-2 in an oblique direction, that is, in a direction of an angle θ downward from the horizontal plane (for example, the bottom surface of the tip portion 31b). .. As long as the positions of the laser irradiation units 32-1 and 32-2 and the laser irradiation direction do not change, the values of x and θ do not change.

The second camera 34 attached to the tip portion 31b photographs the downward direction, that is, the direction of the installation surface 5. The control unit 40 acquires an image taken by the second camera 34, and obtains the distance from the bottom surface of the tip portion 31b (the tip portion of the robot) to the installation surface 5 as follows. That is, the control unit 40 performs scaling processing based on the captured image, and the point of the irradiation position of the light ray A on the installation surface 5 (irradiation from the laser irradiation unit 32-1) and the installation surface 5 of the light ray B. The distance from the point of the irradiation position to (irradiation from the laser irradiation unit 32-2) is obtained. Let the distance be d. Further, since the distance between the two light sources is x as already described, the control unit 40 calculates the distance z from the bottom surface of the tip portion 31b to the installation surface 5 by the following formula. Here, tan () is a tangent function.
z = (x−d) · tan (θ)

As described above, the laser irradiation unit 32-1 attached to the tip portion 31b so as to irradiate directly below (vertically), and the laser 32-2 attached so as to irradiate diagonally (at an angle θ). However, each of the installation surfaces 5 is irradiated with a laser beam. Then, based on the image taken by the second camera 34, the control unit 40 recognizes the two laser beams irradiated on the installation surface 5. The control unit 40 measures the distance (height) from the bottom surface of the tip portion 31b to the installation surface 5 based on the distance between these two points. As a result, the control unit 40 can determine the amount of movement in the height direction when grasping the object by using the finger 33.

The height of the tip portion 31b may be measured by the control unit 40 according to the following modification. For example, when the laser irradiation unit 32-1 is attached to the tip portion 31b, the irradiation direction does not have to be directly below (perpendicular to the bottom surface of the tip portion 31b). Even if the irradiation direction of the laser irradiation unit 32-1 is oblique (however, irradiation at a known angle), the control unit 40 can appropriately calculate the height. Further, for example, in order to recognize the laser beam emitted to the installation surface 5, an image taken by the first camera 14L or 14R may be used instead of the image taken by the second camera 34.

Further, the installation surface 5 may be irradiated with the laser light emitted by the laser irradiation units 32-1 and 32-2 after crossing.
FIG. 12 is a schematic view (side view) of the tip portion 31b showing an example in which the installation surface is irradiated after the two laser beams intersect. As shown in the figure, the laser irradiation unit 32-1 irradiates the laser beam (ray A) directly below (perpendicular to the installation surface 5). Further, the laser irradiation unit 32-2 irradiates the laser beam (light ray B) diagonally downward at an angle θ from the tip portion 31b. After the rays A and B intersect, the installation surface 5 is irradiated. The distance between the irradiation points on the installation surface 5 is d. The distance between the light sources of the laser irradiation units 32-1 and 32-2 is x. In this case, the control unit 40 calculates the distance z from the bottom surface of the tip portion 31b to the installation surface 5 by the following formula on the assumption that the installation surface 5 is irradiated after the light rays A and B intersect. do.
z = (x + d) · tan (θ)

As described above, according to the present embodiment, the control unit 40 can automatically calculate the position for grasping the object. Therefore, even if the installation location of the robot is changed, the operation for setting the position for grasping the object can be performed. No need to do. That is, it is possible to realize automation of the robot operation without impairing the portability of the small robot.

[Third Embodiment]
Next, the third embodiment will be described. The feature of the third embodiment is that the robot is provided with an autonomous moving unit. As a result, the robot can be moved not only fixedly on the installation surface but also over a wide range.

FIG. 13 is a schematic view showing an outline of the moving operation of the robot 2 according to the present embodiment. In the illustrated example, the robot 2a represents the position before the movement, and the robot 2b represents the position after the movement. As shown in the figure, a rail 201 is laid at a place where the robot 2 operates. Further, an object 202 to be operated by the robot 2 is placed near the rail 201. The robot 2 can move in the front-rear direction along the rail 201 by driving the wheels (not shown) by, for example, a motor. Alternatively, the robot 2 may be able to freely move in any direction not only on the rail 201 but also on a plane by wheels (for example, four wheels). By controlling the control unit 40 of the robot 2 to drive the wheels, the robot 2 can approach the object 202. Further, the robot 2 can move to a desired position depending on the situation.

The wheels and the mechanism for driving those wheels are referred to as "autonomous moving units". That is, the autonomous movement unit is a function for the robot to autonomously move on the installation surface. The autonomous movement unit is controlled by the control unit 40. Due to the function of the automatic moving unit, the robot 2 autonomously performs a desired target even when there is an object or place in a position where the arm cannot reach, or when the object or place cannot be visually recognized from the first camera 14. You can move to the vicinity of an object or place.

14, FIG. 15, FIG. 16, and FIG. 17 are flowcharts showing the processing procedure of the robot 2 in the present implementation. This flowchart shows a processing procedure for controlling an operation in which the robot 2 grabs a target object and carries it to a place position. 14, 15, 16, 16 and 17 are one flowchart connected by a coupler. Hereinafter, the processing procedure will be described with reference to this flowchart.

The processes from steps S101 to S105 in FIG. 14 are the same as the corresponding processes from steps S1 to S5 in FIG.
In step S106, the control unit 40 determines whether or not the arm can reach the target. Even if the target is not found as a result of performing the target recognition process (step S105), the control unit 40 determines that the arm is not within the reach of the target. If it is determined that the arm is within reach of the target (step S105: YES), the process proceeds to step S108. If it is determined that the arm is not within reach of the target (step S105: NO), the process proceeds to step S107 in order to move the robot and re-recognize the target.

When the process proceeds to step S107, the control unit 40 controls the autonomous movement unit and moves the robot 2 itself. Specifically, by moving the wheels of the robot 2, the robot 2 is moved to a predetermined position. After the end of this step, the process returns to step S101 to continue the process.

When the process proceeds to step S108, in steps S108 to S111, the control unit 40 controls to move the arm and recognizes the position of the target based on the image acquired from the second camera 34. This process is the same as the process from steps S6 to S9 in FIG.
Next, in step S112, the control unit 40 makes the same determination as in step S10 of FIG. 4, and branches according to the determination result. That is, when the position of the tip portion 31 (position of the arm) is appropriate (step S112: YES), the process proceeds to step S114 of FIG. If the position of the tip portion 31 is not appropriate (step S112: NO), the process proceeds to S113 to correct the position.
When the process proceeds to step S113, the control unit 40 makes the same determination as in step S11 of FIG. 4, and branches according to the determination result. That is, when the target is within the field of view of the second camera (step S113: YES), the process proceeds to step S108 in order to correct the position by removing the arm. If the target is not within the field of view of the second camera (step S113: NO), the process returns to step S101 in order to start over from capturing the target by the first camera.

The processes of steps S114 to S124 of FIG. 15 are the same as the processes of steps S12 to S22 of FIG. 5, respectively. When the process of step S124 of FIG. 15 is completed, the process proceeds to step S125.

The processes from steps S125 to S127 in FIG. 16 are the same as the processes from steps S23 to S25 in FIG. 6, respectively.
Next, in step S128, the control unit 40 determines whether or not the arm can reach the place. Even if the place is not found as a result of performing the place recognition process (step S127), the control unit 40 determines that the place is not within the reach of the arm. If it is determined that the arm is within reach of the place (step S128: YES), the process proceeds to step S130. If it is determined that the arm is not within reach of the place (step S128: NO), the process proceeds to step S129 in order to move the robot and re-recognize the target.

When the process proceeds to step S129, the control unit 40 controls the autonomous movement unit and moves the robot 2 itself. Specifically, by moving the wheels of the robot 2, the robot 2 is moved to a predetermined position. After the end of this step, the process returns to step S125 to continue the process.

When the process proceeds to step S130, the robot 2 performs the processes from steps S130 to S135. The processes from steps S130 to S135 are the same as the processes from steps S26 to S31 in FIG.
When the control unit 40 determines in step S134 that the arm position is appropriate for the place (step S134: YES), the process proceeds to step S136 of FIG. If the control unit 40 determines in step S134 that the arm position is not appropriate for the place (step S134: NO), the process proceeds to step S135. By making the same determination as in step S31 of FIG. 6 in step S135, the process returns to the process on the appropriate side of either step S125 or S130.

When the process proceeds to step S136 of FIG. 17, the robot 2 performs the processes of steps S136 and S137. The processes of steps S136 and S137 are the same as the processes of steps S32 and S33 of FIG. 7, respectively.
When the process of step S137 is completed, the process procedure of the entire flowchart is completed.

As described above, when the robot 2 moves autonomously, the robot 2 reacquires the surrounding image using the first camera at the moving destination, performs coordinate conversion and the like to generate a synthesized image. ..
At this time, the control unit 40 can recognize the coordinate relationship between the tip of the arm and the installation surface at the moving destination based on the position of the pedestal 11 in the plane image (combined image).

As described above, according to the present embodiment, the first camera and the second camera are always installed at positions where the surroundings of the robot can be observed even after moving to an arbitrary place, and the target or the place can be used. The position can be recognized. It is possible to automate the movement of the mobile robot without impairing the autonomous mobility of the mobile robot.

[Fourth Embodiment]
Next, the fourth embodiment will be described. The feature of the fourth embodiment is to operate the robot while using the first camera and the second camera at the same time. That is, in the present embodiment, under the control of the control unit 40, the first cameras 14L and 14R and the second camera 34 simultaneously photograph the object. Further, the control unit 40 controls to change the position and orientation of the arm based on the images simultaneously captured by the first cameras 14L and 14R and the second camera 34, respectively.

18, FIG. 19, and FIG. 20 are flowcharts showing the processing procedure of the robot 3 in the present implementation. This flowchart shows a processing procedure for controlling an operation in which the robot 3 grabs a target object and carries it to a place position. 18, 19, and 20 are one flowchart connected by a combiner. Hereinafter, the processing procedure will be described with reference to this flowchart.

The processes from steps S201 to S207 in FIG. 18 are the same as the processes from steps S1 to S7 in FIG. 4, respectively.
Next, in step S208, the control unit 40 controls both the first cameras 14L and 14R and the second camera 34 to take pictures, respectively. The control unit 40 acquires images from the first cameras 14L and 14R and the second camera 34. That is, the control unit 40 uses the first cameras 14L and 14R and the second camera 34 at the same time.

Next, in step S209, the control unit 40 recognizes the position of the target using the images acquired from the first cameras 14L and 14R and the second camera 34. By using both the first cameras 14L and 14R and the second camera 34, for example, even if there is no target in the field of view of the second camera 34, the composite has been performed based on the images taken by the first cameras 14L and 14R. Based on the image, the control unit 40 can recognize in which direction the target is located.

Next, in step S210, the control unit 40 determines whether or not the position of the tip portion 31 of the arm is appropriate for the target (for example, the tip portion 31 is above the target and can approach the target). do. If the arm position is appropriate for the target (step S210: YES), the process proceeds to step S211 in FIG. If the arm position is not appropriate (step S210: NO), the process returns to step S206 in order to move the arm again.

The processes from step S211 to step S221 in FIG. 19 are the same as the processes from step S12 to step S22 in FIG. 5, respectively.

Next, the processes from step S222 to step S226 in FIG. 20 are the same as the processes from step S23 to step S27 in FIG. 6, respectively.
Next, in step S227, the control unit 40 controls both the first cameras 14L and 14R and the second camera 34 to take pictures, respectively. The control unit 40 acquires images from the first cameras 14L and 14R and the second camera 34.
Next, in step S228, the control unit 40 recognizes the position of the place using the images acquired from the first cameras 14L and 14R and the second camera 34. By using both the first cameras 14L and 14R and the second camera 34, for example, even if there is no place in the field of view of the second camera 34, it has been synthesized based on the images taken by the first cameras 14L and 14R. Based on the image, the control unit 40 can recognize in which direction the place is located.

Next, in step S229, the control unit 40 determines whether or not the position of the tip portion 31 of the arm is appropriate for the place. If the arm position is appropriate for the place (step S229: YES), the process proceeds to step S230. If the arm position is not appropriate (step S229: NO), the process returns to step S225 in order to move the arm again.
The processing of the next steps S230 and S231 is the same as the processing of steps S32 and S33 of FIG. 7, respectively.
When the process of step S231 is completed, the robot 3 ends all the processes of this flowchart.

As described above, in the present embodiment, the control unit 40 recognizes the position of the target and the position of the place by using the first camera and the second camera at the same time.
Thereby, for example, even if there is no target or place in the field of view of the second camera 34, the first cameras 14L and 14R can recognize in which direction the target or place is located. Therefore, the control unit 40 can control the center unit of the arm to move to the position of the target or the place more quickly.
That is, according to this embodiment, it is possible to realize automation of robot operation without impairing the portability and maneuverability of a relatively small robot.

[Fifth Embodiment]
Next, the fifth embodiment will be described. The feature of the robot 4 according to the fifth embodiment is that the installation surface irradiated by the laser irradiation unit is photographed, and the step difference of the installation surface is grasped and measured based on the obtained image.

FIG. 21 is a perspective view showing the operating environment of the robot when the installation surface has a step. As shown in the figure, the installation surface 5 has a step. The upper step of the step is 5U, and the lower step is 5L. In the illustrated example, the laser irradiation unit 32 provided at the tip end portion 31 of the robot 4 irradiates the laser beam so as to form a cross-shaped pattern. The first cameras 14L and 14R photograph the installation surface having a step, which is irradiated with a laser beam. The control unit 40 can recognize that there is a step on the installation surface based on the shape of the laser beam irradiation included in the captured image.

FIG. 22 shows a composited image obtained by performing coordinate transformation and compositing processing on the installation surface having a step based on the images taken by the first cameras 14L and 14R. In the figure, the upper stage of the installation surface is 5U, and the lower stage is 5L. Further, 5G shown by a broken line represents a step.

FIG. 23 is a schematic view showing an example of the irradiation method by the laser irradiation unit 32. The figure is a composited image obtained by synthesizing images taken by the first cameras 14L and 14R at a plurality of timings. In the illustrated example, the laser irradiation unit 32 irradiates a predetermined area of the installation surface 5 with a total of 12 points (P11 to P14, P21 to P24, P31 to P34) in 3 rows and 4 columns. In the example shown in the figure, there are no steps or irregularities in the area within the installation surface 5. The robot 4 takes pictures with the first cameras 14L and 14R while sequentially changing the position of the laser irradiation unit 32 provided on the tip portion 31 by moving the arm. That is, first, when the laser irradiation unit 32 is irradiating the position of P11, the first cameras 14L and 14R take a picture, and then, when the laser irradiation unit 32 is irradiating the position of P12, the first camera is taken. The cameras 14L and 14R take pictures, and so on. That is, while the laser irradiation unit 32 sequentially irradiates 12 points from P11 to P34, the first cameras 14L and 14R take pictures at the timing of irradiating each point. The control unit 40 sequentially performs control for moving the arm and control for executing shooting by the first cameras 14L and 14R. Further, the control unit 40 generates a total of 12 combined images corresponding to each irradiation timing by performing coordinate conversion and left-right composition of the images acquired from the first cameras 14L and 14R. Further, the control unit 40 performs a compositing process of superimposing all 12 images corresponding to the irradiation timing of each point. As a result, the control unit 40 generates one composite image including the 12 points (irradiated points) shown in FIG. 23.

FIG. 24 is a schematic view showing a synthesized image generated by the same procedure as described with reference to FIG. 23 when the installation surface 5 has a step. The image shown in FIG. 24 is an image taken sequentially from the same 12 types of positions as in the case of FIG. 23 by the laser irradiation device 32 in which the control unit 40 controls to move the arm in the same procedure as described in FIG. It is a composite. However, while the image of FIG. 23 is a photograph of the laser beam radiated to the installation surface 5 having no step, the image of FIG. 24 shows the installation surface 5 having the step shown in FIG. It was obtained by shooting. As shown, the points P11 to P14 appear at the same positions in the image of FIG. 23 and the image of FIG. 24. On the other hand, since there is a step at the position of 5G, the positions of the points from P21 to P24 and from P31 to P34 are shifted between the image of FIG. 23 and the image of FIG. 24.

25, 26, and 27 are schematic views for explaining the principle that the positions of the above points are displaced.
FIG. 25 is a side view showing the relationship between the robot 4, the installation surface 5, the point of irradiation by the robot 4, and the camera (first camera) that photographs the area including the point. Further, FIG. 26 is a plan view corresponding to FIG. 25.

As shown in the figure, the laser irradiation unit 32 included in the robot 4 irradiates the point P21 on the installation surface 5U. However, in this example, due to the step, the laser light emitted from the laser irradiation unit 32 irradiates a point on the surface 5L located at a position lower than the surface 5U. The first camera 14 photographs the surface having the step. The laser irradiation unit 32 irradiates the P21, but the image taken by the first camera 14 is that the irradiated point exists at the position of P21', which is a point on the installation surface 5. The image looks as if it were there. With reference to FIG. 26 (plan view), an image is generated in which the laser beam irradiating the direction of P21, which is a point on the surface 5L, appears to irradiate the position of P21', which is another point. Will be done.

FIG. 27 is a side view similar to that of FIG. 25, and is a side view from which information on the distance between typical positions appearing in FIG. 25 is extracted. Let z be the height of the light source of the laser irradiation unit 32 when the installation surface 5 is used as a reference. Let z'be the height of the light source of the laser irradiation unit 32 when the surface 6 (below the installation surface 5) is used as a reference. Let r ′ be the horizontal distance from the point P21 to the principal point of the lens of the first camera 14. Further, let r be the horizontal distance from the point P21'to the principal point of the lens of the first camera 14. d, which is the difference between r and r', is the amount of deviation of the laser irradiation position due to the presence of a step. That is, d = r'-r.
Using these quantities, the control unit 40 can calculate the height Δz between the installation surface 5 and the surface 6 based on the following equation.
Δz = (z'-z) = d * z'/ r = d * z / (rd)

Although P21 has been described with reference to FIGS. 25 and 26, the control unit 40 can grasp the state of steps and the state of unevenness on the surface by performing the same calculation for all the irradiation points. The control unit 40 can determine the height at which the target is grabbed and released based on the calculated height information.

The irradiation pattern by the laser irradiation unit 32 may have another shape. For example, the irradiation pattern may be cross-shaped (due to two intersecting line segments).
Further, the laser irradiation unit 32 may have a plurality of light emitting units (light sources). In this case, for example, a plurality of points on the installation surface can be irradiated at the same time. For example, the laser irradiation unit 32 can simultaneously irradiate some of the 12 points shown in FIG. 23. Further, the laser irradiation unit 32 may irradiate a plurality of points in a time-division manner. Further, the number of irradiation points does not have to be twelve. Further, the laser irradiation unit 32 does not necessarily have to irradiate the installation surface perpendicularly. The laser irradiation unit 32 may irradiate the installation surface in an oblique direction.

Instead of synthesizing (superimposing) the images taken at a plurality of timings to generate the image shown in FIG. 24, the images taken at each timing are based on the position of the irradiation light in the synthesized images obtained at each timing. The step or unevenness of the installation surface may be calculated.

That is, in the present embodiment, the control unit 40 controls the laser irradiation unit 32 while moving the tip portion 31 so that the laser irradiation unit 32 sequentially irradiates light in the directions of a plurality of predetermined positions on the installation surface 5. Further, the control unit 40 generates a plurality of synthesized images (plan view images) corresponding to each of the timings at which the laser irradiation unit 32 irradiates light in the directions of the plurality of predetermined positions. Further, the control unit 40 calculates steps and irregularities of the installation surface 5 based on the positions of light of a predetermined pattern included in the plurality of combined images (plan view images). In this process, a process of superimposing the composited images at the plurality of timings may be performed to generate one composited image (plan view image).

As described above, according to the present embodiment, the robot 4 can automatically grasp the steps and undulations on the installation surface. Therefore, even if the installation location of the robot 4 is changed, it is not necessary to perform an operation of setting a position such as a height for grasping the target. That is, it is possible to realize automation of the robot operation without impairing the portability of the small robot.

It should be noted that at least a part of the functions of the robot control unit 40 in each of the above-described embodiments can be realized by a computer. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, or a USB memory, or a storage device such as a hard disk built in a computer system. Say that. Furthermore, a "computer-readable recording medium" is a device that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, it may include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

The present invention can be used for all kinds of robots, for example, for industrial use or entertainment use. However, the scope of use of the present invention is not limited to those exemplified here.

1,2,2a, 2b, 3,4 Robot 5,5L, 5U Installation surface 11 Pedestal 12 Struts 14, 14L, 14R 1st camera 15 1st arm 16 2nd arm 21 1st joint 22 2nd joint 23 3rd Joints 31, 31b Tip (hand)
32, 32-1, 32-2 Laser irradiation unit 33 Finger 34 Second camera 40 Control unit 101 Synthesized image (plan view image)
101L, 101R coordinate converted image (plan view partial image)
201 Rail 202 Object

Claims (8)

A pedestal with a point in contact with the installation surface,
A strut that is mounted vertically on the pedestal and can rotate around the vertical axis,
An arm that is connected to the strut and can be turned by a single or multiple joints.
A tip portion connected to the tip of the arm and capable of changing the direction with the arm,
A plurality of first cameras provided at a position fixed to the support column and capable of photographing the pedestal itself and the installation surface around the pedestal, and a plurality of first cameras.
The pedestal is obtained by acquiring a first image from the plurality of first cameras, generating a partial plan view image by coordinate-transforming the first image, and aligning and synthesizing the plurality of the partial plan views. A control unit that generates a plan view image including the image of the above and controls to change the direction of the arm in order to move the tip portion to a predetermined position based on the plan view image.
A robot equipped with. An irradiation unit that is fixed to the tip and irradiates light having a predetermined pattern in the direction of the installation surface.
The robot according to claim 1. A second camera provided at the tip,
With more
The control unit controls to change the direction of the arm based on the second image acquired from the second camera.
The robot according to claim 1 or 2. An autonomous moving unit for autonomously moving on the installation surface,
The robot according to claim 1, further comprising. The first camera and the second camera simultaneously photograph an object, and the object is photographed.
The control unit controls to change the direction of the arm based on the images taken by the first camera and the second camera at the same time.
The robot according to claim 3. The first camera and the second camera photograph an object in different phases.
The control unit controls each phase to change the direction of the arm based on the first image captured by the first camera, and the arm is based on the second image captured by the second camera. Control to change the direction of
The robot according to claim 3. With a plurality of the above-mentioned irradiation units,
At least a part of the irradiation units of the plurality of irradiation units irradiates the light in a direction different from that of the other irradiation units.
The robot according to claim 2. The control unit controls the irradiation unit while moving the tip portion so that the irradiation unit sequentially irradiates the light in the direction of a plurality of predetermined positions on the installation surface.
The control unit generates a plurality of the plan view images corresponding to each of the timings at which the irradiation unit irradiates the light in the direction of the plurality of predetermined positions.
The control unit calculates the step or unevenness of the installation surface based on the position of the light of the predetermined pattern included in the plurality of plan view images.
The robot according to claim 2.

Images