JP2003157435A - Image processing device, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To segment an image with the object shown in balance from the original regardless the object photographed large or small. SOLUTION: The invention extracts a candidate area of an object from the original image provided (S20), and sets a segmenting area for segmenting the image of the object from the original image based on the candidate area (S40). The reference for the position setting of the segmenting area is switched based on the occupying size of the candidate area in the original image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique for cutting out an image including an object from an original image.

[0002]

2. Description of the Related Art FIG. 16 shows a conventional processing procedure for clipping an object image. In FIG. 16, in step S51, the original image is read in CIF format, and in step S5
In 2, the target area is extracted using the background difference. Then, in step S53, the presence / absence of the target area is determined. If the target area exists in the frame, the center of the circumscribed rectangle of the target area is determined in step S54. Next, in step S55, the cutout area having the number of pixels corresponding to QCIF is set so that the circumscribed rectangles of the target area obtained in step S54 and the centers thereof coincide with each other. Then, in step S56, the image in the cutout area on the original image is cut out,
At 57, the cutout image is displayed. Further, the position information of the set cutout area is the cutout area information DB1.
Stored in 10.

On the other hand, when it is determined in step S53 that the target area does not exist in the frame, in step S58, the position information of the cutout area of the previous frame is read from the cutout area information DB 110 and the image is cut out.

[0004]

In the conventional method of cutting out an object image, the position of the cutout area is set with reference to the center of the circumscribed rectangle of the candidate area of the object. In the case of this method, when the candidate area CA1 of the object is sufficiently smaller than the cut-out area Z1 as shown in FIG. 17A, the object is captured in the center as shown in FIG. 17B. A cutout image can be obtained.

However, as shown in FIG. 18A, when the candidate area CA2 of the object is so large as to extend beyond the cut-out area Z2, the conventional method shown in FIG.
As shown in FIG. 8 (b), the image of the person's head is cut off, making it very difficult to see.

Here, if the cutout area is set to be shifted slightly above the position where the circumscribing rectangle and the centers coincide with each other, in the example of FIG. 18, the head of the person should not be cut. Is possible. However, in this case, in the case of FIG. 17, a wasteful space is created in the information of the target object, and it becomes difficult to grasp the environment around the target object.

In view of the above problems, the present invention provides an image for displaying an object in a well-balanced manner in image processing for cutting out an image including an object from an original image, whether the object is small or large. The task is to enable cutting out.

[0008]

In order to solve the above-mentioned problems, the present invention provides, as an image processing apparatus, means for extracting a candidate area of an object from a given original image, and based on the candidate area. And a means for setting a cutout area for cutting out the image of the object from the original image, wherein the cutout area setting means changes the reference for setting the position of the cutout area according to the situation. It is configured to be possible.

According to the present invention, the reference for setting the position of the cut-out area for cutting out the image of the object from the original image can be switched according to the situation, so that, for example, the object can be changed depending on the situation. Even if an object appears small or large, it is possible to cut out an image that displays the object in a well-balanced manner.

It is preferable that the cutout area setting means in the image processing apparatus according to the present invention switches the reference according to the characteristics of the candidate area.

It is preferable that the cutout area setting means uses the size occupied by the candidate area in the original image as a characteristic of the candidate area.

Further, the cut-out area setting means sets the cut-out area so that the candidate area occupies substantially the center of the cut-out area when the candidate area is smaller than the cut-out area. When the area is larger than the cutout area, it is preferable to set the cutout area so that a specific portion of the candidate area fits within the cutout area. Furthermore, it is preferable that the object is a human or an animal, and the specific portion is a face or a head of the object. Alternatively, it is preferable that the specific portion is an upper end portion of the candidate area.

Further, it is preferable that the cutout area setting means uses a position of the candidate area in the original image as a characteristic of the candidate area.

Further, it is preferable that the cutout area setting means uses the moving direction of the candidate area as a characteristic of the candidate area.

Further, it is preferable that the cut-out area setting means in the image processing apparatus according to the present invention switches the reference according to the distance from the camera that photographed the original image to the object.

Further, the cutout area setting means in the image processing apparatus according to the present invention sets the cutout area based on the candidate area having the largest area among the plurality of candidate areas when a plurality of candidate areas are detected. Is preferred.

Further, the image processing apparatus according to the present invention is
It is preferable to include means for performing a smoothing process on the cutout region in the time direction of the original image.

Further, according to the present invention, as an image processing apparatus, means for extracting a candidate area of an object from a given original image, and the image of the object is cut out from the original image based on the candidate area. And a means for performing a smoothing process on the cutout area in the time direction of the original image.

According to the present invention, since the cutout area is smoothed in the time direction, the cutout area is moved smoothly even if the detected candidate area is slightly different for each frame, and the cutout image is easy to see. Is obtained.

The image processing apparatus according to the present invention preferably comprises means for delaying the original image so as to match the delay in the movement of the cutout area caused by the smoothing processing.

Further, it is preferable that the smoothing means in the image processing apparatus according to the present invention changes the time constant in the smoothing processing when the cutout area is enlarged and when it is reduced.

Further, it is preferable that the smoothing means in the image processing apparatus according to the present invention changes the time constant in the smoothing processing according to the moving direction of the cutout area.

Further, it is preferable that the cutout area setting means in the image processing apparatus according to the present invention is configured so that the setting conditions of the cutout area can be switched according to the moving direction of the candidate area. Further, it is preferable that the cutout area setting means sets the cutout area with a margin above the candidate area when the candidate area moves upward.

Further, according to the present invention, as an image processing apparatus, means for extracting a candidate area of an object from a given original image, and an image of the object are cut out from the original image based on the candidate area. Means for setting a cutout area for setting the cutout area based on the candidate area having the largest area among the plurality of candidate areas when a plurality of the candidate areas are detected. Is.

According to the present invention, even when a plurality of candidate areas are extracted, only the candidate area having the largest area is used for setting the cutout area, so that stable tracking of the object is possible.

The present invention also provides, as an image processing apparatus, means for extracting a candidate area of an object from a given original image, and cutting out the image of the object from the original image based on the candidate area. And a parameter set DB that stores a plurality of types of parameter sets related to the cutout area, and the cutout area setting unit includes a plurality of types of parameter sets stored in the parameter set DB. The cutout area is set using the parameter set selected from the inside.

The image processing apparatus according to the present invention preferably comprises means for selecting a parameter set according to a selection signal received from the outside of the image processing apparatus.

Further, the image processing apparatus according to the present invention is
It is preferable to include means for selecting a parameter set in accordance with at least one of the type of camera that captured the original image and the size of the candidate area.

The present invention also provides, as an image processing apparatus, means for extracting a candidate area of an object from a given original image, and when a plurality of the candidate areas are detected, a cutout image of each candidate area is extracted. Means for respectively generating and outputting images for displaying each clipped image as a list, means for receiving a selection signal for selecting one of the plurality of clipped images, and selectively clipping the clipped images selected by the selection signal. And means for outputting to.

The present invention also provides, as an image processing apparatus, means for extracting a candidate area of an object from a given original image, means for generating and outputting a cutout image related to the candidate area, and the cutout image. And a means for receiving an instruction signal for instructing the correction of the position of the cutout area, the cutout image generating means corrects the position of the cutout area according to the instruction signal, and generates and outputs a new cutout image. It is a thing.

Further, according to the present invention, as an image processing method, an image processing apparatus extracts candidate regions of an object from a given original image, and when a plurality of the candidate regions are detected, the candidate regions are related. Each cutout image is generated, an image for displaying each cutout image in a list is transmitted, the mobile terminal receives the list display image and displays it on the screen, and among the cutout images displayed in the list display image, The user is allowed to input a selection signal for selecting one of them, the input selection signal is transmitted to the image processing apparatus, the image processing apparatus receives the selection signal, and a cutout image is displayed according to the selection signal. It is selected and transmitted to the mobile terminal.

In the present invention, as an image processing method, an image processing apparatus extracts a candidate area of an object from a given original image, generates and transmits a cutout image of the candidate area, and transmits the portable terminal. Is received on the screen to receive the cutout image, the user to input an instruction signal for instructing the correction of the position of the cutout region in the cutout image, the input instruction signal is transmitted to the image processing device, The image processing device receives the instruction signal, corrects the position of the cutout area according to the instruction signal, generates a new cutout image, and transmits the new cutout image to the mobile terminal.

The present invention also provides, as an image processing method, a step of extracting a candidate area of an object from a given original image, and cutting out the image of the object from the original image based on the candidate area. And a step of setting a cutout area for setting the cutout area, and in the cutout area setting step, the reference for setting the position of the cutout area is switched according to the situation. Further, the present invention is a program for causing a computer to execute this image processing method.

The present invention also provides, as an image processing method, a step of extracting a candidate area of an object from a given original image, and cutting out the image of the object from the original image based on the candidate area. And a step of performing a smoothing process on the cutout area in the time direction of the original image. Further, the present invention is a program for causing a computer to execute this image processing method.

The present invention also provides, as an image processing method, a step of extracting a candidate area of an object from a given original image, and cutting out the image of the object from the original image based on the candidate area. For setting a cutout area for setting the cutout area based on the candidate area having the largest area among the plurality of candidate areas detected in the cutout area setting step. Is. Further, the present invention is a program for causing a computer to execute this image processing method.

[0036]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 shows the first to the first embodiments of the present invention.
9 is an example of a hardware configuration of an image processing apparatus common to the third embodiment. In FIG. 1, 10 is a wide-angle camera installed in one corner of the room, 20 is an image input interface, 30 is a data bus, 40 is a CPU, 50 is an image memory, 60 is a work memory, 70 is a program memory, and 80 is an image. The output interface 90 is a display device.

In the configuration shown in FIG. 1, the wide-angle camera 10 captures a field of view in the room. The video signal captured by the wide-angle camera 10 is stored in the image memory 50 from the image input interface 20 via the data bus 30.
The CPU 40 uses the work memory 60 to process the image data stored in the image memory 50 in accordance with the program stored in the program memory 70, and displays the result of this image processing via the image output interface 80 on the display device 90. To display.

FIG. 2 is a flow chart showing the flow of image processing according to the first embodiment. In FIG. 2, S10
Is a step of inputting an image captured by the wide-angle camera 10 as an original image, S20 is a step of extracting a candidate area of the object, S30 is a step of determining the presence or absence of the candidate area, S40 is a step of setting a cutout area, S5
0 is a step of cutting out the image of the target object from the original image using the cutout area set in step S40, and S60 is a step of displaying the cutout image. Further, in S70, when the candidate area is not detected, the cutout area information D
This is a step of reading the cutout area in the previous frame from B110. The cutout area information DB 110 is set in the work memory 60.

Steps S20 to S50 and S70 are
It is executed by the CPU 40 executing the program stored in the program memory 70. Here, step S20 corresponds to the candidate area detecting means in the present invention, and step S40 corresponds to the cutout area setting means in the present invention.

The operation of the image processing apparatus according to this embodiment will be described with reference to the flowchart of FIG.

In step S10, the image of the wide-angle camera 10 as the original image is in CIF format (356 pixels ×
288 lines) are read into the data bus 30 via the image input interface 20, stored in the image memory 50, and read into the CPU 40 for processing.

Next, in step S20, candidate regions of the object are detected from the given original image. There are various possible methods for detecting the candidate area, but here, the area is extracted by the background difference. Then, in step S30, the number of pixels of the candidate area is compared with a predetermined threshold value to determine whether or not an appropriate candidate area is detected. When the candidate area is detected, the process proceeds to cutout area setting step S40. If not detected, the process proceeds to step S70.

In the cutout area setting step S40, a cutout area for cutting out the image of the object from the original image is set. Here, the reference for setting the position of the cutout area can be switched according to the characteristic of the candidate area, and the size occupied by the candidate area in the original image is used as the characteristic of the candidate area.

First, in step S41, the size of the candidate area in the original image is compared with the cutout area. Here, the size of the cutout area is QCI.
It is assumed that the size of the F image is 178 pixels × 144 lines, and the vertical pixel numbers of the circumscribed rectangle of the candidate region and the cutout region are compared. When the vertical pixel number of the circumscribed rectangle is smaller than the vertical pixel number of the cutout area (No in S41), the center of the circumscribed rectangle of the candidate area is obtained (S42), and the center of the cutout area is the circumscribed rectangle of the candidate area. The cutout area is set so as to coincide with the center (S43). On the other hand, when the number of vertical pixels of the circumscribed rectangle> the number of vertical pixels of the cutout area (Yes in S41), the cutout area is set so that the upper end of the circumscribed rectangle as the specific portion of the candidate area falls within the area. Yes (S44).

Then, in step S50, the image in the cutout area on the original image is cut out, and in step S60, by the processing of the image memory 50 and the CPU 40,
Display device 90 via image output interface 80
The clipped image is displayed on. Further, the information on the position of the set cutout area is the cutout area information DB 110.
Stored in.

On the other hand, if it is determined in step S30 that the candidate area does not exist in the frame, in step S70 the position information of the cutout area in the previous frame is read from the cutout area information DB 110, and the cutout area is set at the read position. To do. This makes it possible to set the cutout area even when the candidate area cannot be extracted based on the difference when the object remains stationary for a long time.

3 and 4 are examples of the original image and the cut-out image according to the present embodiment, in which the wide-angle camera 10 photographs a person walking in a room. 3 and 4, (a) is an original image, (b) is a cutout image, CA1, CA2 are candidate regions of the object, R1, R2
Is a circumscribed rectangle of the candidate areas CA1 and CA2, and A1 and A2 are cutout areas.

If the candidate area CA1 is sufficiently smaller than the cutout area A1 as in the example of FIG. 3, the cutout area A1 is set with reference to the center CN1 of the candidate area CA1 in step S43. A clipped image in which the object is captured in the center of the image can be obtained. On the other hand, as in the example of FIG. 4, when the number of pixels in the vertical direction of the candidate area CA2 is larger than the number of pixels in the vertical direction of the cutout area A2,
In step S44, the vertical position of the cutout area A2 is determined so that the upper end UE of the circumscribed rectangle R2 of the candidate area CA2 fits within the cutout area A2.
The position in the left-right direction is determined so that the candidate area CA2 is located at the center thereof.

As described above, according to the present embodiment, the reference for setting the position of the cutout area can be switched according to the size relationship between the candidate area of the object and the cutout area. Therefore, for example, when the candidate area is sufficiently smaller than the cutout area, a cutout image that captures the target object is obtained, while when the candidate area is large enough to extend beyond the cutout area, When the object is a person, it is possible to obtain a clipped image such that the upper end portion corresponding to the head does not protrude. That is, it is possible to always cut out the object image in which the surrounding environment is easy to grasp.

Here, the size comparison between the candidate area and the cutout area is performed by comparing the number of pixels in the vertical direction between the circumscribed rectangle of the candidate area and the cutout area. However, the size comparison is performed using other parameters. Good.

Although the reference for switching the position of the cutout area is switched according to the size of the candidate area, the reference may be switched with reference to other information. For example, the criterion may be switched according to the position of the candidate area in the original image as the characteristic of the candidate area. When the actual size of the object is known or can be estimated,
When the position of the object on the camera image is known, the size of the area occupied by the object on the camera image can be estimated. Therefore, for example, when the candidate area is far from the camera, while setting the cutout area based on the center of the candidate area,
When the candidate area is close to the camera, the cutout area may be set so that the upper end of the candidate area is included.

Alternatively, when the distance from the camera to the object can be measured by, for example, a range finder, the reference may be switched according to the distance.

Further, here, when the candidate area is relatively large, the upper portion of the specific portion is set to fit within the cutout area, but the present invention is not limited to this. When the object is a person or an animal, it is preferable to set the face or head as the specific portion. At this time, the position of the face or the head may be detected by some means, and the cutout area may be set so that the position falls. Alternatively, the position of the leg may be detected and the position of the face or the head may be estimated from the position. Alternatively, in the case of an animal, the side edge of the candidate area may be the specific portion.

(Second Embodiment) FIG. 5 is a flow chart showing the flow of image processing according to the second embodiment of the present invention, and the steps common to those in FIG. 2 are designated by the same reference numerals. In FIG. 5, S81 is a step of selecting a candidate area having the largest area when a plurality of candidate areas are detected in step S20, and S82 is a smoothing in the time direction of the original image with respect to the cutout area set in step S40. The step of performing the smoothing process, S83 is a step of delaying the original image by the processing time in the smoothing process in step S82. Here, step S82 corresponds to the smoothing means in the present invention, and step S83 corresponds to the delay means in the present invention. Further, steps S81 and S40 correspond to the cutout area setting means in the present invention.

When the candidate area is extracted by using the background difference or the like, a phenomenon in which the circumscribed rectangle fluctuates for each frame occurs due to a noise factor such as illumination variation. As a result, the cutout area may move due to rattling in each frame, resulting in a display screen that is very difficult to see as a moving image. In the present embodiment, in order to deal with such a problem, the cutout area is smoothed in the time direction.

The operation of the image processing apparatus according to this embodiment will be described with reference to the flowchart of FIG.

In step S10, the image of the camera 10 as the original image is read in the CIF format, and in step S20, the candidate area of the object is extracted from the original image. When the candidate area is detected, step S8
On the other hand, if it is not detected, the process proceeds to step S70.
(S30).

In step S81, when there are a plurality of candidate areas in the frame, the candidate area having the largest area is selected. If there is only a single candidate area, that candidate area is selected. Then, in the cutout area setting step S40, the cutout area is set based on the candidate area selected in step S81 in the same manner as in the first embodiment. The position information of the set cutout area is stored in the cutout area information DB 110.

Then, in step S82, the cutout area set in step S40 is smoothed in the time direction. Here, the center coordinates of the set cutout area are (Xin, Yin), and the frame interval is T.
Then, the center coordinates (Xw,
Yw) is set according to the following equation. Xw (t) = τ * Xin (t)-(1-τ) * Xw (t-T) (1) Yw (t) = τ * Yin (t)-(1-τ) * Yw (t- T) (2) where τ is a time constant for smoothing, and 0 ≦ τ ≦ 1

For example, when T = 0.1 seconds, the smoothing time constant τ is set to about 0.1 to 0.3. As a result, even if the result of the area extraction in step S20 varies slightly for each frame, the cutout area can be moved smoothly.

On the other hand, in step S83, the original image is delayed by N frames (N is a natural number). This is because the cutout area is smoothed in the time direction in step S82, and the movement of the cutout area is delayed with respect to the actual object, so that the timing of the cutout area and the original image are matched. For example, when T = 0.1 seconds and τ = 0.1, the number of delayed frames N may be set to about 5.

Then, in step S50, the image in the smoothed cut-out region is cut out from the original image delayed by N frames, and this is displayed in step S60.

On the other hand, if it is determined in step S30 that the candidate area is not in the frame, in step S70 the position information of the cutout area in the previous frame is read from the cutout area information DB 110, and the read position is smoothed in step S82. It is used for chemical processing. As a result, even when the candidate area is not detected due to the stationary object, the cutout area gradually approaches the position before disappearance.

As described above, according to this embodiment, first,
Even when a plurality of candidate regions are extracted, only the candidate region having the largest area is used for setting the cutout region, so that stable tracking of the object is possible. Further, by smoothing the cutout region in the time direction, the cutout region can be moved smoothly even if the extracted candidate region is slightly different for each frame.

Although the size of the cutout area is assumed to be constant and only the position is moved here, the size may be changed. Further, in the smoothing process using the temporal primary filter, the time constant τ may be changed according to the moving direction of the cutout region and the changing direction of the cutout region (during expansion and reduction).

Further, the smoothing of the cutout region in the time direction may be performed by using another method instead of using the time direction primary filter.

(Third Embodiment) FIG. 6 is a flowchart showing the flow of image processing according to the third embodiment of the present invention, and the steps common to those in FIG. 5 are designated by the same reference numerals. In contrast to FIG. 5, in FIG. 6, the frame delay step S83 is omitted, and instead, a step S90 of changing the setting of the cutout area according to the moving direction of the candidate area is newly added. Steps S81, S40 and S
90 corresponds to the cutout area setting means in the present invention.

In step S90, first, the moving direction from the previous frame is calculated for the candidate area (S91),
Next, it is determined whether the moving direction is the upward direction on the image (S92). Then, when the moving direction of the candidate area is the upward direction, the cutout area is set so that the upper end of the cutout area has a margin from the upper end of the circumscribed rectangle of the candidate area (S93). On the other hand, when the moving direction of the candidate area is other than the upward direction, the cutout area is set so that the upper end of the circumscribed rectangle of the candidate area and the upper end of the cutout area match (S94).

As described in the second embodiment, when the cutout area is smoothed in the time direction, the movement of the cutout area is delayed with respect to the actual object. For this problem,
In the second embodiment, the original image to be cut out is dealt with by delaying the original image accordingly. On the other hand, in the present embodiment, when the target object moves upward,
By setting the cutout area with a margin above the candidate area, when the target is a person,
The phenomenon that the head is chipped in the cutout image is prevented.

As described above, according to the present embodiment, by changing the setting method of the cutout area according to the moving direction of the candidate area, the cutout image in which the object is not missing is output without delaying the image. can do.

(Fourth Embodiment) FIG. 7 shows a fourth embodiment of the present invention.
It is an example of a hardware configuration common to the seventh embodiment. In FIG. 7, the same components as those in FIG. 1 are designated by the same reference numerals. Reference numeral 1 denotes an image transmission device having a built-in image processing device having a function of cutting out an object image shown in FIG. Image transmission device 1 as image processing device
Is an intercom camera 10a instead of the wide-angle camera 10.
And a home camera 10b. The cameras 10a and 10b are connected to the image input interfaces 20a and 20b, respectively. The image input interfaces 20a and 20b are connected to the data bus 30 via the switch 101. The image transmission apparatus 1 includes an encoder / decoder 102 and a transmission / reception circuit 103, and a mobile phone 3 serving as a mobile terminal via the public line network 2.
It is configured to communicate with.

Intercom camera 10a and home camera 10
The switching device 101 selects which image of b is to be processed.
It is done by setting. The cut-out image obtained by processing is encoded by the encoder / decoder 102 instead of being output to the display device 90, and transferred by the transmission / reception circuit 103 to the arbitrary mobile phone 3 via the public line network 2. Is possible. On the contrary, the data from the mobile phone 3 is received by the transmission / reception circuit 103, and the encoder / decoder 1
It is also possible to decrypt with 02 and use it as a processing parameter.

FIG. 8 is a flow chart showing the flow of processing according to the fourth embodiment. In FIG. 8, steps or components having the same functions as those in FIG. 5 are designated by the same reference numerals. In comparison with FIG. 5, a switch SWA for switching an input image is provided instead of the original image input step S10, and switches SWB and 3 are provided instead of the cutout area setting step S40.
Individual cutout area setting steps S101, S102, and S103 are provided.

The image transmission apparatus 1 also includes a parameter set DB 120 having a plurality of types of parameter sets related to the cutout area. The parameters used here are selection by the switch SWA, selection by the switch SWB, lower limit threshold SL and upper limit threshold SU of the number of pixels of the candidate area extracted in step S20, step S8.
2, the time constant τ of the smoothing process, and step S8
The number N of delayed frames in 3. Here, it is assumed that three types of parameter sets are preset: a set 1 for intercom images, a set 2 for seeing a person with a home camera, and a set 3 for seeing a pet with a home camera.

The operation of the system configured as shown in FIGS. 7 and 8 will be described.

Now, it is assumed that the image transmission device 1 and the mobile phone 3 are in communication via the public line network 2. In step S130, the user of the mobile phone 3 sets 1 for an intercom camera image, set 2 for seeing a person with a home camera, and set 3 for seeing a pet with a home camera.
Select one of these parameter sets. The selection here is performed using, for example, a numeric key or a joystick by looking at the selection screen displayed on the display of the mobile phone 3.

The selection result in step S130 is received by the image transmission device 1 via the public line network 2 (S14).
0), the image transmission apparatus 1 accesses the parameter set DB 120 and sets the parameter set according to the received selection result (S150).

Set 1 is a parameter for an intercom camera image, and the switch SWA is set to the a side and the switch SWB is set to the A side. In the intercom camera image, the target visitor can be captured in a large amount in the image. Therefore, the thresholds SL and SU are set to be large, and the time constant τ is set to be relatively large. , Increase the response speed of moving the cutout area. Then, in step S101, the cutout area is set so that the upper end portion of the candidate area of the person is included, and the head of the person is caught in the cutout image in any case.

Set 2 is a parameter for a home camera image and is intended for a person. The switch SWA is set to the b side and the switch SWB is set to the B side. When the indoor camera 10b is used to capture a person in the room, the area of the candidate area is smaller than that of the intercom camera 10a, so the thresholds SL and SU are set smaller than in the set 1.
In addition, since the moving speed of the object on the image becomes slow and the detection accuracy of the candidate region becomes poor, the time constant τ is set small, and the number N of delayed frames is increased accordingly.
The response speed of movement of the cutout area is reduced to prevent the cutout image from rattling. Then, in step S102, the same processing as the cutout area setting step S40 in the first embodiment is performed. That is, when the candidate area of the object is sufficiently smaller than the cutout area, the cutout area is set so as to catch the object in the center thereof, while when the candidate area is large enough to extend beyond the cutout area, In this case, the cutout area is set so that the upper end corresponding to the head does not stick out.

The set 3 is a parameter for a home camera image and is intended for a pet. The switch SWA is set to b.
The switch SWB is set to the C side. When capturing small animals such as pets with the home camera 10b,
Since the area of the candidate region is smaller than that in the case of photographing a person, the threshold values SL and SU are further reduced. In addition, since the moving speed of the object on the image is slow and the detection accuracy of the candidate area is deteriorated, the time constant τ is set to the value in set 2 or less, and the number N of delayed frames is also set in accordance with this
Set it to a value greater than or equal to to reduce the response speed of movement of the cutout area to prevent rattling. And step S1
In 03, a cutout area is set so that a cutout image that captures the object is obtained at the center.

Then, in step S82, the time constant τ
Is used to perform the smoothing process of the cutout region, the delay process of the original image is performed in step S83, and the image cutout is performed in step S50. The generated cutout image is transmitted in the image transmitting step S110, and is displayed on the display screen of the mobile phone 3 in the image displaying step S120.

As described above, according to the present embodiment, a plurality of types of parameter sets related to the cutout area are preset, and the cutout area is set using the parameter set selected from the preset, so that the appropriate cutout area is set. Images can be generated. Further, since the parameter set can be selected according to a selection signal received from the outside, the user can select an appropriate parameter set according to the application and obtain an optimum cut-out image.

Parameters other than those shown in the present embodiment may be included in the parameter set, and some of the parameters shown here may be omitted. Also,
The preset of the parameter set may be performed by the producer at the time of shipping the product, or may be set by the user.

(Fifth Embodiment) FIG. 9 is a flow chart showing the flow of processing according to the fifth embodiment of the present invention. Steps or components common to those in FIG. 8 are designated by the same reference numerals. There is. Compared with FIG. 8, the switch S
An automatic input switching step S161 is provided instead of WA, and steps S130, S140, S15 are also provided.
Instead of 0, the parameter set automatic selection step S1
62 is provided. Furthermore, parameter set DB
The parameter stored in 120A does not include the upper limit and the lower limit of the area of the candidate area, and has a data structure that calls the parameter set with the type of camera that captured the original image and the area of the candidate area as keys. ing.

In the automatic input switching step S161, when a video signal is output from the intercom camera 10a, the video signal is output, while when there is no video signal from the intercom camera 10a, the video signal of the home camera 10b is output. . Parameter set automatic selection step S16
Reference numeral 2 denotes three types of parameters preset in the parameter set DB 120A according to the type of the camera that outputs the video signal selected in the automatic input switching step S161 and the size of the candidate area of the object detected in step S20. Select any one from the set.

That is, when the video signal of the intercom camera 10a is selected in step S161, step S
At 162, set 1 is selected regardless of the size of the candidate region of the object. In step S161, the video signal of the home camera 10b is selected, and the area of the candidate area is 20.
If it is larger than 0 pixel, set 2 is selected in step S162. Furthermore, when the image signal of the home camera 10b is selected in step S161 and the area of the candidate region of the object is larger than 40 pixels and smaller than 200 pixels,
In step S162, set 3 is selected. By such an operation, the parameter set is automatically switched even if the user does not select the parameter set.

As described above, according to the present embodiment, the parameter set is automatically selected according to the type of camera that has captured the original image and the size of the candidate area of the object.
Even if the user does not perform a troublesome operation, it is possible to obtain an appropriate clipped image according to the application.

The present embodiment may be executed in combination with the method in which the user selects the parameter set as in the fourth embodiment. Further, other information, such as calling information of the intercom, may be used for selecting the parameter set. Furthermore, the parameter selection may be performed using either one of the type of camera that captured the original image and the size of the candidate area.

(Sixth Embodiment) FIG. 10 shows a sixth embodiment of the present invention.
9 is a flowchart showing the flow of processing according to the embodiment of the present invention, in which steps or components common to those in FIGS. 2 and 8 are given the same reference numerals. 11 is an example of an original image (home camera image), and FIG. 12 is an example of an image displayed on the mobile phone 3B obtained from the original image of FIG.

Initially, the switches SWC, SWD, SWE
Are all installed on the A side. Here, in step S20, three candidate areas CAa, C as shown in FIG.
It is assumed that Ab and CAc are extracted. Tracking step S17
In 1, the respective candidate areas CAa, CAb, CAc are associated with each other and tracked. In steps S40 and S82, by the method already described,
Cutout areas Aa, Ab, and Ac are determined for each of the candidate areas CAa, CAb, and CAc. Then, in image synthesizing step S172, three types of images are cut out from the delayed image obtained in step S83 according to the three cutout areas Aa, Ab, Ac, and are combined. The generated composite image is transferred to the mobile phone 3B in step S110, and the image as shown in FIG. 12A is displayed (S).
120).

The user who sees the composite image of FIG.
One of the child screens is selected (S173). The selection here is, for example, the number attached to each child screen is set to the mobile phone 3B.
It is done by inputting using the number keys of. The selection result is received in step S174, and at this time, all the switches SWC, SWD, and SWE are set to the B side.

Here, for example, if the user selects the child screen 1, in the image selecting step S175, only the position information of the candidate area CAc corresponding to the selection result of the user is displayed.
It is delivered to the cutout area setting step S40. After this,
Cutout area A obtained in steps S40 and S82
c is given to the clipping step S50, and the candidate area CA
The cutout image including c is transmitted to the mobile phone 3B. As a result, the user can see the cutout image as shown in FIG. After that, the tracking step S17
Since the frames are associated with each other by 1, the user can see the moving image including the candidate area CAc on the screen of the mobile phone 3B.

As described above, according to this embodiment, the user can easily see the target object he / she wants to see even if he / she is in a remote place.

(Seventh Embodiment) FIG. 13 shows a seventh embodiment of the present invention.
9 is a flowchart showing the flow of processing according to the embodiment of the present invention, in which steps or components common to those in FIGS. 2 and 8 are given the same reference numerals. 14 is an example of an original image (home camera image), and FIG. 15 is an example of an image displayed on the mobile phone 3C obtained from the original image of FIG.

Here, as described in the first embodiment, in the cutout area setting step S40, when the candidate area extends in the vertical direction of the cutout area, the upper end portion of the candidate area is placed in the cutout area. The position is set, but when the candidate region extends in the lateral direction of the cutout region, the cutout region is set so that the circumscribed rectangle of the candidate region is aligned with the center without performing position adjustment. For this reason, as shown in FIG. 14, for example, when a person is lying in the room, both end portions of the candidate area CAd will protrude from the cutout area Ad. As a result, the cutout image as shown in FIG.
Is displayed on the screen.

Therefore, in the present embodiment, as shown in FIG. 14, the center of each circumscribing rectangle Rd of the candidate area CAd, each vertex, and the midpoint of each side are defined as the points of interest and are numbered in advance. Then, in the cutout region setting step S40, the position of the cutout region is adjusted so that the designated point is included.

That is, in step S181, the user selects one of the points of interest using, for example, the numeric keys of the mobile phone 3C. The selection result is received by the image transmission apparatus 1C side (S182) and given to the cutout area position correction step S183. For example, if (6) is selected, the position of the cutout region Ad is corrected to the right so that the point of interest (6), that is, the midpoint of the right side, falls within the cutout image. As a result, a clipped image showing the head of the person as shown in FIG. 15B is transmitted to the mobile phone 3C.

As described above, according to the present embodiment, the user can easily see the desired area even when the lying person is out of the cutout area.

Although the original image is in the CIF format in the above description, it may be an image in another format. Further, although the size of the cutout region is set to correspond to the QCIF image, it may be another size, or a shape other than a rectangle, such as a circle or an irregular shape. Further, the extraction of the candidate area is performed by using the background difference, but other methods, for example, extraction by a motion vector, optical flow or shape, or extraction using a color feature may be performed.

The camera for obtaining the original image does not have to have a wide angle. Further, it may not be installed indoors. For example, the image taken by the camera mounted on the robot may be the original image, and the object may be another robot.

A camera is not essential in the image processing apparatus according to the present invention. That is, memory or HDD
The present invention can be realized by causing a PC or the like to execute a program with respect to an original image recorded in, for example.

[0103]

As described above, according to the present invention, it is possible to cut out a well-balanced image of an object, whether the object is small or large. Also,
Since the rattling of the cutout area is eliminated, a display screen that is easy to see as a moving image can be obtained. Further, even if the user is in a remote place, a desired object image can be viewed on the screen by performing a simple operation from the mobile terminal.

[Brief description of drawings]

FIG. 1 is an example of a hardware configuration of an image processing apparatus according to first to third embodiments of the present invention.

FIG. 2 is a flowchart showing a flow of image processing according to the first embodiment of the present invention.

FIG. 3 is an example of an original image and a clipped image according to the first embodiment of the present invention.

FIG. 4 is an example of an original image and a cutout image according to the first embodiment of the present invention.

FIG. 5 is a flowchart showing a flow of image processing according to the second embodiment of the present invention.

FIG. 6 is a flowchart showing the flow of image processing according to the third embodiment of the present invention.

FIG. 7 is an example of a hardware configuration common to the fourth to seventh embodiments of the present invention.

FIG. 8 is a flowchart showing a flow of processing according to the fourth embodiment of the present invention.

FIG. 9 is a flowchart showing a flow of processing according to the fifth embodiment of the present invention.

FIG. 10 is a flowchart showing the flow of processing according to the sixth embodiment of the present invention.

FIG. 11 is an example of an original image.

12 is an example of a display image obtained from the original image of FIG.

FIG. 13 is a flowchart showing the flow of processing according to the seventh embodiment of the present invention.

FIG. 14 is an example of an original image.

15 is an example of a display image obtained from the original image of FIG.

FIG. 16 is a flowchart showing a processing procedure of a conventional object image cropping device.

FIG. 17 is an example of a conventional clipped image.

FIG. 18 is an example of a conventional clipped image.

[Explanation of symbols]

1, 1A, 1B, 1C Image transmission device (image processing device) 3, 3A, 3B, 3C Mobile phone (portable terminal) 10 Wide-angle camera 10a Intercom camera 10b Home camera 40 CPU 50 Image memory 70 Program memory 90 Display device 120, 120A Parameter set DB S20 Candidate area extraction step S40 Cutout area setting step S81 Maximum area area selection step S82 Smoothing step S83 Frame delay step CA1, CA2, CAa, CAb, CAc, CAd Candidate areas A1, A2, Aa, Ab, Ac , Ad Cut out area

Claims (31)

[Claims] 1. A means for extracting a candidate area of an object from a given original image, and a means for setting a cutout area for cutting out the image of the object from the original image based on the candidate area. The image processing apparatus, wherein the cutout area setting unit is configured such that a reference for setting the position of the cutout area can be switched depending on a situation. 2. The image processing apparatus according to claim 1, wherein the cutout area setting unit switches the reference according to a characteristic of the candidate area. 3. The image processing apparatus according to claim 2, wherein the cutout area setting unit uses a size occupied by the candidate area in the original image as a characteristic of the candidate area. 4. The cutout area setting means according to claim 3, wherein when the candidate area is smaller than the cutout area,
The cutout area is set so that the candidate area occupies substantially the center of the cutout area, and when the candidate area is larger than the cutout area, a specific portion of the candidate area fits within the cutout area. In the image processing device, the cutout area is set. 5. The image processing apparatus according to claim 4, wherein the object is a person or an animal, and the specific portion is a face or a head of the object. 6. The image processing apparatus according to claim 4, wherein the specific portion is an upper end portion of the candidate area. 7. The image processing apparatus according to claim 2, wherein the cutout area setting unit uses a position of the candidate area in the original image as a characteristic of the candidate area. 8. The image processing apparatus according to claim 2, wherein the cutout area setting unit uses a moving direction of the candidate area as a characteristic of the candidate area. 9. The image processing apparatus according to claim 1, wherein the cutout area setting unit switches the reference according to a distance from a camera that captured the original image to the object. . 10. The cutout area setting means according to claim 1, wherein when a plurality of the candidate areas are detected, the cutout area is set based on the candidate area having the largest area among them. An image processing device characterized by being present. 11. The image processing apparatus according to claim 1, further comprising means for performing a smoothing process on the cutout region in a time direction of the original image. 12. A means for extracting a candidate area of an object from a given original image, and a means for setting a cutout area for cutting out the image of the object from the original image based on the candidate area. An image processing apparatus, comprising: and a means for performing a smoothing process on the cutout region in the time direction of the original image. 13. The image processing apparatus according to claim 12, further comprising: a unit that delays the original image so as to match the delay in the movement of the cutout region caused by the smoothing process. 14. The image processing apparatus according to claim 12, wherein the smoothing unit changes a time constant in the smoothing process when the cutout area is enlarged and when the cutout area is reduced. 15. The image processing apparatus according to claim 12, wherein the smoothing unit changes a time constant in the smoothing processing according to a moving direction of the cutout area. 16. The image processing apparatus according to claim 12, wherein the cutout area setting unit is configured such that the setting condition of the cutout area can be switched according to a moving direction of the candidate area. . 17. The cut area setting unit according to claim 16, wherein the cut area is set with a margin above the candidate area when the candidate area moves upward. Image processing device. 18. A means for extracting a candidate area of an object from a given original image, and a means for setting a cutout area for cutting out the image of the object from the original image based on the candidate area. The cut-out area setting means sets the cut-out area based on the candidate area having the largest area among the plurality of candidate areas when a plurality of the candidate areas are detected. Image processing device. 19. A means for extracting a candidate area of an object from a given original image, and a means for setting a cutout area for cutting out the image of the object from the original image based on the candidate area. And a parameter set DB that stores a plurality of types of parameter sets related to the cutout area, and the cutout area setting unit includes the parameter set D.
An image processing apparatus, wherein a cutout area is set using a parameter set selected from a plurality of types of parameter sets stored in B. 20. In accordance with claim 19, according to a selection signal received from the outside of the image processing apparatus,
An image processing apparatus comprising means for selecting a parameter set. 21. The means according to claim 19, further comprising means for selecting a parameter set according to at least one of a type of a camera that has photographed the original image and a size of the candidate area. Image processing device. 22. A means for extracting a candidate area of an object from a given original image, and when a plurality of the candidate areas are detected, a cutout image for each candidate area is generated and each cutout image is generated. Means for outputting an image to be displayed in a list, means for receiving a selection signal for selecting one of the plurality of cutout images, and means for selectively outputting the cutout image selected by the selection signal. An image processing apparatus comprising: 23. A means for extracting a candidate area of an object from a given original image, a means for generating and outputting a cutout image related to the candidate area, and a step of correcting the position of the cutout area in the cutout image. An instruction signal for instructing, and means for receiving the cut-out image generating means, according to the instruction signal,
An image processing apparatus, which corrects the position of a cutout area and generates and outputs a new cutout image. 24. An image processing apparatus extracts a candidate region of an object from a given original image, and when a plurality of the candidate regions are detected, generates a cutout image for each candidate region, An image for displaying a list of cutout images is transmitted, the mobile terminal receives the list display image, displays it on the screen, and selects one of the cutout images displayed in the list display image Selection A signal is input to the user, the input selection signal is transmitted to the image processing device, the image processing device receives the selection signal, selects a cutout image according to the selection signal, the An image processing method characterized by transmitting to a mobile terminal. 25. An image processing apparatus extracts a candidate area of an object from a given original image, generates and transmits a cutout image related to the candidate area, and a mobile terminal receives the cutout image. Displayed on the screen, the user is allowed to input an instruction signal for instructing the correction of the position of the cutout area in the cutout image, and the input instruction signal is transmitted to the image processing apparatus, and the image processing apparatus An image processing method, comprising: receiving the instruction signal, correcting the position of the cutout area according to the instruction signal, generating a new cutout image, and transmitting the new cutout image to the mobile terminal. 26. A step of extracting a candidate area of an object from a given original image, and a step of setting a cutout area for cutting out the image of the object from the original image based on the candidate area An image processing method comprising: switching the reference for setting the position of the cutout area according to the situation in the cutout area setting step. 27. A program for causing a computer to perform image processing, the step of extracting a candidate area of an object from a given original image; and the step of extracting the candidate area of the object based on the candidate area. And a step of setting a cutout area for cutting out an image from the original image, in the cutout area setting step, the reference for setting the position of the cutout area is switched depending on the situation. An image processing program characterized by the above. 28. A step of extracting a candidate area of an object from a given original image, and a step of setting a cutout area for cutting out the image of the object from the original image based on the candidate area And a step of performing a smoothing process on the cutout region in the time direction of the original image. 29. A program for causing a computer to perform image processing, the step of extracting a candidate area of an object from a given original image; and the step of extracting the candidate area of the object based on the candidate area. Image processing characterized by causing a computer to execute a step of setting a cutout area for cutting out an image from the original image, and a step of performing a smoothing process on the cutout area in a time direction of the original image. program. 30. Extracting a candidate area of an object from a given original image, and setting a cutout area for cutting out the image of the object from the original image based on the candidate area. In the cutout area setting step, when a plurality of candidate areas are detected, the cutout area is set based on the candidate area having the largest area among them. . 31. A program for causing a computer to perform image processing, the step of extracting a candidate region of an object from a given original image; and a step of extracting the candidate region of the object based on the candidate region. The step of setting a cutout area for cutting out an image from the original image is executed by a computer, and in the cutout area setting step, when a plurality of the candidate areas are detected, the area is the largest. An image processing program, wherein the cutout area is set based on a candidate area.