JP6678505B2 - Imaging device, control method therefor, program, and storage medium - Google Patents

Description

  The present invention relates to an imaging device that performs focus adjustment and subject detection.

  2. Description of the Related Art Conventionally, a technique for performing focus detection based on a phase difference between image signals obtained by an image pickup device having pixels that are pupil-divided by a microlens has been known (Patent Document 1). In Patent Literature 1, each pupil-divided pixel receives a light beam that has passed through a different pupil region of an imaging optical system via a microlens. Further, an image signal can be obtained by adding a plurality of image signals.

  In the focus detection of the phase difference method as described above, determining how much image signal is read out and performing arithmetic processing for focus adjustment is a very important factor in terms of detection accuracy and processing speed. . In addition, if an attempt is made to capture all image signals, an image sensor having two divided pixels has a data amount twice as large as that of a captured image, which imposes a heavy load on a subsequent processing circuit.

  Therefore, an imaging device has been proposed in which a distance information acquisition region for focus adjustment can be arbitrarily set in the imaging device, and the reading time of an image signal from the imaging device is reduced (Patent Document 2). Also, an imaging device capable of generating a distribution of distances (distance maps) of subjects in an image using an image signal obtained from a focus adjustment distance information acquisition region has been proposed (Patent Document 3). By using the distance map of Patent Document 3, distance information between the main subject and the other subjects in the image can be obtained, and the main subject can be detected when the main subject and the other subjects pass each other.

JP 2007-325139 A JP 2012-155095 A JP 2014-074891 A

  However, in the above-described related art, the focus adjustment distance information acquisition area is changed depending on the shooting conditions, but the processing load and the power consumption of the signal read from the pixel are not taken into consideration.

  The present invention has been made in view of the above problem, and an object of the present invention is to realize a technique capable of setting a distance information acquisition area in consideration of a system load and power consumption related to signal processing based on imaging conditions.

In order to solve the above problems and achieve the object, an imaging device of the present invention has a plurality of microlenses, and a plurality of photoelectric conversion units are assigned to each of the plurality of microlenses, so that the plurality of photoelectric conversion units an imaging device which forms one pixel on each copy, and read out sequentially signals from the photoelectric conversion unit of the image pickup element, consisting of a plurality of pixels corresponding to the signal from the plurality of photoelectric conversion units in each pixel of the image sensor A reading unit that outputs image signals in a plurality of frames, an acquisition unit that acquires information indicating an area corresponding to the main subject, and information indicating a focus state of the main subject, and a parallax from the image sensor by the reading unit. a setting means for setting a region for reading different signals, and the setting means, the region where the parallax in each frame read out different signals, signal the parallax differs When the in-focus state of the main subject acquired by the acquiring means is the first state while maintaining the angle of view encompassed by the area to be read substantially equal, the second subject farther from the first state is in focus than the first state. An area from which signals with different parallaxes are read at a higher resolution than in the state of 2 .

In addition, the imaging device of the present invention has a plurality of microlenses, a plurality of photoelectric conversion units are assigned to each of the plurality of microlenses, and an imaging element which forms one pixel for each of the plurality of photoelectric conversion units. the then read out sequentially signals from the photoelectric conversion unit of the image pickup element, and reading means for a plurality of frames and outputs an image signal composed of a plurality of pixels corresponding to the signal from the plurality of photoelectric conversion units in each pixel of the image sensor, Detecting means for detecting the number of subjects in the image signal based on the image signal , and setting means for setting an area for reading a signal having a different parallax from the image sensor by the reading means, In each frame, the area from which the signal from which the disparity is different is read, while maintaining the view angles included in the area from which the signal from which the disparity is different read out are substantially equal, When the number of subjects detected by the serial detection means is a first number, the parallax with a high resolution is to set a region for reading different signal than when the first is a small second number than the number of .

  According to the present invention, it is possible to set a distance information acquisition area in consideration of a system load and power consumption related to signal processing based on an imaging condition.

FIG. 1 is a block diagram illustrating a configuration of an imaging device according to an embodiment. FIG. 2 is a diagram schematically illustrating a pixel arrangement of the image sensor according to the embodiment. FIG. 4 is a diagram schematically illustrating a relationship between a light flux emitted from an exit pupil of a photographing lens and a pixel. FIG. 2 is a configuration diagram of an image sensor according to the embodiment. The circuit configuration diagram of the unit pixel of the image sensor of the present embodiment (a), the configuration diagram of the readout circuit of each unit pixel column of the image sensor (b), and the distance measurement frame set for the pixel array of the image sensor are shown in FIG. FIG. 6 is a timing chart of a read operation of a unit pixel row of the image sensor according to the embodiment. 9 is a flowchart illustrating a setting process of a distance information acquisition area by a signal readout control unit according to the first embodiment. FIG. 4 is a diagram illustrating a distance information acquisition area set by a signal readout control unit according to the first embodiment. FIG. 9 is a block diagram illustrating a configuration of an imaging device according to a second embodiment. 9 is a flowchart illustrating a setting process of a distance information acquisition area by a signal readout control unit according to the second embodiment. FIG. 4 is a diagram illustrating a distance information acquisition area necessary for focus adjustment and subject detection according to the embodiment.

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed depending on the configuration of an apparatus to which the present invention is applied and various conditions. However, the present invention is not limited to the embodiment. Further, a part of each embodiment described later may be appropriately combined and configured.

  [Description of Distance Information Acquisition Area] First, the distance information acquisition area of the present embodiment will be specifically described with reference to FIG.

  FIG. 11 shows a distance information acquisition area (for AF control) arbitrarily set on the imaging screen and a distance map thereof, which are arbitrarily set on the imaging screen during AF control during image capturing, and are necessary for subject detection ( 4 illustrates a relationship between a distance information acquisition region (for main subject tracking) and a distance map thereof. In FIGS. 11A, 11B and 11C, (a-1) to (a-4), (b-1) to (b-4), and (c-1) to (c) -4), (d-1) to (d-4) show each frame of the captured image signal in time series.

  FIG. 11A illustrates an image signal when a distance information acquisition area for AF control is set on the imaging screen. In FIG. 11A, (a-1) shows an image signal at a certain time and a distance information acquisition area for AF control, where 1100 is a main subject, 1101 is a subject other than the main subject, and 1102 is distance information acquisition area. The area is shown. Since it is sufficient that the distance information of the main subject can be acquired for the AF control, the distance information acquisition region set for the AF control need only be around the main subject. Therefore, the distance information acquisition area 1102 is locally set on the screen as in (a-1). As time elapses as in (a-1) to (a-4), the subject 1101 other than the main subject approaches the main subject 1100 (a-2), and the subject 1101 other than the main subject and the main subject 1100 overlaps (a-3), the subject 1101 other than the main subject disappears from the screen, and only the main subject 1100 remains (a-4). That is, the images (a-1) to (a-4) show scenes in which the main subject and other subjects have passed each other.

In the distance map of FIG. 11B, a distance information acquisition area for AF control is set for the main subject 1100. FIG. 11B illustrates a distance map obtained from the distance information acquisition area 1102 in FIG. 11A. In FIG. 11B, (b-1) is a distance map obtained from the distance information acquisition area of (a-1), the grid area 1103 is the distance of the main subject 1100, and the white area 1104 is the distance of the background. Is represented. Further, the black portion 1105 indicates that the distance information cannot be acquired because it is outside the distance information acquisition area. A hatched portion 1106 in (b-3) represents the distance of the subject 1101 other than the main subject in (a-3).

  Here, when the main subject 1100 is detected using the distance map of FIG. 11B, only the distance 1103 of the main subject 1100 is obtained in (b-1) and (b-2), and (b-3) ), The distance of the main subject 1100 suddenly changes. Actually, the distance changes because the subject 1101 other than the main subject overlaps the main subject 1100, but there is a possibility that a correct judgment cannot be made.

  FIG. 11C illustrates an example of an image signal when a distance information acquisition region for subject detection is set on the imaging screen. In FIG. 11C, (c-1) shows an image signal at a certain time and a distance information acquisition region for subject detection. In order to detect the main subject 1100 using the distance information, it is necessary to determine the situation around the main subject. Therefore, it is necessary to set the distance information acquisition area so that the entire screen can be overlooked. Therefore, the distance information acquisition area 1107 is discretely set on the entire screen as in (c-1). FIG. 11 (a) shows that the time has passed as in (c-1) to (c-4), and that the subject 1101 other than the main subject overlaps the main subject 1100 in (c-3). ) Is the same as (a-3).

  FIG. 11D illustrates a distance map obtained from the distance information acquisition area 1107 in FIG. 11C. In FIG. 11D, since the distance information acquisition area 1107 is set so that the entire screen can be overlooked, the main subject is taken from the distance map as time elapses from (d-1) to (d-2). It can be seen that subjects 1101 other than are approaching main subject 1100. In (d-3), the main subject 1100 and the subject 1101 other than the main subject overlap, but in (d-1) to (d-2), the subject 1101 other than the main subject approaches the main subject 1100. And the subject 1101 has a distance 1109 indicated by oblique lines, it can be determined that there is a high possibility that the main subject 1100 and the subject 1101 other than the main subject overlap. On the other hand, the distance 1108 of the main subject 1100 can be obtained only discretely.

  As described above, it is necessary to set the distance information acquisition area for subject detection so that the entire screen can be overlooked.

  However, depending on the shooting conditions (such as the brightness of the subject and the number of subjects), the image signal acquired from the subject detection distance information acquisition region may have a low resolution. For example, the closer the focus state of the subject is to the large blur state (the smaller the contrast evaluation value), the less useful the distance map obtained from the subject detection distance information acquisition area is. It is desirable to prioritize signal reading speed and power consumption by reducing the amount of signal reading from the information acquisition area, and to improve convergence in AF control.

  Therefore, in the present embodiment, at the time of AF control during image shooting, by controlling the signal readout amount of the distance information acquisition area for subject detection based on shooting conditions, it is possible to suppress an increase in load and unnecessary power consumption. I can do it. The density of the distance information acquisition area for AF control is not changed.

[Embodiment 1]
In the present embodiment, an example in which the imaging apparatus is realized by a digital video camera having an autofocus (AF) function of a contrast method and a phase difference method and a subject detection function (main subject tracking function) will be described. It is also applicable to electronic devices such as smartphones and tablet terminals.

<Apparatus configuration>
Hereinafter, the configuration of the imaging device 100 according to the present embodiment will be described with reference to FIG.

  In FIG. 1, an optical system 1 includes a zoom lens, a focus lens, and a diaphragm. The optical system drive unit 2 controls the optical system 1 based on optical system drive information output from an AF control unit 8 described later. The imaging element 3 includes a photoelectric conversion element such as a CMOS, converts a subject image formed on the light receiving surface by the optical system 1 into an electric signal, and outputs an image signal.

  The signal readout control unit 4 is controlled by the AF control unit 8 to drive the image pickup device 3 based on a contrast evaluation value (evaluation information) from a contrast detection unit 5 as evaluation information acquisition means described later to read out an image signal. Control. Note that the image sensor 3 of the present embodiment has a plurality of pixel units that respectively receive light beams that have passed through different pupil regions of the optical system 1 and output pupil-divided image signals. Then, image signals (A-image signal and B-image signal) having different parallax can be individually read from the respective pixels pupil-divided by the drive pulse from the signal read control unit 4. The circuit configuration of the image sensor 3 will be described later with reference to FIGS.

  The contrast detector 5 calculates a contrast evaluation value based on the image signal from the image sensor 3 and outputs the calculated contrast evaluation value to the signal read controller 4.

Based on the image signals (A image signal and B image signal) from the image sensor 3, the distance measurement unit 6 performs distance information for AF control (second depth information) and distance information for detecting a subject in the screen (second information). First depth information and distance map data) are calculated and output to the subject tracking unit 7 and the AF control unit 8.

  The subject tracking unit 7 detects a subject in a captured image based on the image signal from the image sensor 3 and the phase difference information from the distance measurement unit 6, determines a main subject from the detected subjects, Information about the size (hereinafter, subject information) is output to the AF control unit 8.

  The AF control unit 8 is controlled by the system control unit 13 to perform contrast type or phase difference type AF control. The AF control unit 8 outputs a control signal based on the contrast evaluation value of the contrast detection unit 5 to the optical system driving unit 2 in the case of contrast AF. Further, in the case of phase-difference AF, the AF control unit 8 obtains subject information from the subject tracking unit 7 and distance information from the distance measuring unit 6 and outputs a control signal to the optical system driving unit 2.

  The signal processing unit 9 generates an image signal obtained by adding the image signals from the image sensor 3, performs predetermined signal processing, and outputs a display or recording image signal. The signal processing unit 9 performs image processing such as color conversion, white balance correction, and gamma correction, resolution conversion processing, and image compression processing on the generated image signal. An image signal for display or recording is output.

  The recording unit 10 is a memory card, a hard disk, or the like from which an image signal generated by the signal processing unit 9 is recorded and from which an already recorded image is read. The display unit 11 is a liquid crystal panel (LCD) that displays images generated by the signal processing unit 9, various menu screens, and the like. The operation unit 12 is various switches (AF on / off, zoom, etc.) for receiving a user operation, and sends an instruction from the user to the system control unit 13.

  The system control unit 13 includes a CPU, a RAM, a ROM, a dedicated circuit, and the like for controlling various functions of the imaging apparatus 100 in a centralized manner. The CPU executes a control sequence described later by expanding and executing a program stored in the ROM, which is a nonvolatile memory, in the RAM as the work memory.

<Configuration of imaging device>
FIG. 2 is a schematic diagram illustrating a pixel arrangement of the image sensor 3. The unit pixels 200 are arranged in a matrix, and a color filter of R (Red) / G (Green) / B (Blue) is arranged in a Bayer shape for each unit pixel 200. In each unit pixel 200, a sub-pixel a and a sub-pixel b are respectively arranged, and photodiodes (hereinafter, PD) 201a and 201b are arranged in the respective sub-pixels a and b. Each imaging signal output from the sub-pixels a and b is used for focus detection, and an a / b composite signal, which is a signal obtained by adding the imaging signals output from the sub-pixels a and b, is used for image generation. Is done.

  FIG. 3 shows the relationship between light fluxes emitted from different exit pupils of the optical system 1 and the unit pixels 200, and the same parts as those in FIG. 2 are denoted by the same reference numerals.

  As shown in FIG. 3, a color filter 301 and a micro lens 302 are formed on each unit pixel 200. The light that has passed through the exit pupil 303 of the lens enters the unit pixel 200 around the optical axis 304. A light beam passing through a pupil region 305 which is a part of the exit pupil 303 passes through the microlens 302 and is received by the sub-pixel a. On the other hand, a light beam passing through a pupil region 306 which is another part of the exit pupil 303 passes through the microlens 302 and is received by the sub-pixel b. Therefore, the sub-pixel a and the sub-pixel b respectively receive light in different pupil regions 305 and 306 of the exit pupil 303 of the optical system 1. For this reason, by comparing the output signal of the sub-pixel a (image signal A) with the output signal of the sub-pixel b (image signal B), it is possible to perform focus detection by the (imaging plane) phase difference method.

  FIG. 4 shows a circuit configuration of the image sensor 3. In the pixel area PA, the unit pixels 200 are arranged in a matrix (n rows × k columns) like p11 to pkn. Here, the configuration of the unit pixel 200 will be described with reference to FIG. FIG. 5A is a diagram illustrating a circuit configuration of a unit pixel of the imaging device.

  In FIG. 5A, the optical signals incident on the PDs (photoelectric conversion units) 501a and 501b of the sub-pixels a and b described above are photoelectrically converted by the PDs 501a and 501b, and charges corresponding to the exposure amounts are stored in the PDs 501a and 501b. Stored. By setting the signals txa and txb applied to the gates of the transfer gates 502a and 502b to High level, the charges accumulated in the PDs 501a and 501b are transferred to the FD (floating diffusion) unit 503 (charge transfer). The FD section 503 is connected to a gate of a floating diffusion amplifier 504 (hereinafter, referred to as an FD amplifier), and the amount of electric charge transferred from the PDs 501a and 501b is converted into a voltage amount by the FD amplifier 504.

  By setting the signal res applied to the gate of the FD reset switch 505 for resetting the FD unit 503 to a high level, the FD unit 503 is reset. When resetting the charges of the PDs 501a and 501b, the signal res and the signals txa and txb are simultaneously set to High level. As a result, both the transfer gates 502a and 502b and the FD reset switch 505 are turned on, and the PDs 501a and 501b are reset via the FD unit 503. By setting the signal sel applied to the gate of the pixel selection switch 506 to High level, the pixel signal converted into a voltage by the FD amplifier 504 is output to the output vout of the unit pixel 200.

  As shown in FIG. 4, the vertical scanning circuit 401 supplies drive signals such as res, txa, txb, and sel for controlling each switch of the unit pixel 200 to each unit pixel 200. These drive signals res, txa, txb, and sel are common to each row. The output vout of each unit pixel 200 is connected to a column common readout circuit 403 via a vertical output line 402 for each column.

  Here, the configuration of the column common read circuit 403 will be described with reference to FIG.

  The vertical output line 402 is provided for each column of the unit pixels 200, and the output vout of the unit pixels 200 for one column is connected. A current source 404 is connected to the vertical output line 402, and a source follower circuit is configured by the current source 404 and the FD amplifier 504 of the unit pixel 200 connected to the vertical output line 402.

  In FIG. 5B, the clamp capacitance 601 has a capacitance of C1, the feedback capacitance 602 has a capacitance of C2, and the operational amplifier 603 has a non-inverting input terminal connected to the reference power supply Vref. Have. The switch 604 is for short-circuiting both ends of the feedback capacitor 602, and the switch 604 is controlled by the signal cfs.

  The transfer switches 605 to 608 are for transferring signals read from the unit pixels 200 to the signal holding capacitors 609 to 612, respectively. The pixel signal Sa output from the sub-pixel a is stored in the first S signal holding capacitor 609 by a read operation described later. The second S signal holding capacitor 611 stores an a / b combined signal Sab that is a signal obtained by combining (adding) a signal output from the sub-pixel a and a signal output from the sub-pixel b. . The first N signal holding capacitor 610 and the second N signal holding capacitor 612 store the noise signal N of the unit pixel 200, respectively. The signal holding capacitors 609 to 612 are connected to the outputs vsa, vna, vsb, and vnb of the column common readout circuit 403, respectively.

  Horizontal transfer switches 405 and 406 are connected to outputs vsa and vna of the column common read circuit 403, respectively. The horizontal transfer switches 405 and 406 are controlled by an output signal ha * (* is a column number) of the horizontal scanning circuit 411.

  The horizontal transfer switches 407 and 408 are connected to the outputs vsb and vnb of the column common read circuit 403, respectively. The horizontal transfer switches 407 and 408 are controlled by an output signal hb * (* is a column number) of the horizontal scanning circuit 411. The horizontal output lines 409 and 410 are connected to the input of the differential amplifier 414. The differential amplifier 414 takes a difference between the S signal and the N signal, applies a predetermined gain at the same time, and outputs the final output signal to the output terminal 415. Output to

  When the signal chres applied to the gates of the horizontal output line reset switches 412 and 413 is set to High, the horizontal output line reset switches 412 and 413 are turned ON, and the horizontal output lines 409 and 410 are reset to the reset voltage Vchres.

  Hereinafter, the reading operation of the image signal A and the reading operation of the image signal AB which is a composite signal of the image signal A and the image signal B will be described.

  FIG. 5C shows a relationship between a focus adjustment distance information acquisition area set in the pixel area PA of the image sensor 3 and a subject detection distance information acquisition area. The distance measurement frame 620 is set by the distance measurement unit 6 based on area information from the AF control unit 8.

  In the pixel area PA composed of pixels of k columns × n rows, the area indicated by the dotted line is the distance measurement frame 620. An A image signal and an A + B image signal are read from a unit pixel row (pixel line) included in the distance information acquisition region R1 indicated by the hatched portion, and used for image generation, focus detection, and subject detection. From the unit pixel row (pixel line) included in the area R2 other than the distance information acquisition area R1, only the addition signal of the A image signal and the B image signal is read and used only for image generation.

  In addition, as shown in FIG. 5C, when a plurality of regions R1 are set in the vertical direction of the pixel region, the number of rows of the unit pixels 200 in each region R1 may be set to be different from each other.

  Next, a read operation of the image sensor 3 will be described with reference to FIG. FIG. 6A is a timing chart of a read operation performed on each row of the region R2.

  First, the signal cfs is set to a high level to turn on the switch 604, thereby putting the operational amplifier 603 into a buffer state. Next, the signal sel is set to High level to turn on the pixel selection switch 506 of the unit pixel. After that, the signal res is set to the low level, the FD reset switch 505 is turned off, and the reset of the FD section 503 is released.

  Subsequently, after the signal cfs is returned to the low level to turn off the switch 604, the signals tna and tnb are set to the high level, and the first N signal holding capacitor 610 and the second N signal are transferred via the transfer switches 606 and 608. The noise signal N is stored in the storage capacitor 612.

  Next, the signals tna and tnb are set to Low, and the transfer switches 606 and 608 are turned OFF. Thereafter, the signal tsb is set to the high level to turn on the transfer switch 607, and the signals txa and txb are set to the high level to turn on the transfer gates 502a and 502b. With this operation, a signal obtained by combining the charge signal stored in the PD 501 a of the sub-pixel a and the charge signal stored in the PD 501 b of the sub-pixel b is sent to the vertical output line 402 via the FD amplifier 504 and the pixel selection switch 506. Is output. The signal on the vertical output line 402 is amplified by the operational amplifier 603 with a gain according to the capacitance ratio between the capacitance C1 of the clamp capacitance 601 and the capacitance C2 of the feedback capacitance 602, and is transferred via the transfer switch 607 to the second S signal holding capacitance. 611 (a / b combined signal Sab). After sequentially turning off the transfer gates 502a and 502b and the transfer switch 607, the signal res is set to a high level, the FD reset switch 505 is turned on, and the FD section 503 is reset.

  Next, when the output hb1 of the horizontal scanning circuit 411 becomes High level, the horizontal transfer switches 407 and 408 are turned on. Thus, the signals of the second S signal holding capacitor 711 and the second N signal holding capacitor 712 are output to the output terminal 415 via the horizontal output lines 409 and 410 and the differential amplifier 414. The horizontal scanning circuit 411 outputs the a / b composite signal (A + B image signal) for one row by sequentially setting the selection signals hb1, hb2,..., Hbk of each column to High. Note that while the signals in each column are read out by the signals hb1 to hbk, the horizontal output line reset switches 412 and 413 are turned on by setting the signal chres to High level, and the horizontal output lines 409 and 410 are temporarily reset to a reset voltage. Reset to Vchres level.

  The above is the reading operation of each row of the unit pixels in the region R2. As a result, the A + B image signal is read.

  Subsequently, a read operation of each row in the region R1 will be described with reference to FIGS. 6B and 6C. FIG. 6B is a timing chart of the operation until the A image signal is read. The operation starting from setting the signal cfs to the High level, setting the signals tna and tnb to Low, and storing the N signal in the first N signal holding capacitor 610 and the second N signal holding capacitor 612 is shown in FIG. The operation is the same as that described in a).

  When the storage of the noise signal N is completed, the signal tsa is set to High level to turn on the transfer switch 605, and the signal txa is set to High level to turn on the transfer gate 502a. By such an operation, the signal accumulated in the PD 501a of the sub-pixel a is output to the vertical output line 402 via the FD amplifier 504 and the pixel selection switch 506. The signal of the vertical output line 402 is amplified by the operational amplifier 603 with a gain corresponding to the capacitance ratio between the capacitance C1 of the clamp capacitance 601 and the capacitance C2 of the feedback capacitance 602, and the first S signal is held via the transfer switch 605. It is stored in the capacitor 609 (pixel signal Sa).

  Next, when the output ha1 of the horizontal scanning circuit 411 becomes High level, the horizontal transfer switches 405 and 406 are turned ON. Thus, the signals of the first S signal holding capacitor 609 and the first N signal holding capacitor 610 are output to the output terminal 415 via the horizontal output lines 409 and 410 and the differential amplifier 414. The horizontal scanning circuit 411 outputs the signal (A image signal) of the sub-pixel a for one row by sequentially setting the selection signals ha1, ha2,..., Hak of each column to High.

  The signal res remains at the low level, the signal sel remains at the high level, and the reading of the A image signal ends. Thus, the A image signal on the FD section 503 is held without being reset.

  When the reading of the A image signal is completed, the operation proceeds to the operation of reading the A + B image signal shown in FIG. The signal tsb is set to High level to turn on the transfer switch 607, and the signals txa and txb are set to High level to turn on the transfer gates 502a and 502b. By such an operation, the signal stored in the PD 502b of the sub-pixel b is added to the signal of the sub-pixel a held in the FD unit 503, and the added signal is transmitted via the FD amplifier 504 and the pixel selection switch 506. And output to the vertical output line 402. Subsequent portions are the same as the operation of the region R2 described with reference to FIG.

  Thus, the reading operation of each row in the region R1 is completed. Thus, in the region R1, the reading of the A image signal and the reading of the A + B image signal are performed, and the A image signal and the A + B image signal are sequentially read.

<Shooting operation>
Next, an operation of the image capturing apparatus 100 having the above-described configuration when capturing an image will be described.

  First, the optical system 1 drives a stop and a lens in response to a drive signal from the optical system driving unit 2 to form a subject image set to an appropriate brightness on the light receiving surface of the image sensor 3. The imaging element 3 is driven by a driving pulse from the signal readout control unit 4, converts a subject image into an electric signal by photoelectric conversion, and outputs the electric signal as an image signal.

  The signal read control unit 4 reads the A image signal and the A + B image signal from the region R1 by the above-described read operation using the drive pulse corresponding to the contrast evaluation value from the contrast detection unit 5, and reads the A + B image signal from the region R2. The signal is read. As described above, the processing load is reduced by reading the A image signal in a partial area. Further, in the region R1 from which the A image signal has been read, the AF control unit 8 obtains the B image signal by subtracting the A image signal from the A + B image signal, and performs AF control using the A image signal and the B image signal. . Note that AF control may be performed by reading out the A image signal and the B image signal individually from the region R1 and reading out the A + B image signal from the region R2 other than the region R1.

  The contrast detection unit 5 calculates a contrast evaluation value in the distance measurement frame based on the image signal from the image sensor 3 and outputs the calculated contrast evaluation value to the signal readout control unit 4. In this case, the contrast detection unit 5 calculates the contrast evaluation value after adding the A image signal and the B image signal into the same format as the A + B image signal read from the area R2 other than the distance information acquisition area R1.

  Here, the outline of the contrast AF will be described. The contrast AF evaluation value calculator 5 relatively shifts the first focus detection signal obtained from the A image signal and the second focus detection signal obtained from the B image signal in the pupil division direction, and adds and shifts the signal. Is generated, and a contrast evaluation value is calculated from the generated shift addition signal.

  The k-th first focus detection signal is A (k), the second focus detection signal is B (k), the range of the number k corresponding to the distance information acquisition region R1 is W, the shift amount by the shift process is s1, and the shift amount is s1. Assuming that the shift range of the amount s1 is τ1, the contrast evaluation value RFCON is calculated by the following equation.

  By the shift processing of the shift amount s1, the k-th first focus detection signal A (k) and the k-s1th second focus detection signal B (k-s1) are added in association with each other to generate a shift addition signal. , A contrast evaluation value RFCON (s1) is calculated from the shift addition signal.

  The distance measuring unit 6 subtracts the A image signal from the A image signal and the A + B image signal read from the AF control distance information acquisition area and the subject detection distance information acquisition area controlled by the signal read control unit 4. The distance information (first depth information, second depth information) of the subject is calculated using the B image signal obtained by the above. In the present embodiment, the distance information is phase difference information (defocus amount) for performing the AF of the (imaging plane) phase difference method.

  Here, the outline of the phase difference AF will be described. The distance measuring unit 6 relatively shifts the first focus detection signal (A image signal) obtained from the A image signal and the second focus detection signal obtained from the B image signal in the pupil division direction, and matches the signal. Is calculated. The k-th first focus detection signal is A (k), the second focus detection signal is B (k), the range of the number k corresponding to the distance information acquisition region R1 is W, the shift amount by the shift process is s2, and the shift amount is s2. Assuming that the shift range of the amount s2 is τ2, the correlation amount COR is calculated by the following equation.

  By the shift processing of the shift amount s2, the k-th first focus detection signal A (k) and the k-s2nd second focus detection signal B (k-s2) are made to correspond to each other and subtracted to generate a shift subtraction signal. The sum of the numbers k is calculated within the range W corresponding to the distance information acquisition area, and the correlation amount COR (s2) is calculated. Then, from the correlation amount, a shift amount of a real value at which the correlation amount becomes a minimum value is calculated by a sub-pixel operation, and is set as an image shift amount p1. The defocus amount is detected by multiplying the image shift amount p1 by the image height of the focus detection area, the F value of the imaging lens (imaging optical system), and a first conversion coefficient K1 according to the exit pupil distance.

  In the present embodiment, an example is shown in which the distance measurement unit 6 calculates distance information from the A image signal and the B image signal of different parallaxes. However, the information for subject detection is not converted into, for example, “distance”. "Information corresponding to depth" may be used. “Information corresponding to depth” includes, for example, information on “parallax amount (image shift amount)” of A image signal and B image signal generated in the process of being converted into “distance”, and “defocus amount”. ”Or“ subject distance ”. Then, in the present embodiment, the “subject distance” of the “information corresponding to the depth” is obtained by distributing it over the entire screen for subject detection. The “information corresponding to the depth” for subject detection may be recorded in association with the image.

  The present invention can be applied in various embodiments as information corresponding to the depth of a subject in an image. That is, the information (depth information) indicated by the data corresponding to the depth of the subject directly indicates the subject distance from the imaging device to the subject in the image, or the subject distance or the depth of the subject in the image. Any information representing the relative relationship may be used.

  Specifically, the image pickup device 3 can output a pair of light fluxes passing through different pupil areas of the optical system 1 as optical images from a plurality of photoelectric conversion units as a pair of image signals. it can. An image shift amount of each area is calculated by a correlation operation between the paired image signals, and an image shift map representing a distribution of the image shift amount is calculated. Alternatively, the image shift amount is further converted to a defocus amount, and a defocus map representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. When this defocus amount is converted into a subject distance based on the conditions of the optical system 1 and the image sensor 3, distance map data representing the distribution of the subject distance is obtained.

  As described above, in the present embodiment, the distance measuring unit 6 may acquire the image shift map data, the defocus map data, or the distance map data of the subject distance converted from the defocus amount. Each piece of map data may be held in block units or in pixel units. In this case, a bit number of about 8 bits may be assigned to each minimum unit like normal image data, and image processing, display, recording, etc. may be performed as a distance image in the same manner as image processing.

  The subject tracking unit 7 detects a subject based on an image signal from the image sensor 3 and distance information from the distance measurement unit 6, specifies a main subject from the detected subjects, and determines the position and size (size) of the main subject. The subject information about the subject is output to the AF control unit 8. When the subject tracking unit 7 tracks a specific person's face as the main subject (main face), the face located closer to the center of the screen is set as the main face, and the motion vector, color, and size of the main face are used. Detect the destination of the main face. Then, the subject tracking unit 7 performs tracking of the main face based on the distance information of the main face and the distance information of the subject around the main face, and determines the main face when another subject and the main face pass each other. Do.

  When performing AF by the contrast method, the AF control unit 8 detects a focus position (a peak position where the contrast evaluation value is the maximum) based on the contrast evaluation value from the contrast AF evaluation value calculation unit 5 and focuses the main subject. The optical system driving unit 2 outputs optical system driving information for bringing the optical system into a focus state. When performing AF by the phase difference method, the AF control unit 8 detects the in-focus position based on distance information from the distance measuring unit 6 (corresponding to an image shift amount and a defocus amount at which the correlation amount is minimum). Then, optical system drive information for bringing the main subject into focus is output to the optical system drive unit 2. The AF control unit 8 performs the phase difference AF (using the distance information from the distance measuring unit 6) to bring the main subject closer to the in-focus state, and further performs the contrast AF (contrast evaluation value). May be controlled so that the main subject is in focus. That is, the AF control unit 8 performs control such that the main subject is brought into a focused state using at least one of the contrast evaluation value from the contrast AF evaluation value calculation unit 5 and the distance information from the distance measurement unit 6. Is also good.

  The signal processing unit 9 generates image data obtained by converting an image signal from the image sensor 3 into a luminance signal and a color difference signal, and outputs the image data to the recording unit 10 and the display unit 11. The recording unit 10 and the display unit 11 record and display the image data generated by the signal processing unit 9.

<Operation of AF control section and signal readout control section>
Next, referring to FIG. 7, at the time of AF control in the photographing operation, the signal readout control unit 4 is controlled by the AF control unit 8, and drives the image sensor 3 to read out an image signal. The setting process of the information acquisition area will be described.

  In S700, the signal read control unit 4 acquires a contrast evaluation value from the contrast detection unit 5.

  In step S701, the signal readout control unit 4 determines the AF state based on the contrast evaluation value acquired in step S700. The signal readout control unit 4 uses the thresholds Th1 and Th2 (Th1 <Th2) to determine a large blur when the contrast evaluation value is smaller than the threshold Th1, and determines that the contrast evaluation value is larger than the threshold Th1 and smaller than the threshold Th2. Is determined to be a medium blur, and when the contrast evaluation value is larger than the threshold Th2, it is determined to be in focus. If the AF state is determined to be large, the process proceeds to step S702; if the AF state is determined, the process proceeds to step S703; if the AF state is determined to be in focus, the process proceeds to step S703.

  Here, as described above, the distance information acquisition area for subject detection (main subject tracking) needs to be set so that the entire screen can be looked down on, but the signal readout amount does not need to be always constant. Therefore, the subject tracking unit 7 does not need the subject detection distance information when the contrast evaluation value is so small that the main subject cannot be determined. Therefore, in the present embodiment, according to the focus state determined from the contrast evaluation value, the resolution of the distance information acquisition area for subject detection is reduced as the closer to the large blur state, to reduce the signal readout amount. Priority is given to reduction of system load and power consumption related to signal processing for subject detection (main subject tracking). In addition, in the case of a middle-blur state in which the main subject can be determined, the subject detection is gradually required. Therefore, the subject detection is performed so as to balance the reduction in system load and power consumption with the signal readout amount. The resolution of the distance information acquisition area for the medium. Further, in the case of a focused state in which the subject can be detected, the resolution of the distance information acquisition region for subject detection is increased so that the distance information of a subject other than the main subject can be acquired.

  Here, the process of controlling the resolution of the distance information acquisition region for subject detection in S702 to S704 of FIG. 7 will be described in detail with reference to FIG. In FIGS. 8A, 8B, and 8C, (a-1) to (a-4), (b-1) to (b-4), and (c-1) to (c) -4) shows each frame related to a captured image signal in time series. 8A, 8B, and 8C, reference numeral 800 denotes a main subject, reference numeral 801 denotes a subject other than the main subject, and reference numeral 802 denotes a distance information acquisition region for subject detection. In FIG. 8, (a) shows an example of an area setting in the large blur state (S702), (b) shows an example of an area setting in the medium blur state (S703), and (c) shows an in-focus state (S704). Each of the area setting examples is illustrated.

  In the case of the large blur state in S702, the signal readout control unit 4 sets an area so that the resolution of the distance information acquisition area 802 for subject detection becomes low as shown in FIG. In this case, the distance information acquisition area 802 for subject detection on the screen is in the coarsest state, and the signal readout amount is the smallest.

  In the case of the middle-blur state in S703, the signal readout control unit 4 sets the area so that the resolution of the distance information acquisition area 802 for subject detection is medium as shown in FIG. In this case, the subject detection distance information acquisition area 802 on the screen is in an intermediate state between coarse and dense, and the amount of signal readout increases. In this way, in the middle-blur state of FIG. 8B, the distance information of the object other than the main object can be acquired from the distance information acquisition area 802 for subject detection in the large-blur state of FIG. 8A. .

  Since S704 is in the in-focus state, the signal readout control unit 4 sets the area so that the resolution of the distance information acquisition area for subject detection becomes high as shown in FIG. In this case, the distance information acquisition area 802 for subject detection on the screen is in the densest state, and the signal readout amount is the largest. In the in-focus state of FIG. 8C, the distance information of the object other than the main object can be acquired at a higher resolution from the object detection distance information acquisition area 802 than in the middle-blur state of FIG. 8B.

  In step S705, the signal readout control unit 4 calculates control information corresponding to the resolution of the distance information acquisition area set in steps S702 to S704, and outputs a drive pulse to the image sensor 3.

  In step S706, the signal readout control unit 4 determines whether or not the photographing operation has been completed, triggered by an instruction to end the photographing from the AF control unit 8, and repeats the processing from S700 until it is determined that the photographing operation has been completed. .

  According to the present embodiment, the system load and power consumption related to signal processing can be reduced by controlling the density of the distance information acquisition area for subject detection based on the contrast evaluation value during AF control during image capturing. it can.

  Second Embodiment Next, a second embodiment will be described.

  In the first embodiment, the signal readout control unit 4 controls the resolution of the subject detection distance information acquisition area based on the contrast evaluation value from the contrast detection unit 5. On the other hand, in the second embodiment, the signal readout control unit 4 controls the resolution of the distance information acquisition region for subject detection based on the number of subjects from the subject tracking unit 7.

  In the second embodiment, the same elements as those in the first embodiment are denoted by the same reference numerals, and the description is omitted.

  FIG. 9 shows a configuration of an imaging apparatus 100 according to the second embodiment. The difference from the first embodiment is that the object tracking unit 7 uses the image signal from the image sensor 3 and the distance information from the distance measurement unit 6 to obtain information. The signal reading unit 4 detects the number of subjects in the captured image and outputs the detected information to the signal reading unit 4, and the signal reading unit 4 determines the resolution of the subject detection distance information acquisition area based on the number of subjects from the subject tracking unit 7. The point to control.

  Next, referring to FIG. 10, at the time of AF control in the photographing operation, the signal readout control unit 4 is controlled by the AF control unit 8, and the distance for subject detection when the image sensor 3 is driven to read out an image signal. The setting process of the information acquisition area will be described.

  In S1000, the signal readout control unit 4 acquires the number of subjects from the subject tracking unit 7.

  In S1001, the signal read control unit 4 determines whether the number of subjects acquired in S1000 is large, small, or medium. The signal readout control unit 4 uses the thresholds Th3 and Th4 (Th3 <Th4) to determine that the number of subjects is small when the number of subjects is smaller than the threshold Th3, and determines that the number of subjects is larger than the threshold Th3 and smaller than the threshold Th4. It is determined to be medium, and when the number of subjects is larger than the threshold Th4, it is determined to be large. When it is determined that the number of subjects is small, the process proceeds to S1002, when it is determined that the number is medium, the process proceeds to S1003, and when it is determined that the number is large, the process proceeds to S1004.

  In the present embodiment, the number of subjects is used as a criterion because the greater the number of subjects, the higher the possibility that other subjects pass by in front of the main subject. This is because the possibility of necessity increases.

  In step S1002, when the number of subjects is large, the signal readout control unit 4 performs area setting such that the resolution of the distance information acquisition area for subject detection becomes high, as in FIG. 8C.

  In S1003, when the number of subjects is medium, the signal readout control unit 4 sets the region so that the resolution of the distance information acquisition region 802 for subject detection is medium, as in FIG. I do.

  In S1004, when the number of subjects is small, the signal readout control unit 4 performs area setting such that the resolution of the distance information acquisition area 802 for subject detection is low, as in FIG. 8A.

  In S1005, the signal read control unit 4 calculates control information corresponding to the resolution of the distance information acquisition area set in S1002 to S1004, and outputs a drive pulse to the image sensor 3.

  In step S <b> 1006, the signal readout control unit 4 determines whether or not the shooting operation has been completed, triggered by an instruction to end the shooting from the AF control unit 8, and repeats the processing from S <b> 1000 until it is determined that the shooting operation has been completed. .

  According to the present embodiment, the system load and power consumption related to signal processing can be reduced by controlling the density of the distance information acquisition region for subject detection based on the number of subjects during AF control during image capturing. it can.

In the above-described embodiment, the distance information acquisition region for subject detection is set based on the number of subjects. However, the motion vector of the subject is detected, and there is a high possibility that other subjects pass by in front of the main subject. and to increase the resolution of the distance information acquisition region when it is determined, it may be lower when it is determined that there is a low possibility that pass each other.

[Other embodiments]
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 1 ... Optical system, 2 ... Optical system drive part, 3 ... Image sensor, 4 ... Signal readout control part, 5 ... Contrast detection part, 6 ... Distance measurement part, 7 ... Subject tracking part, 8 ... AF control part, 9 ... Signal processing unit, 10 recording unit, 11 display unit, 12 operation unit, 13 system control unit

Claims (9)

An image sensor having a plurality of microlenses, a plurality of photoelectric conversion units assigned to each of the plurality of microlenses, and one pixel for each of the plurality of photoelectric conversion units;
Reading means for the image signal composed of a plurality of pixels is a plurality of frames output corresponding to the signals from the plurality of photoelectric conversion units in each pixel of the to read out sequentially signals from the photoelectric conversion portion of the imaging element, the imaging element,
Acquisition means for acquiring information indicating an area corresponding to the main subject and information indicating an in-focus state of the main subject,
Setting means for setting an area from which a signal having a different parallax is read from the image sensor by the reading means ,
The setting unit is configured to read out the signal from which the disparity is different in each frame, while maintaining substantially the same angle of view included in the area from which the signal is read out from the disparity.
When the focus state of the main subject acquired by the acquisition unit is the first state, the parallax differs at a higher resolution than when the focus state is the second state farther from the first state. An imaging device, wherein an area from which a signal is read is set . An image sensor having a plurality of microlenses, a plurality of photoelectric conversion units assigned to each of the plurality of microlenses, and one pixel for each of the plurality of photoelectric conversion units;
Reading means for the image signal composed of a plurality of pixels is a plurality of frames output corresponding to the signals from the plurality of photoelectric conversion units in each pixel of the to read out sequentially signals from the photoelectric conversion portion of the imaging element, the imaging element,
Detecting means for detecting the number of subjects in the image signal based on the image signal;
Setting means for setting an area from which a signal having a different parallax is read from the image sensor by the reading means ,
The setting means, while maintaining a region for reading the signal having the different parallax in each frame, a view angle included in the region for reading the signal having the different parallax is substantially equal,
When the number of subjects detected by the detection means is a first number, an area for reading out the signal having the different parallax with a higher resolution than when the number of subjects detected is a second number smaller than the first number is set. An imaging device characterized by the above-mentioned. 3. The imaging apparatus according to claim 1, further comprising a calculation unit configured to calculate depth information using a signal read from an area from which the signal having the different parallax set by the setting unit is read. 4. The calculating means for calculating depth information using a signal read from a region from which a signal from which the disparity set by the setting means reads a different signal,
  The imaging apparatus according to claim 1, wherein the obtaining unit obtains information indicating a focus state of a subject obtained based on the depth information. The calculating means for calculating depth information using a signal read from a region from which a signal from which the disparity set by the setting means reads a different signal,
  The imaging apparatus according to claim 2, wherein the detection unit detects a position, a size, and a number of the subject based on the depth information. An image sensor having a plurality of microlenses, a plurality of photoelectric conversion units assigned to each of the plurality of microlenses, and one pixel for each of the plurality of photoelectric conversion units;
To read out sequentially signals from the photoelectric conversion unit of the image pickup element, and a readout means for a plurality of frames and outputs an image signal composed of a plurality of pixels corresponding to the signal from the plurality of photoelectric conversion units in each pixel of the image sensor A method for controlling an imaging device having
An acquisition step of acquiring information indicating an area corresponding to the main subject and information indicating a focus state of the main subject,
A setting step of setting a region from which a signal having a different parallax is read from the image sensor by the reading unit,
In the setting step, the area for reading the signal with the different parallax in each frame, while maintaining substantially the same angle of view included in the area for reading the signal with the different parallax,
When the focus state of the main subject acquired in the acquisition step is the first state, the parallax is different at a higher resolution than in the second state farther from the first state. A method for controlling an imaging device, comprising setting an area from which a signal is read . An image sensor having a plurality of microlenses, a plurality of photoelectric conversion units assigned to each of the plurality of microlenses, and one pixel for each of the plurality of photoelectric conversion units;
To read out sequentially signals from the photoelectric conversion unit of the image pickup element, and a readout means for a plurality of frames and outputs an image signal composed of a plurality of pixels corresponding to the signal from the plurality of photoelectric conversion units in each pixel of the image sensor A method for controlling an imaging device having
A detecting step of detecting the number of subjects in the image signal based on the image signal;
A setting step of setting a region from which a signal having a different parallax is read from the image sensor by the reading unit,
In the setting step, the area for reading the signal with the different parallax in each frame, while maintaining substantially the same angle of view included in the area for reading the signal with the different parallax,
When the number of subjects detected in the detection step is the first number, an area for reading out the signal having the different parallax with a higher resolution than when the number of the subjects detected is the second number smaller than the first number is set. A method for controlling an imaging device, comprising: The The control method according to claim 6 or 7, a program to be executed by a computer of an imaging device. The The control method according to claim 6 or 7, read a storage medium according to the stored computer program to be executed by a computer of an imaging device.

Images