JP2011139432A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent the occurrence of interlace-specific comb noise in an imaging apparatus wherein image data is fetched from an image sensor at a frame frequency out of the frame frequency of the NTSC system, progressively written into a frame memory, and read out in an interlaced manner, and also to provide a low-cost imaging apparatus because a clock oscillator is mounted as the only oscillator according to the NTSC system. <P>SOLUTION: A frame memory 105 is divided into at least three areas. A control unit 10 sequentially switches roles of the respective areas as a writing-in area and a read-out area, determines whether writing into the area to be read next is completed when finishing read-out from a certain area, and switches the read-out area. The image sensor 101 is operated by a clock of the oscillator 107 according to the NTSC system, and by changing a blanking period in a vertical direction or a blanking period in a horizontal direction, the image sensor 101 outputs image data at a frame frequency out of the frame frequency of the NTSC system. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image capturing apparatus such as an in-vehicle camera used by being mounted on a vehicle, and more particularly to an NTSC output image capturing apparatus.

  In general, in-vehicle cameras shoot at a shooting frame number of about 30 fps with an image sensor such as a CMOS sensor in order to match an NTSC display device or the like. When, for example, an LED type traffic light on the road is photographed with such a vehicle-mounted camera, the following problems occur.

  Since the LED type traffic light is driven by a driving voltage obtained by full-wave rectification of a commercial AC power supply, it cannot be visually confirmed, but the lighting and extinguishing of the signal is repeated periodically in a very short time. Therefore, if the charge accumulation time of the image sensor and the signal extinction period overlap in a certain frame, the extinction state of that frame is actually off even if any of red, blue, or green is on As a result. Then, when such frames are continued, an image obtained by photographing a traffic light in an extinguished state is output over several tens of frames. This is a particularly serious problem in a drive recorder. A drive recorder records video taken by an in-vehicle camera and is widely used when conducting accident analysis. However, if the exact state of a traffic light at the time of an accident or near-miss can not be determined, Obstacles will arise in responsibility determination.

  In order to cope with the above problems, it is known to set the frame frequency of the image sensor to a frame frequency (offset frame frequency) offset by a predetermined offset frequency with respect to the NTSC frame frequency (for example, , See Patent Document 1). According to this, even if an LED type traffic light or the like is photographed, an unlit video is not output over several tens of frames at the time of lighting, but the frame frequency of the video signal is out of the NTSC frame frequency. Therefore, it cannot be displayed on the NTSC display device as it is.

  As a countermeasure against this, image data captured from the image sensor is stored in a frame memory at a frame frequency (offset frame frequency) that is excluded from the NTSC frame frequency, and is read from the frame memory at the NTSC frame frequency. Can be output to However, this causes the following new problems.

  Normally, reading of image data from the image sensor is performed in a progressive manner. On the other hand, an NTSC display device employs an interlace method. Therefore, in the above method, the image data read from the image sensor is written to the frame memory by the progressive method, and is read from the frame memory by the interlace method. In this case, since the time for writing one frame of image data to the frame memory is different from the time for reading one frame of image data from the frame memory, comb noise peculiar to the interlace method is generated even if the frame memory is a double buffer. . Comb noise is comb-like image shift that occurs in a frame image composed of odd and even fields in different frames.

  The present invention makes it difficult to be affected by a blinking cycle when a lighting device such as an LED traffic light is lit, and an image pickup device at a frame frequency that is out of the NTSC frame frequency so that the NTSC display device can be used as it is. An image pickup apparatus that reads image data from a frame memory, writes it in a frame memory in a progressive manner, and reads out from the frame memory in an interlace manner at an NTSC frame frequency. To do.

  Further, according to the present invention, only the oscillator for generating a clock signal in the NTSC system is mounted on the apparatus, and image data can be read from the image sensor at a frame frequency that is out of the frame frequency of the NTSC system with a simple configuration. An object is to provide a low-cost imaging device.

  According to the first aspect of the present invention, an optical system for forming an optical image of a subject and an optical image formed by the optical system are picked up, and image data is output at a frame frequency Ffs deviating from the NTSC frame frequency Ffr. An image pickup device, a frame memory for storing the image data, and a controller that progressively writes the image data output from the image pickup device to the frame memory and interlaces from the frame memory at an NTSC frame frequency Ffr The frame memory is divided into at least three areas, and the control unit switches each area of the frame memory as a writing area and a reading area, and sequentially switches the roles thereof. The frame image data is written progressively sequentially, and one frame of image data from each area. The features sequentially read it to interlaced.

  According to a second aspect of the present invention, in the image pickup device according to the first aspect, when the reading of one frame of image data from a certain region is completed, the control unit stores one frame of image data in a region to be read next. It is characterized in that it is determined whether or not writing is completed, and if not completed, one frame of image data is read out again from the area that has just been read out.

  According to a third aspect of the present invention, in the image pickup apparatus according to the second aspect, if the writing of one frame of image data is completed in the next read area, the control unit reads the next area. If it is not completed, reading of the image data is started from the area to be read next, and if it is completed, the area to be read next is skipped. The image data reading is started from the next area to be read next.

  According to a fourth aspect of the present invention, in the image imaging device according to any one of the first to third aspects, the control unit performs a predetermined image conversion process when reading the image data from the frame memory. Features.

  According to a fifth aspect of the present invention, in the image imaging device according to any one of the first to fourth aspects, the image sensor outputs image data at a frame frequency Ffs having a constant frame period.

  According to a sixth aspect of the present invention, in the image pickup device according to any one of the first to fourth aspects, the image sensor has an indefinite frame period adjacent to each other and an average of the frame periods of the NTSC system. The image data is output at a variable frame frequency Ffs.

  A seventh aspect of the present invention is the image pickup apparatus according to any one of the first to sixth aspects, further comprising a single oscillator for generating a clock signal compatible with the NTSC system, wherein the control unit is configured from the oscillator. The interlaced readout from the frame memory at the frame frequency Ffr of the NTSC system is controlled based on the clock of the clock frequency compatible with the NTSC system, and the imaging device inputs the clock of the clock frequency compatible with the NTSC system from the oscillator. The image data is output at a frame frequency Ffs deviating from the NTSC frame frequency Ffr by changing the vertical blanking period or the horizontal blanking period.

  According to an eighth aspect of the present invention, in the image pickup device according to the seventh aspect, the image pickup device has a register, and a vertical blanking period or a horizontal blanking period is set according to a parameter value set in the register. It is characterized by changing.

  A ninth aspect of the present invention is the image pickup apparatus according to any one of the first to sixth aspects, further comprising a single oscillator for generating a clock signal compatible with the NTSC system, wherein the control unit is configured from the oscillator. The interlaced readout from the frame memory at the frame frequency Ffr of the NTSC system is controlled based on the clock of the clock frequency compatible with the NTSC system, and the imaging device is a clock deviated from the clock frequency compatible with the NTSC system from the oscillator. It is characterized by operating.

  According to a tenth aspect of the present invention, in the image pickup device according to the ninth aspect, the image pickup device converts a clock deviating from the clock frequency corresponding to the NTSC system to a frequency n times by a PLL inside the image pickup device, Image data is output at a frequency after conversion.

  According to an eleventh aspect of the present invention, in the image pickup apparatus according to the tenth aspect, the clock and the clock after conversion are set to other than the common divisor and common multiple of the clock frequency corresponding to the NTSC system.

  According to a twelfth aspect of the present invention, in the image pickup device according to the eleventh aspect, the image pickup device deviates from the NTSC frame frequency Ffr by changing a vertical blanking period or a horizontal blanking period. The image data is output at the frame frequency Ffs.

  According to a thirteenth aspect of the present invention, in the image pickup device according to the eighth or twelfth aspect, the image pickup apparatus has a CPU for controlling the entire apparatus, and the CPU sets a parameter value of a register in the image pickup element. To do.

  According to a fourteenth aspect of the present invention, in the image pickup device according to any one of the first to thirteenth aspects, the image pickup device outputs a vertical synchronization signal having a frame frequency Ffr together with image data, and the control unit The read timing of the frame memory is controlled based on the vertical synchronization signal from the image sensor.

  In the present invention, an image sensor is used at a frame frequency that is out of the NTSC frame frequency so that it is not easily affected by the blinking cycle when the lighting device such as an LED traffic light is turned on, and the NTSC display device can be used as it is. In the image pickup apparatus in which image data is read out from the frame memory, written in the frame memory in a progressive manner, and read out from the frame memory in an interlaced manner at an NTSC frame frequency, the frame memory is divided into at least three regions, By switching the area sequentially as the writing area / reading area, switching the reading area by checking whether the writing to the next reading area is completed when reading from one area is completed, Generation of comb noise peculiar to the interlace method can be prevented.

  In the image pickup apparatus of the present invention, the clock oscillator to be mounted on the apparatus is only an oscillator for generating a clock signal compatible with the NTSC system, and the image pickup device is operated with a clock having a clock frequency corresponding to the NTSC system from the oscillator. A low-cost image capturing apparatus can be provided by changing the vertical blanking period or the horizontal blanking period to output image data at a frame frequency that is out of the NTSC frame frequency.

  Note that various functions and effects of the image pickup apparatus of the present invention other than those described above will become apparent from the description of the following embodiment.

1 is an overall configuration diagram of an embodiment of an image capturing apparatus of the present invention. The figure which shows the relationship between the output signal from the conventional image sensor, and the read signal from a frame memory. The figure which shows the relationship between the output signal from the image pick-up element of this invention, and the read signal from a frame memory. The detailed block diagram of the control part in FIG. The figure which shows the process image of the coordinate calculation means in FIG. Shows exposure timing of the illumination light and an image sensor that flashing period T _o (Part 1). Shows a reflected way of the traffic due to the exposure timing of the imaging element of the illumination light and the rotor ring shutter for flashing period T _o. Shows exposure timing of the illumination light and an image sensor that flashing period T _o (Part 2). The figure which shows the transition of the writing / reading in the case of the conventional double buffer, and the state of an output image (writing time = reading time). Write / read timing chart for double buffer (write time> read time). The figure which shows the transition of the writing / reading in the case of a double buffer, and the state of an output image (writing time> reading time). Write / read timing diagram for double buffer (write time <read time). The figure which shows the transition of writing and reading in the case of a double buffer, and the state of an output image (writing time <reading time). The figure which shows the specific example of the output image of comb noise generation | occurrence | production. The figure which shows the example of a processing flow of image data reading at the time of setting it as the triple buffer of this invention. Write / read timing diagram for triple buffer (write time> read time). The figure which shows the transition of the writing and reading in the case of a triple buffer, and the state of an output image (writing time> reading time). FIG. 17B is a continuation of FIG. 17A. Write / read timing chart for triple buffer (write time <read time). The figure which shows the transition of the writing and reading in the case of a triple buffer, and the state of an output image (write time <read time). FIG. 19B is a continuation diagram of FIG. 19A. The figure which shows the specific implementation example of the address preparation part of FIG. FIG. 21 is a state transition diagram of the state machine in FIG. 20. A truth table for explaining the operation of the address calculation unit in FIG. The whole block diagram of another embodiment of the image pick-up device of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an overall configuration diagram of an embodiment (first embodiment) of an image pickup apparatus according to the present invention. The image capturing apparatus includes an optical system 101, an image sensor 102, a processing unit 103, an NTSC encoder 104, and a CPU 108 that controls the overall operation. The processing unit 103 includes a frame memory 105, a control unit 106, and an oscillator 107 therein. The NTSC encoder 104 is connected to an NTSC display device or an image recorder, but is omitted in FIG. Here, the oscillator 107 is a crystal element for generating a clock signal compatible with the NTSC system. Specifically, the clock frequency is 13.5 MHz. The clock (CK) of the oscillator 107 is supplied to the control unit 106 in the processing unit 10 and simultaneously to the NTSC encoder 104 and further to the image sensor 102.

  The image sensor 102 receives an NTSC clock frequency (13.5 MHz) clock from the oscillator 107 as an input, and an offset frame frequency Ffs offset by a predetermined offset frequency from the NTSC frame frequency Ffr (about 30 FPS). The image data photographed through the optical system 101 is read progressively (for example, 27 FPS) and sent to the processing unit 103. Specifically, the image sensor 102 includes a register (sensor register), and changes the vertical blanking period or the horizontal blanking period in accordance with the parameter value set in the register from the CPU 108, thereby The frame frequency of data is set to Ffs (for example, 27 FPS). The image sensor 102 sends the vertical synchronization signal Sv having the frame frequency Ffs to the processing unit 103 together with the image data.

  Image data output from the image sensor 102 is progressively written sequentially into the frame memory 105 in the processing unit 103 at the timing of the frame frequency Ffs (for example, 27 FPS). The image data written in the frame memory 105 is sequentially read in an interlaced manner at the timing of the NTSC frame frequency Ffr (about 30 FPS) under the control of the control unit 106. The control unit 106 designates a read address for the frame memory 105, controls the read timing of the frame memory 105 based on the vertical synchronization signal Sv from the image sensor 102, and receives a clock corresponding to the NTSC system from the oscillator 107. A clock having a frequency (13.5 MHz) is received, and interlaced reading from the frame memory 105 at the timing of the NTSC frame frequency Ffr is controlled. The configuration of the control unit 106 will be described later.

  The image data read from the frame memory 105 is subjected to digital / analog conversion processing by the NTSC encoder 104 to be converted into an NTSC video signal and sent to a display device or recorder (not shown).

  As in the configuration of FIG. 1, image data is captured from the image sensor 102 at an offset frame frequency Ffs offset by a predetermined offset frequency with respect to the NTSC frame frequency Ffr, and written into the frame memory 105 (progressive). Reading from the memory at the frame frequency Ffr of the NTSC system (interleaving) makes the image less susceptible to the blinking cycle when the lighting device such as the LED traffic light is turned on, and can use the NTSC display device as it is. An imaging device can be realized. A countermeasure against occurrence of comb noise will be described later.

  The image sensor 102 also receives a clock (CK) having a clock frequency corresponding to the NTSC system from an oscillator 107 mounted in the processing unit 103, and image data output from the image sensor 102 is an NTSC frame. Since the frame frequency Ffs deviates from the frequency Ffr, a separate oscillator for the frame frequency Ffs is unnecessary, and a low-cost imaging device can be realized. This will be described in detail below.

  FIG. 2 shows the relationship between the output signal from the conventional image sensor and the read signal from the frame memory. FIG. 2 (A) shows the output signal (progressive) from the image sensor, and FIG. Is a read signal (interlace) from the frame memory. That is, conventionally, in accordance with the NTSC system, the vertical blanking period of the output signal from the image sensor and the read signal from the frame memory is made the same. For this reason, in order to set the image data output from the image sensor to a frame frequency Ffs that is out of the NTSC frame frequency Ffr, an NTSC-compatible clock signal generation for reading image data from the frame memory is generated. It is necessary to prepare an oscillator (for example, 12.2 MHz) having a different clock frequency in addition to the oscillator (13.5 MHz) for reading image data from the image sensor. In the embodiment of the present invention, such an oscillator is not necessary.

  FIG. 3 shows the relationship between the output signal from the image sensor and the readout signal from the frame memory in the embodiment of the present invention. FIG. 3 (a) shows the output signal (progressive) from the image sensor. 3 (b) is a read signal (interlace) from the frame memory. Here, the read signal from the frame memory is the same as in FIG. On the other hand, the vertical blanking period of the output signal from the image sensor is changed. That is, in the present embodiment, image data reading from the image sensor 102 and image data reading from the frame memory 105 are performed with the same clock (13.5 MHz) from the oscillator 107 for generating a clock signal compatible with the NTSC system. At this time, by changing the vertical blanking period for reading the image data from the image sensor 102, the output image data of the image sensor 102 has a frame frequency Ffs deviating from the NTSC frame frequency Ffr. I have to. As described above, the image sensor 102 includes a register (sensor register), and changes the blanking period in the vertical direction according to the parameter value set in the register from the CPU 108. The example of FIG. 3 is a case where Ffs <Ffr, but it is also possible to satisfy Ffs> Ffr by changing the parameter value of the sensor register. Also, by sequentially changing the parameter value of the sensor register for each frame by the CPU 108, as will be described later, when the adjacent frame periods are indeterminate and averaged, the image sensor is arranged at the NTSC frame frequency Ffr. It becomes possible to drive.

  In FIG. 3, the vertical blanking period for reading image data from the image sensor 102 is changed. However, the horizontal blanking period or both the vertical and horizontal blanking periods are changed. The output image data of the image sensor 1402 may have a frame frequency Ffs deviating from the NTSC frame frequency Ffr.

  Next, the configuration operation of the control unit 106 will be described. FIG. 4 shows a detailed configuration diagram of the control unit 106. The control unit 106 includes a frame memory read start trigger creation unit 201, an output vertical / horizontal synchronization signal creation unit 202, and an address creation unit 203. Note that the address creation unit 203 may include coordinate calculation means 204 for distortion correction, viewpoint conversion, and the like.

  The frame memory read start trigger creation unit 201 receives the vertical synchronization signal Sv from the image sensor 102, and operates as follows immediately after the imaging apparatus is powered on. Note that the image sensor 102 is assumed to perform progressive reading. One Sv immediately after power-on of the imaging device is skipped, and the frame memory read start trigger (Tr) is asserted (validated) in the second Sv.

  By doing so, it is possible to prevent image data from being read out from the frame memory 105 when no image data exists from the frame memory 105.

  The output vertical horizontal synchronizing signal generation unit 202 uses the NTSC clock frequency (13.5 MHz) clock from the oscillator 107 to read the image from the frame memory 105 to the NTSC frame frequency Ffr (about 30 FPS). In this way, a vertical synchronization signal and a horizontal synchronization signal are created.

  The address creating unit 203 sequentially designates the read address of the frame memory 105 based on the clock (CK) from the oscillator 107, the vertical synchronizing signal and the horizontal synchronizing signal from the output vertical and horizontal synchronizing signal creating unit 202, and the frame The image data written in the memory 105 is read at the timing of the NTSC frame frequency Ffr (about 30 FPS).

  Here, the address creating unit 203 does not sequentially specify the read address of the frame memory 105, but instead uses a certain coordinate conversion rule (for example, distortion correction, viewpoint conversion such as a look-down image, center cutout conversion, side cutout conversion, three-plane conversion, etc. Coordinate calculation means 204 for designating according to mirror conversion or the like may be included.

  FIG. 5 shows a processing image of the coordinate calculation means 204. This is an example of outputting an image obtained by rotating and deforming image data (input image) from an image sensor by 90 °. Here, FIG. 5A shows a positional relationship between coordinates before and after conversion of a certain two pixels, and FIG. 5B shows an input image and an output image stored in the frame memory 105. The positional relationship of corresponding pixels is shown.

  When the image data is read from the frame memory 105, the coordinate calculation unit 204 reads pixel data of a pixel having coordinates (x1, y1) on the input image as a pixel having coordinates (X1, Y1) on the output image. As described above, the read address is calculated. Further, as the pixel at the coordinates (X2, Y2) of the output image, the read address is calculated so that the image data of the pixel at the coordinates (x2, y2) on the input image is read. By reading the image data from the corresponding address in the frame memory 105 based on this read address, an image obtained by rotating the input image by 90 ° can be obtained as the output image.

  Similarly, the image of the fisheye camera can be transformed to correct the distortion of the image caused by the fisheye image, or it can be transformed into an image seen from directly above by image transformation based on viewpoint transformation. An easy-to-see image can be displayed on the driver on a back monitor or the like that looks at the rear part. Since various configurations and processing algorithms of the coordinate calculation means 204 have been proposed in the past, detailed description thereof will be omitted.

  Next, by reading out image data from the image sensor 102 at a frame frequency (offset frame frequency) Ffs offset by a predetermined offset frequency with respect to the NTSC frame frequency Ffr, a lighting device such as an LED traffic light is turned on. The effect of the flicker cycle in the case will be specifically described.

  FIG. 6 and FIG. 7 show the case where the sampling period of the frame is performed at a constant period excluded from the NTSC system. In this case, the parameter value of the sensor register in the image sensor 102 is constant.

6, the graph on the 1st, represents the light amount change of the illumination light flashing period T _o (e.g., 60 Hz signal machine). The convex part of the graph is the brightest state (bright) and the valley part is dark (dark). Here, the turn-off period and _{t o.}

FIGS. 6A and 6B show the exposure timing of an image sensor for progressive readout with a global shutter. In the figure, t _i is the exposure time. (A) is a case where the image sensor is driven at 30 FPS, and (b) is a case where the image sensor is removed from 30 FPS. In the case of (a), since it is almost synchronized with the frame frequency Ffr (29.97 Hz) of the NTSC system, when the exposure enters the timing of the illumination light extinction, the illumination light of the extinction is continuously captured for several seconds. The On the other hand, in the case of (b), since it does not synchronize with the frame frequency Ffr of the NTSC system, even if the exposure enters the timing of extinction of the illumination light, it is easy to escape from it. It is avoided that the image is taken.

The same can be said for the rolling shutter. FIGS. 6C and 6D show the exposure timing of an image sensor for progressive reading with a rolling shutter. In the figure, t _i is the exposure time. Unlike the global shutter, the rolling shutter exposes each line independently. FIGS. 6C and 6D show that the exposure time gradually shifts to the long side, such as the first row, the second row, the third row,... (C) When the image sensor is driven at 30 FPS, (d) is removed from 30 FPS, and the line exposed in the shaded area in the figure shows the blinking illumination light in that line. The image is captured in a state. In the case of (c), since it is almost synchronized with the NTSC frame frequency Ffr (29.97 Hz), if the exposure enters the timing of the extinction of the illumination light, the extinction illumination light is continuously captured for several seconds. The On the other hand, in the case of (d), since it is not synchronized with the frame frequency Ffr of the NTSC system, even if exposure enters the timing of extinction of the illumination light, it is easy to escape from it, and the illumination light of the extinction continues for several seconds. However, imaging is avoided.

7, an illumination light flashing period T _o, a rolling shutter, shows the reflected how traffic by the exposure timing of the image pickup device of the progressive read. On the left side of FIG. 7, since the exposure has entered the illumination light extinction timing, the color of the traffic light is not imaged. On the right side, since the exposure is out of the illumination light extinction timing, the color of the traffic light is ( For example, blue) is imaged.

  Next, FIG. 8 shows a case in which the sampling period of adjacent frames is indefinite and averaged so as to correspond to the NTSC system (30 frames per second). In this case, the parameter value of the sensor register in the image sensor 102 changes for each frame.

Graph top of FIG. 8, similarly to FIG. 6 represent the light amount change of the illumination light flashing period T _o (e.g., 60 Hz signal machine). The convex part of the graph is the brightest state (bright) and the valley part is dark (dark). Again, the turn-off period and _{t o.}

  8A to 8D, ti is an exposure time, Tf, Tf ′, Tf ″ is a frame sampling period. Here, Tf is an NTSC sampling period, and Tf ′, Tf ″ is NTSC. This is a period shifted from the sampling period of the system to the short time side or the long time side. However, in order to support the NTSC system, 30 frames can be output per second.

  FIGS. 8A and 8B show the exposure timing of an image sensor for progressive readout with a global shutter. (A) shows a case where the sampling period is fixed (Tf) to drive the image sensor, and (b) shows a case where the sampling period is indefinite (Tf ′, Tf ″, etc.) and the image sensor is driven. In the case of a), if the exposure enters the timing of extinction of the illumination light, the extinction illumination light is continuously imaged for several seconds, whereas in the case of (b), the exposure is performed at the timing of extinction of the illumination light. Even if it enters, it is easy to escape from the timing of extinction, so it is avoided that the extinction illumination light is continuously imaged for several seconds, and the sampling period of adjacent frames is made indefinite and averaged. By outputting 30 frames per second, it is possible to correspond to the NTSC system.

  The same can be said for the rolling shutter. FIGS. 8C and 8D show the exposure timing of the progressive reading image sensor with a rolling shutter. Unlike the global shutter, the rolling shutter exposes each line independently. Similarly to FIGS. 6C and 6D, in FIGS. 8C and 8D, the exposure time is gradually increased as shown in the first row, the second row, the third row,... It shows that it has moved to. The row exposed in the shaded area in the figure is imaged in a blinking state when the blinking illumination light is transferred to that row. (C) shows a case where the sampling period is fixed (Tf) and the image pickup element is driven, and (d) shows a case where the sampling period is set indefinite (Tf ′, Tf ″, etc.) and the image pickup element is driven. In the case of c), if the exposure enters the timing of extinction of the illumination light, the extinction illumination light is continuously imaged for several seconds, whereas in the case of (d), the exposure is performed at the timing of extinction of the illumination light. Even if it enters, it is easy to escape from the timing of extinction, so it is avoided that the extinction illumination light is continuously imaged for several seconds, and the sampling period of adjacent frames is made indefinite and averaged. By outputting 30 frames per second, it is possible to correspond to the NTSC system.

  The image data progressively read out from the image sensor 102 at the frame frequency Ffs deviating from the NTSC frame frequency Ffr is written in the frame memory 105 as it is, and is interleaved from the frame memory 105 at the NTSC frame frequency. Read to the race. Thereby, an NTSC display device and an image recorder can be used as they are. However, since the time for writing one frame of image data to the frame memory 105 is different from the time for reading one frame of image data from the frame memory 105, comb noise peculiar to the interlace method is generated. This will be specifically described below.

  First, a case will be described in which progressive writing is performed to the frame memory 105 at the NTSC frame frequency Ffr, and interlace reading is performed from the frame memory 105 at the same NTSC frame frequency Ffr. In this case, if the frame memory 105 is divided into at least two areas such as area A and area B (double buffer), each area is set as a write area and a read area, and their roles are switched sequentially, no comb noise occurs.

  FIG. 9 shows the transition of writing / reading in area A and area B and the state of the output image (receiver side) in this case. Here, writing is 30 frames / second, progressive, and reading is 60 fields / second, interlaced. Note that the same pattern in the figure represents image data of the same frame output from the image sensor 102. The same applies to the subsequent drawings.

  In the case of FIG. 9, the time for writing one frame of image data to one area of the frame memory 105 is the same as the time for reading one frame (odd field + even field) of image data from the other area. Once the start and read start timings are matched once (for example, (a)), the write end and the read end are permanently aligned. Therefore, the write area and the read area can be switched at the write end timing. For example, when the writing to the area A is completed, the writing to the area B is started and the reading is started from the area A (for example, (d)). As a result, a frame composed of an odd field and an even field of the same frame can be output at all times, so that no comb noise is generated (for example, (c), (e)).

  Next, a case will be described in which progressive writing is performed to the frame memory 105 at a frame frequency Ffs removed from the NTSC frame frequency Ffr, and interlaced reading is performed from the frame memory 105 at the NTSC frame frequency Ffr. In this case, comb noise is generated even if the frame memory 105 is a double buffer.

  First, a case where Ffs <Ffr, that is, a case where the writing time of one frame of image data is longer than the reading time of one frame of image data will be described.

  FIG. 10 shows a timing chart of writing / reading to / from the double buffer (area A, area B) in this case. Attention is now paid to the dotted line in FIG. 10, that is, the timing when reading from the region A is completed. Since the reading from the area A is completed, it is desired to start the reading from the area B and to start the writing to the area A by switching the writing / reading area. However, at this time, the writing is not finished in the region B. For this reason, reading (interlacing) is performed again from the area A, and when the writing of the area B is completed, the reading / writing area is switched, reading from the area B is started, and writing to the area A is started. It becomes. In this case, comb noise in which odd field rows and even field rows of different frames are mixed in the output image occurs. The same applies to the case where the relationship between the target read area and the write area is reversed.

  FIG. 11 shows the writing / reading of the double buffer (area A, area B) when Ffs <Ffr, that is, when the writing time of one frame of image data is longer than the reading time of one frame of image data. And the state of the output image (receiver side). FIG. 11 shows a case where reading is performed at 30 frames / second and interlaced reading corresponding to the NTSC system, while writing is performed at 27 frames / second and progressive writing.

  Now, progressive writing to the area A is started, and at the same time, interlaced reading is started from the area B (a), and attention is paid to the time when the reading from the area B is completed (c). At this time, in the area A, writing is in progress. For this reason, the interlace reading is started again from the area B, and when the writing of the area A is completed (d), the writing / reading area is switched, the area B is written, and the area A is read. At this time, from area A, interlaced reading after the state of (d) is continued (e). As a result, the output image when one frame of image data is read is as shown in (f). That is, comb noise is generated.

  Next, a case where Ffs> Ffr, that is, a case where the writing time of one frame of image data is shorter than the reading time of one frame of image data will be described.

  FIG. 12 shows a timing chart of writing / reading to / from the double buffer (area A, area B) in this case. Attention is now paid to the dotted line in FIG. 12, that is, the timing at which writing to the region B is completed. Since the writing to the area B is completed, the writing / reading area is switched, the writing to the area A is started, and the reading from the area B is started. However, at this point, the area A is being read out. When the write / read area is switched in this state, comb noise in which odd field rows and even field rows of different frames are mixed is generated in the output image as in the case of Ffs <Ffr. The same applies when the relationship between the noted write area and read area is reversed.

  FIG. 13 shows writing / reading of the double buffer (area A, area B) when Ffs> Ffr, that is, when the writing time of one frame of image data is shorter than the reading time of one frame of image data. And the state of the output image (receiver side). FIG. 13 shows a case where reading is performed at 30 frames / second and interlaced reading corresponding to the NTSC system, while writing is performed at 33 frames / second and progressive writing.

  Now, the progressive writing to the area A is started, and at the same time, the interlaced reading from the area B is started (a), and attention is paid to the time when the writing to the area A is completed (b). At this time, in the region B, reading is still in progress. Specifically, the reading of the odd field has been completed, but the even field is still being read. However, since the writing to the area A is completed, the writing / reading area is switched, the area B is written, and the area A is read. At this time, interlaced reading from the area A after the state of (b) is continued. As a result, the output image when one frame of image data is read is as shown in (c). That is, comb noise is generated.

  FIG. 14 shows an example of the output image. FIG. 14A shows an image sensor driven at an NTSC frame frequency Ffr, and progressively writes image data output from the image sensor to a frame memory. The frame memory is also interlaced at an NTSC frame frequency Ffr. No noise is generated. On the other hand, FIG. 14B shows that the image sensor is driven at a frame frequency Ffs (here, Ffs <Ffr) deviating from the NTSC frame frequency, and output image data from the image sensor is progressively written to the frame memory. It can be seen that the image is an interlaced read from the frame memory at the NTSC frame frequency Ffr, and that comb noise is generated in a part of the image.

  In this way, the present invention captures image data from the image sensor at a frame frequency Ffs removed from the NTSC frame frequency Ffr, writes it progressively to the frame memory, and reads it from the frame memory to the interlace at the NTSC frame frequency Ffr. In some cases, the generation of comb noise peculiar to the interlace method is to be prevented. Hereinafter, the device / configuration will be described in detail.

  In the present invention, the frame memory 105 is divided into at least three areas such as area A, area B, and area C (triple buffer), and the roles are sequentially switched using each area as a write area and a read area.

  Here, the transition of the writing area is repeated from area A → area C → area B → area A. The transition of the read area is repeated in the order of area B (state 1) → area A (state 2) → area C (state 3) → area B (state 1). However, for example, if the writing of the area A is not completed at the time of transition from the state 1 to the state 2, all the data in the area B is read again. In addition, when the transition from state 1 to state 2 is completed, the writing of the area A is completed, and if the writing of the area C is also completed, the state 2 is skipped and the transition is made to the state 3. The same applies to the transition from state 2 to state 3 and from state 3 to state 1.

  FIG. 15 shows a processing flowchart for reading image data from the frame memory 105 when the frame memory 105 of the present invention has a triple buffer configuration. Hereinafter, State 1 will be described, but the same applies to State 2 and State 3.

  Now, data reading from area B (interlaced reading; the same applies hereinafter) is started (step 1). When reading of all data from area B is completed (step 2), data writing in area A (progressive writing; (Same as above) is confirmed (step 3). Here, when the data writing in the area A is not completed, all the data in the area B is read again (steps 1 and 2), and the process proceeds to step 3 again.

  When it is confirmed in step 3 that data writing in area A has been completed, it is next confirmed whether data writing in area C has been completed (step 4). If the data writing in the area C has not been completed, the state shifts to the state 2 and data reading is started from the area A (step 5). If the data writing in the area C has been completed, the state 2 is skipped and the process proceeds to the state 3 to start reading data from the area C (step 9).

  In this way, the frame memory is at least a triple buffer (area A, area B, area C), and all data is read (or reread) from a certain area, and when the reading is completed, writing to the next area to be read is performed. The generation of comb noise can be prevented by confirming whether or not the reading has been completed and switching the reading area. Also, at the end of reading from a certain area, check whether writing to the next area to be read is completed, and if writing has been completed, switch the reading area to that area, Image data can be provided. The frame memory needs to be at least a triple buffer, but of course may have four or more memory areas.

  The processing flowchart of FIG. 15 can deal with the relationship between the writing time and the reading time when either writing time> reading time or writing time <reading time. Hereinafter, each case will be described in detail.

  First, the case where write time (write cycle)> read time (read cycle) will be described. This is because image data is captured from the image sensor 101 at a frame frequency Ffs (Ffs <Ffr) lower than the NTSC frame frequency Ffr, written progressively into the frame memory 105, and read out from the frame memory 105 at the NTSC frame frequency Ffr. It is a case.

  FIG. 16 shows a timing chart of writing / reading to / from the triple buffer (area A, area B, area C) in this case. In FIG. 16, Sv is a vertical synchronization signal from the image sensor 102. The write completion signal is asserted when writing to a certain area is completed, and negated when reading is started from that area.

  Now, pay attention to timing a in FIG. Timing a is when data reading from the region B is completed in state 1 (step 2 in FIG. 15). According to the processing flow of FIG. 15, when it is confirmed whether or not the data writing is completed in the area A (step 3 in FIG. 15), the writing to the area A is not completed as can be seen from the negation of the writing completion signal. is there. Therefore, reading is started again from the area B (step 1 in FIG. 15).

  Next, pay attention to timing b in FIG. Timing b is the time when the second reading from the region B is completed (step 2 in FIG. 15). According to the processing flow of FIG. 15, it is confirmed whether or not writing to the area A is completed again (step 3 of FIG. 15). As can be seen from the assertion of the writing completion signal, the writing to the area A is completed. . Therefore, according to the processing flow of FIG. 15, if it is further confirmed whether or not writing to the area C is completed (step 4 in FIG. 15), since the area C is being written, reading is started from the area A (FIG. 15). Step 5).

  In this case, since the data written in the area B is read twice, the same image is output once every several frames. By switching, even when the writing time of one frame in the frame memory is longer than the reading time of one frame, it is possible to output a frame composed of an odd / even frame image of the same frame, and order of comb noise can be prevented.

  FIGS. 17A and 17B show the write / read transition and output of the triple buffer (region A, region B, region C) when the above write time (write cycle)> read time (read cycle). It shows the state of the image. Also here, an example is shown in which reading is performed at 30 frames / second and interlaced reading corresponding to the NTSC system, whereas writing is performed at 27 frames / second and progressive writing.

  Now, pay attention to (e). This is a case where (a) to (d) and state 1 → state 2 → state 3 are repeated several times to return to state 1. (E) represents a state in which reading of all data (one frame) in the odd and even fields from the region B is completed. At this time, the area A is also being written. Therefore, the reading is started again from the area B (f). Then, the reading (re-reading) from the area B is completed. At this time, when the writing to the area A is completed and the writing to the area C is not completed, the reading is started from the area A (state 2). (G) represents this state.

  As shown in FIGS. 17E and 17G, in the case of writing time> reading time, the same frame is output once every several frames, but it is possible to prevent occurrence of comb noise.

  Next, a case where write time (write cycle) <read time (read cycle) will be described. This is because image data is captured from the image sensor 101 at a frame frequency Ffs (Ffs> Ffr) higher than the NTSC frame frequency Ffr, written progressively into the frame memory 105, and read out from the frame memory 105 at the NTSC frame frequency Ffr. It is a case.

  FIG. 18 shows a timing chart of writing / reading to / from the triple buffer (area A, area B, area C) in this case. In FIG. 18, Sv is a vertical synchronization signal from the image sensor 102. The write completion signal is asserted when writing to a certain area is completed, and negated when reading is started from that area.

  Now, pay attention to timing b in FIG. Timing b is the point in time when reading from region B is completed in state 1 (step 2 in FIG. 15). When it is confirmed whether writing to the area A is completed according to the processing flow of FIG. 15 (step 3 of FIG. 15), the writing to the area A is completed. Normally, the read area is switched to area A (transition to state 2), but it is further confirmed whether writing to area C is completed according to the processing flow of FIG. 15 (step 4). In FIG. 18, the writing to the area C is completed at the timing a before the timing b. Therefore, the area A is skipped, the reading area is switched to the area C (transition to state 3), and reading is started from the area C (step 5 in FIG. 15).

  As described above, when the writing time is smaller than the reading time, the image data is thinned out and output once every several frames, but the generation of comb noise can be prevented. Further, it is possible to absorb the difference between the frame frequency Ffs for capturing image data and the frame frequency Ffr corresponding to the NTSC system. Further, since the area C stores image data newer than the area A, the latest image data can be provided by skipping the area A and selecting the area C.

  FIGS. 19A and 19B show the transition and output of write / read of the triple buffer (area A, area B, area C) when the above write time (write cycle) <read time (read cycle). It shows the state of the image. Also here, an example is shown in which reading is performed at 30 frames / second and interlaced reading corresponding to the NTSC system, whereas writing is performed at 33 frames / second and progressive writing.

  Now, pay attention to (d). This is a case where (a) to (c) and state 1 → state 2 → state 3 are repeated several times to return to state 1. (D) represents a state in which reading of all data (one frame) in the odd and even fields from the region B is completed. At this time, the writing to the area A is completed (c), and the writing to the area C is also completed (d). Therefore, the area A is skipped, and reading is started from the area C (transition to state 3). (E) represents this state. Note that (e) represents a state in which reading of all data (one frame) in the odd and even fields from the region C has been completed, and further represents a state in which writing to the region B has been completed.

  As shown in FIGS. 19D and 19E, in the case of writing time <reading time, image data is thinned out and output once every several frames, but generation of comb noise can be prevented.

  The processing flow of FIG. 15 is handled by the address creation unit 203 (FIG. 4) in the control unit 106 of FIG. FIG. 20 shows a specific implementation example of the address creation unit 203. Here, the address creation unit 203 includes a state machine 1100, an address calculation unit 1101, and a RAM / IF 1102. The address calculation unit 1101 also serves as the coordinate calculation unit 204 in FIG. In the figure, Sv is a vertical synchronizing signal according to the frame frequency Ffs from the image sensor 102. Vv is a vertical synchronizing signal according to the NTSC frame frequency Ffr, and is created by the output vertical / horizontal synchronizing signal creating unit 202 (FIG. 4).

  When the RAM / IF 1102 receives Sv, the RAM / IF 1102 sequentially designates the write address o_calram_a and outputs o_calram_xwe = 00. When Vv is received, the read address o_calram_a is sequentially specified and o_calram_xwe = 01 is output. o_calram_xwe is a write enable signal (having negative logic).

  The state machine 1100 operates (transitions) as shown in FIG. In FIG. 21, (a) is a transition diagram at the time of writing, and (b) is a transition diagram at the time of reading. In (b), a solid line indicates completion of writing in the next read area (hereinafter, write completion signal 1), and a broken line indicates completion of writing in the next next read area (hereinafter, write completion signal 2).

  Write (WRITE) transitions to the next state at the rise of Sv. Otherwise, data is written to the next address. For example, in the state B (writing to the region B), the state transits to the state A (writing to the region A) at the rising edge of Sv. The output o_write is 00 → 01. Except for the rise of Sv, if it is in state B, data is written to the next address in area B.

  Reading (READ) shifts to the next state if the writing completion signal is asserted at the rise of Vv. Otherwise, the next address data is read. When all the data has been read, it returns to the first address in the current area and sequentially reads. For example, if the write completion signal 1 is asserted at the rise of Vv and the write completion signal 2 is negated in the state B (reading the region B), the state transitions to the state A (reading the region A). The output o_read is 00 → 01. If the write completion signal is negated at a time other than the rise of Vv or at the rise of Vv, for example, in state B, the data at the next address in region B is read. When all the data in the area B has been read out, for example, the address returns to the first address in the area B and is read sequentially. Further, for example, in the state B (reading the region B), if the write completion signal 1 is asserted at the rise of Vv and the write completion signal 2 is also asserted, the state transitions to the state C (reading the region C). . The output o_read is 00 → 10.

  The address calculator 1101 operates with the truth table shown in FIG. I_write and i_read are received from the state machine 1100, and i_addr and i_xwe are received from the RAM / IF 1102. i_write indicates to which area to write, i_read indicates from which area to read, i_addr is an address of the memory, and i_xwe is a write / read designation to the memory. As shown in FIG. 22, i_addr is arranged according to the area specified by i_write and i_read, and the address of the frame memory 105 is specified.

  FIG. 23 shows an overall configuration diagram of another embodiment (Embodiment 2) of the image pickup apparatus according to the present invention. The image capturing apparatus includes an optical system 101, an image sensor 102, a processing unit 103, an NTSC encoder 104, and a CPU 108 that controls the overall operation. The processing unit 103 includes a frame memory 105, a control unit 106, an oscillator 107, and a PLL 109 therein. The NTSC encoder 104 is connected to an NTSC display device or image recorder, but is omitted in FIG. Here, the oscillator 107 is a crystal element for generating a clock signal compatible with the NTSC system, and the PLL 109 outputs the clock frequency CK (13.5 MHz) compatible with the NTSC system received from the oscillator 107 as it is, and at the same time, the NTSC system. A clock frequency CK ′ deviating from the clock frequency (13.5 MHz) corresponding to the system is created. Since the PLL 109 incorporates the PLL in the chip in which the control unit is built in, it is not necessary to add new parts, so that the configuration in FIG. 23 can be realized without considering the cost and the installation area of the base.

  The clock frequency CK (13.5 MHz) corresponding to the NTSC system from the PLL 109 is supplied to the control unit 106 in the processing unit 103 and simultaneously to the NTSC encoder 104. A clock frequency (CK ′) deviating from the clock frequency corresponding to the NTSC system is given to the image sensor 102. In the EMC countermeasure, since the smaller the clock frequency, the more effective the clock frequency CK ′ is preferably lower than the clock frequency CK. For example, CK ′ = 12.656525 MHz.

  The image sensor 102 receives the clock frequency CK ′ deviated from the clock frequency corresponding to the NTSC system from the PLL 109 and converts the clock frequency CK ′ into an n-fold frequency by a PLL (not shown) inside the image sensor 102. Hereinafter, the converted frequency is referred to as a sensor output clock frequency. For example, when n = 2, the sensor output clock frequency is 25.3125 MHz. Comparing the sensor output clock frequency of 27 MHz of normal NTSC with the sensor output clock frequency of 25.3125 MHz of this configuration, the sensor output clock frequency of this configuration is lower, which is effective for EMC countermeasures. Also, setting the sensor output clock frequency to other than the common divisor and common multiple of the normal NTSC sensor output clock frequency of 27 MHz is also effective for EMC countermeasures. The image sensor 102 progressively converts image data captured through the optical system 101 with an offset frame frequency Ffs (for example, 27 FPS) offset by a predetermined offset frequency with respect to the NTSC frame frequency Ffr (about 30 FPS). Read and send to processing unit 103.

  As described in the first embodiment, the image sensor 102 includes a register (sensor register), and the vertical blanking period or the horizontal blanking period is changed from the CPU 108 according to the parameter value set in the register. Thus, the frame frequency of the image data is set to Ffs (for example, 27 FPS). The image sensor 102 sends the vertical synchronization signal Sv having the frame frequency Ffs to the processing unit 103 together with the image data. Since the operation of the processing unit 103 is the same as that of the first embodiment, the description thereof is omitted.

DESCRIPTION OF SYMBOLS 101 Optical system 102 Image pick-up element 103 Processing part 104 NTSC encoder 105 Frame memory 106 Control part 107 Oscillator 108 CPU
109 PLL
201 frame memory read start trigger creation unit 202 output vertical / horizontal synchronization signal creation unit 203 address creation unit 204 coordinate calculation means

JP 2009-181339 A

Claims (14)

An optical system that forms an optical image of the subject;
An image pickup device that picks up an optical image formed by the optical system and outputs image data at a frame frequency Ffs deviating from the NTSC frame frequency Ffr;
A frame memory for storing the image data;
A control unit that progressively writes image data output from the image sensor to the frame memory and reads the frame memory interlaced at a frame frequency Ffr of the NTSC system;
Dividing the frame memory into at least three areas;
The control unit sequentially switches the role of each area of the frame memory as a writing area / reading area, sequentially writes one frame of image data to each area, and interlaces one frame of image data from each area An image pickup apparatus characterized by sequentially reading out images.   When the reading of one frame of image data from a certain area is completed, the control unit determines whether writing of one frame of image data has been completed in the next reading area. The image capturing apparatus according to claim 1, wherein one frame of image data is read again from the region.   If the writing of one frame of image data to the next reading area is completed, the control unit determines whether writing of one frame of image data to the next reading area is completed. If not, start reading image data from the area to be read next, and if completed, skip the area to be read next and start reading image data from the area to be read next. The image capturing apparatus according to claim 2, wherein   The image capturing apparatus according to claim 1, wherein the control unit performs predetermined image conversion processing when reading image data from the frame memory.   The image pickup device according to claim 1, wherein the image pickup device outputs image data at a frame frequency Ffs having a constant frame period.   5. The image sensor outputs image data at a variable frame frequency Ffs, in which adjacent frame periods are indefinite and an average of them becomes an NTSC frame period, 5. The image pickup device according to claim 1. It has a single oscillator for clock signal generation compatible with the NTSC system,
The control unit controls interlace reading from the frame memory at the frame frequency Ffr of the NTSC system based on the clock of the clock frequency corresponding to the NTSC system from the oscillator,
The imaging device operates by inputting a clock having a clock frequency corresponding to the NTSC system from the oscillator, and changes from a vertical blanking period to a horizontal blanking period to deviate from the NTSC frame frequency Ffr. The image pickup apparatus according to claim 1, wherein the image data is output at a frame frequency Ffs.   The image pickup apparatus according to claim 7, wherein the image pickup device includes a register, and changes a vertical blanking period or a horizontal blanking period in accordance with a parameter value set in the register. It has a single oscillator for clock signal generation compatible with the NTSC system,
The control unit controls interlace reading from the frame memory at the frame frequency Ffr of the NTSC system based on the clock of the clock frequency corresponding to the NTSC system from the oscillator,
The image pickup device according to claim 1, wherein the image pickup device operates by inputting a clock deviating from a clock frequency compatible with the NTSC system from the oscillator.   The image pickup device converts a clock deviating from the clock frequency compatible with the NTSC system into a frequency n times as high as a PLL inside the image pickup device, and outputs image data at the converted frequency. An image pickup apparatus according to claim 1.   The image pickup apparatus according to claim 10, wherein the clock and the clock after conversion are set to a value other than a common divisor and a common multiple of a clock frequency compatible with the NTSC system.   12. The image sensor outputs image data at a frame frequency Ffs deviating from the NTSC frame frequency Ffr by changing a vertical blanking period or a horizontal blanking period. An image pickup apparatus according to claim 1.   The image capturing apparatus according to claim 8, further comprising a CPU for controlling the entire apparatus, wherein the CPU sets a parameter value of a register in the image sensor. The image sensor outputs a vertical synchronization signal having a frame frequency Ffr together with image data,
The image capturing apparatus according to claim 1, wherein the control unit controls a read timing of a frame memory based on a vertical synchronization signal from the image sensor.