JP2012234425A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device having high responsiveness to an input by a user, even when a long time is required for image processing.SOLUTION: An image processing device comprises: a storage device for storing RAW image data and simple image data associated with the RAW image; an output control section for performing an output request for each partial area when requesting an output of the RAW image data; and an image processing section for performing RAW development processing to an entire image area of the stored RAW image data corresponding to the output request from the output control section. The image processing section outputs, for the output request, the simple image data corresponding to the partial area until the RAW development processing is completed, and after the completion of the RAW development processing, outputs the developed RAW image data corresponding to the partial area. After the image data corresponding to the partial area is outputted by the image processing section, the output control section performs an output request for the next partial area if there is not a cancellation request.

Description

  The present invention relates to an image processing apparatus and an image processing method for processing and outputting image data.

  When a user views an image related to image data on a personal computer or the like, if the image data is in a format suitable for display on the personal computer, the image desired by the user can be displayed with good response. However, there is image data that can only be displayed after image processing. As such image data, for example, there is RAW image data obtained by photographing in a RAW photographing mode which is currently common with a single-lens reflex digital camera. In order to display an image (RAW image) related to RAW image data on a personal computer, it is necessary to perform image processing including a number of processing steps in order to generate a format that can be displayed on a personal computer such as the BMP format. Such image processing is generally called RAW development.

  RAW image data has different preferred image processing contents for each camera vendor, and is recorded in a format unique to each camera vendor. Therefore, each camera vendor distributes a codec for image processing, and the user installs this codec in a personal computer and uses it. For example, when a RAW image is displayed by image browsing software, RAW development is performed using an installed codec corresponding to the RAW image data to display the RAW image.

  In order to improve the response of the user operation, the image browsing software does not issue a RAW development request for the entire image, but issues a partial area development request (partial development request) such as one line to the codec. There is something to issue. In this way, when the user cancels the display of the RAW image, an attempt is made to immediately stop the RAW development processing and reduce wasteful consumption of the CPU and memory.

  As another technique for improving the response to the user operation, there is also an image processing apparatus that temporarily displays a thumbnail image with a low resolution when image processing takes time and the processing result cannot be returned to the user at high speed. Exists (see Patent Document 1).

JP 2002-374482 A

  However, when the codec cannot perform partial development, it is impossible to return control to a partial development request from the image browsing software at a high speed, which hinders user input and results in an image processing apparatus with poor response. There is. Whether or not the partial development can be performed depends on the content of the image processing algorithm of the RAW development processing. That is, depending on the image processing algorithm, there may be a process performed by referring to information of the entire image. If a processing block that performs processing with reference to the information for the entire image area is located at the latter stage of the processing flow, it waits for processing in the processing block that refers to the information for the entire image area even if partial development processing is performed. In the end, the result cannot be output at high speed. Further, even if an image having a low resolution is returned once as in Patent Document 1, it is not a display result desired by the user, and the result of RAW development is required immediately, so the same problem exists.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus that is highly responsive to user input even when image processing takes a long time. .

  The image processing apparatus of the present invention stores first image data output after image processing, and second image data that is lower resolution and image processed than the first image data. Means for requesting output of the first image data stored in the storage means, output control means for making an output request for each partial area, and receiving an output request from the output control means And image processing means for performing image processing on the entire image of the first image data stored in the storage means, wherein the image processing means responds to an output request from the output control means. Until the image processing for the entire image of the first image data is completed, the second image data corresponding to the partial area requested to be output is output, and the image processing for the entire image of the first image data is completed. After that, the first image data subjected to image processing corresponding to the requested partial area is output, and the output control means outputs the image data corresponding to the partial area from the image processing means. Thereafter, if there is no request from the outside, an output request for the next partial area is made.

  According to the present invention, when there is a request from the outside during the execution of the image processing of the first image data, it is possible to cancel the unnecessary image processing and release resources, and to respond to the user input. Improves.

It is a figure which shows the structural example of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the image output process in 1st Embodiment. 10 is a flowchart illustrating a development result output process according to the second embodiment. It is a flowchart which shows the image output process in 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present invention.
In FIG. 1, reference numeral 101 denotes a control unit that controls the entire computer apparatus 100 that realizes the image processing apparatus according to the present embodiment. The control unit 101 includes an image processing unit 114 that performs predetermined image processing on the image data, and an output control unit 115 that controls output (display) of an image related to the image data. The control unit 101 is a CPU (Central Processing Unit), for example, and realizes the functions of the image processing unit 114 and the output control unit 115 by executing a processing program stored in the RAM 103.

  A ROM (Read Only Memory) 102 stores programs and parameters that do not need to be changed. A RAM (Random Access Memory) 103 temporarily stores programs and data supplied from an external device or the like. Reference numeral 104 denotes an external storage device. The external storage device 104 includes a hard disk and a memory card that are fixedly installed in the computer device 100. Further, the external storage device 104 includes a flexible disk removable from the computer apparatus 100, an optical disk such as a CD (Compact Disk), a magnetic or optical card, and an IC card.

  Reference numeral 105 denotes an interface (operation input interface) with an input device 109 such as a pointing device or a keyboard that receives data from the user and inputs data. Reference numeral 106 denotes an interface (display interface) with the display 110 for displaying data held by the computer apparatus 100, supplied data, and the like. Reference numeral 107 denotes a network interface for connecting to a network line 111 such as the Internet. Reference numeral 108 denotes an interface (image input interface) with the image input device 112 for inputting image data.

  A RAW file group 113 includes a plurality of RAW image files. In the RAW file group 113, RAW image data previously captured by the user with the image input device 112 is transferred via the image input interface 108, or acquired by the user from the network line 111 and stored as a file in the external storage device 104. Is. In the RAW image file in the present embodiment, RAW image data (first image data) is stored inside the file. Also, low-resolution image data (second image data) that has been developed by the image input device 112 and the like together with the RAW image data is stored inside the file, and the image that has been processed as necessary is a simple image. Is output. The simple image is, for example, a JPEG image. A system bus 116 connects the units 101 to 108 so that they can communicate with each other.

(First embodiment)
FIG. 2 is a flowchart illustrating an example of an image output process according to the first embodiment. The image output process shown in FIG. 2 is executed when the control unit 101 of the image processing apparatus according to the present embodiment shown in FIG. 1 executes a control program stored in the RAM 103. When RAW image data desired to be displayed is selected from the RAW image file group 113 by the user operating an input device 109 such as a pointing device or a keyboard, the image processing apparatus according to the present embodiment starts processing. Then, an image related to the selected image data is displayed by, for example, image browsing software.

  First, in step S <b> 201, the output control unit 115 determines the line (partial region) to be processed as the first line, and issues an output request for the partial region to the image processing unit 114. In step S202, when the image processing unit 114 receives the output request issued from the output control unit 115 in step S201, the image processing unit 114 sets 1 to a variable N held (reserved) in the RAM 103. Next, in step S203, the image processing unit 114 generates a worker thread. Note that the control unit 101 including the image processing unit 114 and the output control unit 115 can perform multi-thread processing. In step S203 and subsequent steps, the control unit 101 simultaneously executes the main thread including the processes of steps S204 to S211 and S221 to S223 and the worker thread 201 including the processes of steps S212 to S218. In the following, the main thread process will be described first, and then the worker thread process will be described.

  In step S <b> 204, the image processing unit 114 refers to the development completion flag held (secured) in the RAM 103 and determines whether or not the value is “1”. When the value of the development completion flag is 1, it indicates that the RAW development processing for the entire image by the worker thread has been completed. When the value is 0, the RAW development processing for the entire image by the worker thread has not been completed. Indicates.

  As a result of the determination in step S204, if the value of the development completion flag is 0 (No in step S204), the process proceeds to step S205. In step S205, the image processing unit 114 sleeps the process of the main thread for a predetermined time (performs a wait process for a predetermined time). The sleep time may be determined appropriately from, for example, the RAW development processing time for the entire image, the operation method of the image processing apparatus, and the like. Subsequently, in step S206, the image processing unit 114 performs development-processed simple image data (second image data) stored in association with the RAW image data (first image data) of the selected RAW image file. The current line (line N) of (image data) is output.

  On the other hand, as a result of the determination in step S204, if the value of the development completion flag is 1 (Yes in step S204), the process proceeds to step S207. In step S207, the image processing unit 114 refers to the image data obtained as a result of the RAW development processing by the worker thread from the RAM 103, and outputs the current line (line N) as the development result.

  In step S208, the output control unit 115 causes the display 110 to display the image data of the line N output from the image processing unit 114 in step S206 or step S207. In step S209, the output control unit 115 determines whether a cancel request by a user operation has been input for the display of the currently processed RAW image. As a result of the determination, if a cancel request is input (Yes), the process proceeds to step S221. If a cancel request is not input (No), the process proceeds to step S210.

  In step 210, the output control unit 115 determines whether or not the current line is the last line of the RAW image. If the result of determination in step S210 is that the current line is the last line of the RAW image (Yes in step S210), the entire image has been displayed, so the output control unit 115 proceeds to step S223. Proceed. On the other hand, as a result of the determination in step S210, if the current line is not the last line of the RAW image (No in step S210), the output control unit 115 advances the process to step S211. In step S211, the output control unit 115 completes the display of the current line N, so that the variable N held in the RAM 103 is incremented by 1 to advance the process to the next line, and the process returns to step S204. In this way, output requests are sequentially issued for each partial area up to the final line of the RAW image, and the above-described processing is repeated.

  As a result of the determination in step S209, in step S221 that proceeds when a cancel request has been input (Yes), the image processing unit 114 cancels the processing related to the selected RAW image data. Is set to 1. Here, it is assumed that the cancel flag is a variable held (secured) in the RAM 403. A value of 1 indicates that a cancel request by a user operation has been input, and a value of 0 indicates a user. Indicates that no cancel request was input by the operation.

  In step S222, the image processing unit 114 performs a loop process waiting for the worker thread to end. When the worker thread ends (when the determination is Yes), the process proceeds to step S223. In step S223, the output control unit 115 releases the resources secured by the main thread, and ends the image output process in the first embodiment.

  In step S203, the worker thread 201 generated by the image processing unit 114 starts RAW development processing for the entire image from step S212. In the present embodiment, the RAW development processing is composed of a plurality of processing blocks as in general RAW development processing, and here is composed of K processing blocks of development processing <1> to <K>. And

  In step 212, image processing unit 114 executes development process <1>. In step S213, the image processing unit 114 refers to a cancel flag indicating that the display of the selected RAW image has been canceled by a user operation. As a result of the determination in step S213, if the value of the cancel flag is 1 (Yes in step S213), there is no need to perform RAW development processing by the worker thread, and the image processing unit 114 performs subsequent RAW development processing. Skip and proceed to step S218. On the other hand, if the result of determination in step S213 is that the value of the cancel flag is 0 (No in step S213), the image processing unit 114 advances the processing to step S214.

  In step S214, image processing unit 114 executes development process <2>. In step S215, the image processing unit 114 refers to the cancel flag as in step S213. As a result of the determination in step S215, if the value of the cancel flag is 1 (Yes in step S215), the image processing unit 114 skips the subsequent RAW development processing and proceeds to step S218. On the other hand, as a result of the determination in step S215, if the value of the cancel flag is 0 (No in step S215), a plurality of development processes (not shown) and determination of the cancel flag are sequentially performed, and then the process proceeds to step S216. And proceed.

  In step S216, the image processing unit 114 executes the development process <K>. In the present embodiment, since the RAW development processing is composed of K processing blocks of development processing <1> to <K>, the processing of development processing <K> executed by the image processing unit 114 in step S216 is finished. At this point, the RAW development process has been completed. In the present embodiment, the development process <K> is a lens distortion correction process for correcting lens distortion. The lens distortion correction process requires a geometric calculation over the entire image. Further, if it is performed at the front stage of the RAW development processing, an error due to geometric calculation processing occurs, and this is accumulated in the subsequent image processing. In step 117, the image processing unit 114 has set the value of the development completion flag to 1 because the RAW development processing has been completed.

  In step S218, the image processing unit 114 completes the RAW development processing, and thus releases all resources secured by the worker thread. In the present embodiment, it is assumed that the development processing result is stored in a storage area secured by the main thread on the RAM 103, and the development processing result is held after the resource is released in step S218. After executing the process of step S218, the image processing unit 114 ends the worker thread.

  According to the first embodiment, even if an image processing algorithm is not capable of partial development, a request for canceling the display of a RAW image by a user operation is received within a predetermined time to release a worker thread and release resources. It can be carried out. Therefore, it is possible to provide an image processing apparatus that is highly responsive to user operations. In addition, since a simple image is used only every predetermined time, a simple image is not used over the entire image, and when there is no cancel request by a user operation, a good development processing result is appreciated in most of the image. be able to.

(Second Embodiment)
In the second embodiment described below, when the development result is output in step S207 of the image output process described above, the image processing unit 114 determines the difference in image characteristics for the boundary portion with the already displayed simple image. The processing to reduce is performed. FIG. 3 is a flowchart illustrating an example of the development result output process in step S207 of the image output process according to the second embodiment.

  When outputting the image of line N (step S207), the image processing unit 114 starts processing from step S301. In step S <b> 301, the image processing unit 114 acquires a line L stored in advance in the RAM 103. Here, the line L indicates the final line of the line output in step S206 and displayed in step S208 in the image output process shown in FIG.

  In step S302, the image processing unit 114 determines whether or not the difference between the line N and the line L, which are the current output lines, is within a predetermined threshold T. If the result of determination in step S302 is that the difference is greater than or equal to the threshold value T (the distance of the line is far), the difference in characteristics at the boundary is not affected, and the development results differ in image characteristics from the simple image. In order to output the line N as it is, the processing proceeds to step S307. In step S307, the image processing unit 114 outputs the development result line N as the development result.

  On the other hand, if the result of determination in step S302 is that the difference is smaller than the threshold T (the distance between the lines is short), if a development result line having a different image characteristic from the simple image is output as it is, the characteristic at the boundary portion is output. The difference in level is visible. For this reason, the process proceeds to step S303 in order to perform the process of making the characteristic step at the boundary portion inconspicuous. In the present embodiment, the threshold T is, for example, 5.

  In step S303, the image processing unit 114 acquires (N-1) to (N-T) lines in the already displayed simple image data. Here, T corresponds to a predetermined low-pass filter (LPF) application range, and in this embodiment, the same value as the threshold value in step S302 is used. In this embodiment, filter processing using LPF is performed in order to diffuse the step of the image characteristic at the boundary portion, but the present invention is not limited to this. For example, when the resolution in the horizontal direction is low in a simple image, a process for reducing the resolution in the horizontal direction may be performed on the development result in order to bring the image characteristics closer.

  In step 304, the image processing unit 114 acquires N lines to (N + T) lines in the development result data. In step S305, the image processing unit 114 acquires the data of the (N-1) line to (N-T) line of the simplified image and the data of the N line to (N + T) line of the development result data acquired in steps S303 and S304. Is used to perform LPF processing in the vertical direction. From the above, the number of taps of the LPF is 11 taps by adding the central line and the upper and lower 5 lines. The coefficients may be the same as those generally used in the image processing blurring process. In step S306, the image processing unit 114 outputs the result obtained by performing the LPF process as the development result of the line N, and ends the development result output process in step S207.

  According to the second embodiment, the same effect as in the first embodiment can be obtained. In addition, even when the development results displayed after the RAW development processing is finished and the simple images displayed until the end of the development processing have different image characteristics, a good display image in which the difference in the image characteristics of the boundary portion is not noticeable is obtained. can get.

(Third embodiment)
FIG. 4 is a flowchart illustrating an example of image output processing according to the third embodiment. The image output process in the third embodiment shown in FIG. 4 is almost the same as the image output process in the first embodiment shown in FIG. 2, but a process for limiting the output of a simple image is added. . By adding this processing, a simple image is used only in a limited area such as the upper part of the image. For areas that are important when viewing, such as near the center of the image, you can view higher-quality development results. can do.

  Steps S401 to S404, S406 to S412, and S421 to S423 shown in FIG. 4 correspond to steps S201 to S204, S205 to S211, and S221 to S223, respectively, shown in FIG. Description is omitted. Also, the worker thread 401 (steps S413 to S419) shown in FIG. 4 is the same as the worker thread 201 (steps S212 to S218) shown in FIG.

  In step S405 of the image output process shown in FIG. 4, the image processing unit 114 determines whether or not the current line N is smaller than a predetermined line. For example, if the RAW image to be processed is about 3000 lines in the vertical direction and if it is about 30 lines at the top, it is an area of about 1% of the edge of the image. Absent. Therefore, for example, if the RAW image to be processed is about 3000 lines in the vertical direction, the predetermined line may be 30.

  If the result of determination in step S405 is that the current line N is smaller than the predetermined line (Yes in step S405), there is no problem for viewing even if a simple image is used. The process proceeds to the process to output. On the other hand, if the result of determination in step S405 is that the current line N is greater than or equal to a predetermined line (No in step S405), the line is an important line for viewing. The process proceeds to S404.

  According to the third embodiment, since a simple image is not used for an area important for viewing, an image processing apparatus capable of displaying an image with sufficient image quality to be viewed by a user is provided. it can.

  In the above-described embodiment, the RAW development processing is described as an example of the content of the image processing. However, the present invention is not limited to this, and other time-consuming image processing may be used. For example, if noise removal processing is taken as an example, a high-speed simple noise processing execution result may be output as a simple image while performing time-consuming high-performance noise processing.

(Other embodiments of the present invention)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

101 control unit, 103 RAM, 104 external storage device, 113 RAW file group, 114 image processing unit, 105 output control unit

Claims (10)

Storage means for storing first image data that is output after image processing, and second image data that is lower in resolution than the first image data and is image-processed;
An output control means for making an output request for each partial area when requesting output of the first image data stored in the storage means;
Image processing means for performing image processing on the entire image area of the first image data stored in the storage means in response to receiving an output request from the output control means,
In response to the output request from the output control means, the image processing means outputs the second image corresponding to the partial area requested to be output until image processing for the entire image of the first image data is completed. Output data, and after the image processing for the entire image of the first image data is completed, output the first image data subjected to image processing corresponding to the partial area requested to be output,
The image processing apparatus according to claim 1, wherein after the image data corresponding to the partial region is output from the image processing unit, the output control unit makes an output request for the next partial region if there is no external request.   2. The image according to claim 1, wherein when the image processing means outputs the second image data corresponding to the partial area requested to be output, the image processing means outputs the second image data after performing a wait process for a predetermined time. Processing equipment.   The image processing means completes image processing for the entire image of the first image data without outputting the second image data corresponding to the partial area requested to be output for a preset area. 3. The image processing apparatus according to claim 1, wherein the first image data subjected to image processing corresponding to the partial area requested to be output is output.   When outputting the first image data subjected to image processing corresponding to the partial area requested to be output, the image processing means performs image processing at a boundary portion with the output second image data. 4. The image processing apparatus according to claim 1, wherein the first image data is subjected to a process of reducing a difference in image characteristics and is output. 5.   The image processing apparatus according to claim 1, wherein the first image data is RAW image data.   6. The image processing apparatus according to claim 5, wherein the image processing performed on the first image data is RAW development processing.   The image processing apparatus according to claim 5, wherein the second image data is a JPEG image related to the first image data.   The image processing according to claim 1, wherein the image processing performed on the first image data includes lens distortion correction processing for correcting lens distortion. apparatus. The first image data output after the image processing and the second image data stored in the storage means for storing the second image data which is lower in resolution than the first image data and subjected to the image processing. An output control step of requesting output for each partial area when requesting output of one image data;
An image processing step for performing image processing on the entire image area of the first image data stored in the storage unit in response to receiving an output request by the output control step;
In the image processing step, in response to the output request in the output control step, the second image corresponding to the partial area requested to be output until image processing for the entire image of the first image data is completed. Output data, and after the image processing for the entire image of the first image data is completed, output the first image data subjected to image processing corresponding to the partial area requested to be output,
In the output control step, after image data corresponding to the partial region is output in the image processing step, an output request for the next partial region is made if there is no external request. . The first image data output after the image processing and the second image data stored in the storage means for storing the second image data which is lower in resolution than the first image data and subjected to the image processing. An output control step for making an output request for each partial area when requesting output of one image data;
In response to receiving the output request in the output control step, the computer executes an image processing step for performing image processing on the entire image area of the first image data stored in the storage unit,
In the image processing step, in response to the output request in the output control step, until the image processing for the entire image area of the first image data is completed, the second area corresponding to the partial area requested to be output. Output image data, and after the image processing for the entire image area of the first image data is completed, output the first image data subjected to image processing corresponding to the partial area requested to be output,
In the output control step, after the image data corresponding to the partial area is output in the image processing step, an output request for the next partial area is made if there is no external request.

Images