CN101426085B - Imaging arrangements and methods therefor - Google Patents

Abstract

Image data is processed to facilitate focusing and/or optical correction. According to an example embodiment of the present invention, an imaging arrangement collects light data corresponding to light passing through a particular focal plane. The light data is collected using an approach that facilitates the determination of the direction from which various portions of the light incident upon a portion of the focal plane emanate from. Using this directional information in connection with value of the light as detected by photosensors, an image represented by the light is selectively focused and/or corrected.

Description

Imaging device and method thereof

The patent application for this invention is an invention patent with the international application number PCT/US2005/035189, the international application date is September 30, 2005, the application number entering the Chinese national phase is 200580039822.X, and the name is "imaging device and its method" Divisional application of the application.

Related Patent Documents

This patent document claims pursuant to 35 U.S.C. §119(e) to U.S. Provisional Patent Applications Serial No. 60/615,179 filed October 1, 2004, and Serial No. 60/647,492 filed January 27, 2005 priority of the U.S. Provisional Patent Application of , both of which are hereby incorporated by reference in their entirety.

technical field

The present invention relates generally to imaging applications, and more particularly to processing image data to focus and/or correct the image data.

Background technique

Imaging applications such as those involving cameras, video cameras, microscopes, telescopes, etc. are generally limited by the amount of light collected. That is, most imaging devices do not record most information about the distribution of light entering the device. For example, conventional cameras such as digital still cameras and video cameras do not record most information on the distribution of light entering from the outside. In these devices, the collected light often cannot be manipulated in various ways, such as focusing at different depths (distances from the imaging device), correcting for lens aberrations, or controlling the viewing angle.

For still imaging applications, a typical imaging device capturing a particular scene is typically focused on a target or object in the scene, while other portions of the scene fall out of focus. A similar problem exists for video imaging applications, where the image acquisition used in the video application cannot capture the scene in focus.

Many imaging applications suffer from aberrations of the devices (lenses) used to collect light. Such aberrations may include, for example, spherical aberration, chromatic aberration, distortion, curvature of field, oblique astigmatism, and coma. Correction for aberrations generally involves the use of corrective optics, which tend to add bulk, expense and weight to the imaging device. In some applications, such as camera phones and security cameras, that benefit from small optics, the physical constraints associated with the application do not make it desirable to include additional optics.

The difficulties associated with the above have presented challenges to image applications, including those involving the acquisition and alteration of digital images.

Contents of the invention

The present invention is directed to overcoming the above-mentioned challenges and others associated with imaging devices and their implementation. The invention is exemplified in many implementations and applications, some of which are outlined below.

According to an example embodiment of the invention, the light is detected with directional information characterizing the light being detected. The direction information is used with the detected light to generate a virtual image corresponding to one or both of the refocused image and the corrected image.

According to another example embodiment of the present invention, two or more objects at different depths of focus in a scene are imaged, wherein portions of the scene corresponding to the respective objects are imaged at different focal planes. Light from a scene is focused on a physical focal plane and detected together with information characterizing the direction from which the light reaches a specific location on the physical focal plane. For at least one object at a depth of field that is not in focus on the physical focal plane, a virtual focal plane is determined that is different from the physical focal plane. Using the detected light and its directional properties, the portion of the light corresponding to the in-focus image of the at least one object on the virtual focal plane is acquired and accumulated to form a virtual in-focus image of the at least one object.

According to yet another example embodiment of the present invention, the scene is digitally imaged. The light from the scene transmitted to different locations on the focal plane is detected, and the angle of incidence of the light detected at different locations on the focal plane is detected. The depth of field of the portion of the scene from which the detected light originates is detected and used together with the determined angle of incidence to digitally rearrange the detected light. Depending on the application, realignment includes refocusing and/or correcting lens aberrations.

The above summary of the present invention is not intended to describe each illustrated embodiment, or every implementation, of the present invention. The Figures and Detailed Description that follow more particularly exemplify these embodiments.

Description of drawings

A more complete understanding of the invention may be obtained by considering the following detailed description of various embodiments of the invention in connection with the accompanying drawings, in which:

Figure 1 is a light capture and processing device according to an example embodiment of the invention;

FIG. 2 is an optical imaging device according to another example embodiment of the present invention;

3 is a process flow diagram of image processing according to yet another exemplary embodiment of the present invention;

FIG. 4 is a process flow diagram for generating a preview image according to another example embodiment of the present invention;

5 is a process flow diagram for processing and compressing image data according to yet another example embodiment of the present invention;

FIG. 6 is a process flow diagram for image synthesis according to another example embodiment of the present invention;

FIG. 7 is a process flow diagram for image refocusing according to yet another example embodiment of the present invention;

8 is a flow diagram of a process for extending depth of field in an image according to another example embodiment of the invention;

9 is a process flow diagram of another method for extending depth of field in an image according to yet another example embodiment of the present invention;

Figure 10 illustrates an example method of splitting rays according to another example embodiment of the present invention;

11 illustrates a method of mapping sensor pixel positions to light rays in L(u, v, s, t) space with respect to acquired data, according to yet another example embodiment of the present invention;

Figure 12 shows several images refocused at different depths according to another example embodiment of the invention;

Figure 13A shows a 2D imaging configuration according to yet another example embodiment of the invention;

FIG. 13B illustrates cones of rays from one 3D point totaling one pixel, according to another example embodiment of the invention;

14A-14C illustrate a method of calculating images with different depths of field according to yet another example embodiment of the present invention;

15 illustrates a method of tracing rays from a 3D point on a virtual film plane according to another example embodiment of the present invention;

FIG. 16 illustrates a method of obtaining light values according to yet another exemplary embodiment of the present invention;

Figure 17A shows an idealized 512x512 photo according to another example embodiment of the invention;

Figure 17B shows an image produced with an f/2 bi-convex spherical lens according to yet another example embodiment of the present invention;

17C shows an image calculated using an image correction method according to another example embodiment of the present invention;

Figures 18A-18C illustrate tracking common rays in a color imaging system according to yet another example embodiment of the present invention;

19A-19F illustrate a method of implementing a mosaic array according to another example embodiment of the present invention;

20 is a process flow diagram of a calculation method for refocusing in the Fourier domain according to yet another exemplary embodiment of the present invention;

FIG. 21A shows a triangular filtering method according to another example embodiment of the present invention;

Fig. 21 B shows the Fourier transform of the triangular filtering method according to yet another example embodiment of the present invention;

22 is a flowchart of a method for refocusing in the frequency domain according to another exemplary embodiment of the present invention;

Figure 23 illustrates a collection of rays passing through a desired focal point according to yet another example embodiment of the present invention;

24A-B show different views of a portion of a microlens array according to another example embodiment of the invention;

Figure 24C shows an image appearing on a light sensor according to yet another example embodiment of the invention;

Figure 25 illustrates an example embodiment of the invention, wherein a virtual image is computed as if it appeared on a virtual film;

FIG. 26 illustrates a method of manipulating a virtual lens plane according to another example embodiment of the present invention;

Figure 27 shows that a virtual film can take any shape according to yet another example embodiment of the present invention;

FIG. 28 illustrates an imaging device according to another example embodiment of the present invention;

29 is a process flow diagram for precomputing a database of weights associated with individual output image pixels and individual light sensor values in accordance with yet another example embodiment of the present invention;

30 is a flowchart of a process for calculating an output image using a weight database according to another example embodiment of the present invention;

31A-D illustrate various scalar functions selectively implemented as virtual aperture functions according to yet another example embodiment of the invention;

FIG. 32 shows a virtual aperture function varying pixel by pixel according to another example embodiment of the present invention;

33 is a process flow diagram of a user selecting an area of an output image, editing an image portion, and saving the output image according to yet another example embodiment of the present invention;

34 is a flowchart for extending depth of field in an image according to another example embodiment of the invention; and

35 is a process flow diagram for computing a refocused image from received light sensor data according to yet another example embodiment of the invention.

While the invention is susceptible to various modifications and other forms, details thereof are shown by way of example in the drawings and will be described in detail. It should be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the contrary, the invention covers all modifications, equivalents and alternatives falling within the spirit and scope of the invention.

A detailed description

The present invention is believed to be useful in a variety of different types of devices, and it has been found to be particularly useful in electro-imaging devices and applications. While the invention is not necessarily limited to this application, various aspects of the invention can be appreciated by discussing various examples using the context.

According to an example embodiment of the present invention, a four-dimensional (4D) light field (eg, along each ray propagating in a region such as free space) is detected using a method involving determining the amount and direction of light reaching a sensor located at a focal plane. The two-dimensional position of the light in the focal plane is detected together with information characterizing the direction from which the light reaches a specific position on the plane. Using this method, the directional illumination distribution reaching different locations on the sensor is determined and used to form an image. In various discussions herein, a component or components implemented for sensing and/or measuring a light field are referred to as "light sensors" or "ray sensors".

In one application, a method similar to the above is implemented using an imaging system with optics and sensors that spatially sample the rays incident on the imaging plane, with computations that render images from sets of measured rays in different ways Function. Depending on the implementation, different approaches are used in combination or separately to implement optics, sensors, and computing functions. For example, a camera with a lens (optics) that focuses an image on an array of light sensors (multiple sensors) located at the imaging plane can be used to sample the ray space. The output from the photosensor array is used with computational functions (e.g., on a processor internal to and/or external to the camera) to render images, such as by computationally focusing on photographs at different depths or with different depths of field, and/or by computationally correcting for perspective. Mirroring difference to produce a higher quality image.

In another example embodiment, the optics and sensor components of the imaging system direct light onto the sensor elements such that each sensor element senses a set of light that includes light diverging from a particular direction. In many applications, this collection of rays is a bundle of rays localized in space and direction. For many applications, this beam of light is converged into a single geometric ray as the optics and sensor resolution increase. Thus, the various descriptive parts herein refer to the values sensed by the sensor elements as "rays of light" or "rays" or simply "rays", even though in general they are not limited to geometrical rays.

Referring now to the drawings, FIG. 28 illustrates an imaging device 2890 according to another example embodiment of the present invention. The imaging device 2890 includes a main lens 2810, and a light sensor for measuring the value of light reaching different positions on the sensor and coming from different incident directions. In this context, the measurement of the value of light can be achieved by detecting the light reaching different locations on the sensor and a characteristic of the light such as intensity to generate a value.

FIG. 1 shows an imaging system 100 according to another embodiment of the present invention. The imaging system 100 includes an imaging device 190 having a main lens 110 , a microlens array 120 and a photosensor array 130 . In this case, the microlens array 120 and the photosensor array 130 implement a light sensor. Although FIG. 1 shows a particular main lens 110 (single element) and a particular microlens array 120, those skilled in the art will appreciate that various lenses and And/or microlens arrays (currently available or to be developed) can optionally be implemented in a similar manner.

Rays from a single point on object 105 in the imaged scene may reach a single focal point on the focal plane of microlens array 120 . For example, when the imaging point on object 105 is at a distance from the main lens conjugate to the distance from the microlens array to the main lens, distance "d" is approximately equal to distance "s" as shown. A microlens 122 at the converging point splits these rays according to the direction of the light, producing a focused image of the aperture of the main lens 110 on a light sensor beneath the microlens.

Photosensor array 130 detects light incident thereon and produces an output that is processed by one or more different components. In the present application, the output light data is passed to sensor data processing circuitry 140 which uses this data along with positional information about the various light sensors providing the data to generate an image of the scene (eg including objects 105, 106 and 107). The sensor data processing circuit 140 is implemented with, for example, a computer or other processing circuit implemented in a common component (eg, a chip) or in a different component. In one implementation, a portion of sensor data processing circuitry 140 is implemented in imaging device 190 and another portion is implemented in an external computer. Using the detected light (and, for example, the characteristics of the detected light) and the known direction of incidence of the light reaching the microlens array (as calculated using the known positions of the individual photosensors), the sensor data processing circuitry 140 selectively reproduces the Focusing and/or correcting light data (where reimaging can be corrected). Various methods of processing detected light data are described in detail below with or without reference to other figures. These methods may optionally be implemented with sensor data processing circuitry 140 consistent with the above.

Different parts of imaging system 100 are selectively implemented in common or separate physical devices depending on the particular application. For example, microlens array 120 and photosensor array 130 may be combined into a common device 180 when implemented with various applications. In some applications, microlens array 120 and photosensor array 130 are coupled together on a common chip or other circuitry. When implemented with a handheld device such as a camera-like device, the main lens 110, microlens array 120 and photosensor array 130 are selectively combined into a common imaging device 190 integrated with the handheld device. Furthermore, certain applications involve implementing some or all of the sensor data processing circuitry 140 in a common circuit arrangement (eg, on a common chip) with the photosensor array 130 .

In some applications, imaging device 100 includes preview device 150 for presenting a preview image to a user capturing the image. The preview device is communicatively coupled to receive image data from photosensor array 130 . The preview processor 160 processes the image data to generate a preview image displayed on the preview screen 170 . In some applications, preview processor 160 is implemented on a common chip and/or in common circuitry with image sensor 180 . In the application where the sensor data processing circuit 140 is implemented together with the photosensor array 130 as described above, the preview processor 160 is selectively implemented with the sensor data processing circuit 140, wherein part or all of the image data collected by the photosensor array 130 is used for Generate a preview image.

The preview image may be generated using relatively less computational power and/or less data than was used to generate the final image. For example, when implemented with a hand-held imaging device such as a camera or cell phone, a preview image without any focus or lens correction is sufficient. As such, it is desirable to implement relatively inexpensive and/or small processing circuitry to generate the preview image. In these applications, the preview processor generates images at relatively low computational cost and/or using less data, for example by using the first extended depth-of-field calculation method described above.

Depending on the application, imaging system 100 can be implemented in various ways. For example, while microlens array 120 is shown as an example with several resolvable microlenses, such arrays are typically implemented using multiple (eg, thousands, millions) of microlenses. Photosensor array 130 typically includes a relatively finer pitch than microlens array 120 , where there are several photosensors for each microlens in microlens array 120 . Furthermore, the microlenses in microlens array 120 and the photosensors in photosensor array 130 are generally positioned such that light propagating through each microlens to the photosensor array does not overlap light propagating through adjacent microlenses.

In a different application, the main lens 110 is translated along its optical axis (horizontally as shown in FIG. . By way of example, a ray from a single point on target 105 is shown for discussion purposes. These rays travel to a single converging point at microlenses 122 on the focal plane of microlens array 120 . The microlenses 122 split these rays according to direction, thereby producing a focused image of the aperture of the main lens 110 on a set of pixels 132 in the pixel array beneath the microlenses. 10 illustrates an example method of splitting light rays such that all light rays diverging from a point on the main lens 1010 and arriving anywhere on the surface of the same microlens (e.g., 1022) are conducted by the microlens to converge at the light sensor ( For example the same point on 1023). The method shown in FIG. 10 can be implemented, for example, in relation to FIG. 1 (i.e. main lens 110 with main lens 1010, microlens array 120 with microlens array 1020, and photosensor array 130 with photosensor array 1030) .

The image formed under a particular microlens in microlens array 122 represents the directional resolution of the system for that location on the imaging plane. In some applications, directional resolution is enhanced by helping to sharpen the microlens image by focusing the microlens on the principal plane of the primary lens. In some applications, the microlenses are at least two orders of magnitude smaller than the spacing between the microlens array and the main lens 110 . In these applications, the main lens 110 is effective at the optical infinity of the microlenses; to focus the microlenses, the photosensor array 130 is placed in a plane at the focal depth of the microlenses.

The separation "s" between the main lens 110 and the microlens array 120 is selected to achieve a sharpened image within the depth of field of the microlenses. In many applications, this spacing is accurate to within about Δx _p ·(f _m /Δx _m ), where Δx _p is the width of the sensor pixel, f _m is the focal depth of the microlens and Δx _m is the width of the microlens. In one particular application, Δx _p is about 9 microns, f _m is about 500 microns and Δx _m is about 125 microns, and the spacing between microlens array 120 and photosensor array 130 is exactly about 36 microns.

Microlens array 120 is implemented using one or more of individual microlenses and configurations thereof. In an example embodiment, a microlens plane with potentially spatially varying properties is used as the microlens array 120 . For example, microlens arrays may include lenses that are uniform and/or non-uniform, square or non-square in extension, regularly distributed or randomly distributed, and in repeatable or non-repeatable patterns, and optionally masked portions . The microlenses themselves can be convex, non-convex, or have any shape to achieve the desired physical direction of light, and the shape can vary from one microlens to another in a plane. Different distributions and lens shapes can be combined optionally. These various embodiments provide sampling patterns that are spatially higher (respectively lower angles) in some areas of the array and higher angle (respectively lower spatial) in other areas. One use of such data is to facilitate interpolation to match the desired spatial and angular resolution in 4D space.

Figure 24A shows a view (line of sight perpendicular to plane) of a portion of a microlens array according to another example embodiment of the present invention. The microlenses are square and regularly distributed in the array.

FIG. 24B shows a view of a portion of a microlens array according to another example embodiment of the present invention. The plane distribution of the microlenses is irregular or non-repetitive, and the microlenses are of arbitrary shape.

Fig. 24C shows an image appearing on a photosensor by using a profile with a convex profile as shown in Fig. 24A and a main lens with a circular aperture, related to another example embodiment of the present invention.

In other example embodiments, a conventional mosaic of larger and smaller microlenses is used. In one implementation, the resulting light sensor data is interpolated to provide uniform sampling with maximum spatial and angular resolution of the one or more microlenses in the mosaic.

Figures 19A-19F illustrate a method of implementing a mosaic array such as that described above in connection with one or more example embodiments of the present invention. FIG. 19A is a top plan view showing example relative sizes and arrangements of a plurality of microlenses. Figure 19B is a view of the shape of the image formed on the photosensor array after projection through the various microlenses in Figure 19A. Figure 19C is a cross-sectional view of the array in Figure 19A, showing the microlenses having the same f-number and their focal points on the same plane. This requires smaller microlenses to be placed closer to the focal plane than larger microlenses. This results in a large, non-overlapping main lens image appearing behind each microlens and focused on the light sensor placed at the plane containing the focal point.

Figure 19D shows a cross-sectional view of the microlenses shown in Figures 19A and 19C implemented in a complete imaging device comprising a master lens 1910, a mosaic microlens array 1920 and a light sensor device 1930 according to another example embodiment of the present invention. Note that although the figures show several microlenses and pixels per microlens, the actual number of microlenses and pixels can be selected in different ways, such as by determining the resolution requirements for a given application and implementing the appropriate number.

Figure 19E is a Cartesian ray diagram representing the ray space starting at u on the main lens 1910 and ending at s on the microlens array 1920 (although the ray space is in 4D, the ray space is shown as 2D for clarity ). The set of rays summed by the various light sensors (labeled A-P) shown in Figure 19D is shown in the Cartesian ray diagram in Figure 19E. In a full 4D light space, individual light sensors are combined with 4D light boxes. Compared with the light sensor under the smaller microlens, the light sensor 4D box under the larger microlens has half the width (twice the resolution) in the (u, v) direction axis, and the (x, y ) spatial axis has twice the width (half the resolution).

In another example embodiment, the light sensor values are interpolated into a regular grid such that the resolution in all axes matches the maximum resolution in all axes. Figure 19F illustrates this approach, where the ray boxes representing individual light sensor values are segmented by interpolating nearby box values. In the 2D ray space shown, each box is divided into two, but in 4D space, each box is divided into four (one along each of its two longer sides). In some implementations, interpolated values are calculated by analyzing neighboring values. In another embodiment, the interpolation is implemented as a weighted sum of the values of the initial, undivided boxes of the neighborhood of the expected value.

In some applications, weighting is achieved in a manner dependent on a decision function based on neighborhood medians. For example, the weights may be interpolated along the axes least likely to contain edges in the 4D function space. The likelihood of edges near this value can be estimated from the magnitude of the gradient of the function's values at these locations and the Laplacian component of the function.

In another example embodiment, individual microlenses (eg, in array 1920 in FIG. 19D or the like) are tilted inward so that their optical axes are all centered on the aperture of the main lens. This approach reduces aberrations in the image formed under the microlenses towards the edge of the array.

Referring again to FIG. 1 , the aperture size (eg, the effective size of the opening in the lens) of main lens 110 and microlenses in microlens array 120 is also selected to suit the particular application in which imaging arrangement 100 is implemented. In many applications, the relative aperture size is chosen such that the captured images are as large as possible without overlapping (ie, so that light does not undesirably overlap adjacent light sensors). This method works by matching the f-stops of the main lens and the microlens (focal ratio, ie the ratio of the f-stop of the lens to the effective focal length). In this case, the effective focal length of the main lens 110 expressed in f-stop is a ratio between the diameter of the main lens aperture and the distance “s” between the main lens 110 and the microlens array 120 . In applications where the principal plane of master lens 110 is translated relative to the plane in which microlens array 120 resides, the aperture of the master lens is selectively altered to preserve the ratio and thus size of the image formed under each microlens in the microlens array. In some applications, different main lens aperture shapes, such as a square aperture, are used to achieve the desired (eg, effective) image array packet on the light sensor surface under the microlens array.

The following discussion relates to general applications of the imaging device 100 of FIG. 1 in relation to one or more example embodiments of the present invention. Consider the two-plane (two-plane) light field "L" in the imaging device 100, L(u, v, s, t) represents the intersection with the main lens 110 at (u, v) and the microlens array 12 Light propagated by a ray that intersects the plane at (s, t). Assuming ideal microlenses in microlens array 120 and ideal photosensors (e.g., pixels) on an aligned grid in photosensor array 130, all light transmitted to the photosensors is also transmitted through its square mother microlens in microlens array 120, and transmitted through the conjugate square of the photosensor on the main lens 110 . These two square areas on the main lens 110 and the microlens designate a small four-dimensional box in the light field, and the light sensor measures the total amount of light in the collection of rays represented by this box. Accordingly, each photosensor detects such a four-dimensional box in the light field, so the light field detected by the photosensor array 130 is a box-filtered, linear sampling of L(u, v, s, t).

FIG. 11 illustrates a method of mapping sensor pixel positions to light rays in L(u, v, s, t) space with respect to acquired data, according to another example embodiment of the present invention. The approach shown in FIG. 11 and discussed herein may be used, for example, in FIG. 1 , where each photosensor in photosensor array 130 corresponds to a sensor pixel. The bottom right image 1170 is a downsample of the raw data read from the light sensor (light sensor) 1130 and has the circled image 1150 formed under a circular microlens. The lower left image 1180 is a magnified representation of the portion of raw data surrounding the circled microlens image 1150 with one light sensor value 1140 circled within the microlens. Since this circular image 1150 is an image of the lens aperture, the location of the selected pixel within the disk provides the (u,v) coordinates of the origin of the ray 110 shown on the master lens. The position of the microlens image 1150 within the sensor image 1170 provides the (x, y) coordinates of the ray 1120 .

While the mapping of sensor elements to light rays is discussed with respect to the figures (and other example embodiments), the values associated with each sensor element are selectively represented by the value of the set of light rays transmitted through the optics to each particular sensor element . Thus in the context of FIG. 1 , individual photosensors in the photosensor array may be implemented to provide a value representative of the set of light rays transmitted through main lens 110 and microlens array 120 to the photosensor. That is, each photosensor generates an output in response to light incident on the photosensor, and the position of each photosensor relative to the microlens array 120 is used to provide directional information about the incident light.

In an example embodiment, the resolution of the microlens array 120 is selected to match the desired resolution of the final image in a particular application. The resolution of the photosensor array 130 is chosen such that each microlens covers as many photosensors as necessary to match the desired directional resolution for the application, or the best achievable resolution of the photosensors. As such, the resolution of imaging system 100 (as well as other systems discussed herein) is selectively tuned for a particular application, taking into account factors such as imaging type, cost, complexity, and available equipment for achieving a particular resolution.

Once image data is captured via optics and sensors (eg, using imaging device 190 in FIG. 1 ), various computing functions and devices can be implemented to selectively process the image data. In an example embodiment of the invention, different sets of light sensors capture separate light rays from individual microlenses and communicate information about the captured light rays to a computing component such as a processor. An image of the scene is computed from the set of rays measured.

In the context of FIG. 1 , sensor data processing circuitry 140 is implemented to process image data and compute an image of the scene including objects 105 , 106 and 107 . In some applications, preview device 150 is also implemented to generate a preview image with preview processor 160 , where the preview image is displayed on preview screen 170 . Preview processor 160 is optionally implemented by sensor data processing circuitry 140 , wherein preview images are generated, for example, in a manner consistent with the methods discussed below.

In another embodiment, for each pixel in the image output from the sensor arrangement, the calculation means weights and sums a subset of the measurement beams. Furthermore, the calculation means may analyze and combine the image collection calculated in the above-described manner, for example using an image composition method. While the invention is not necessarily limited to this application, aspects of the invention can be understood through a discussion of several specific example implementations of such computing components.

In connection with various example embodiments, image data processing involves refocusing at least a portion of an image being captured. In some implementations, the output image is generated from a photograph focusing on desired elements of a particular scene. In some implementations, the computed photo is focused at a particular desired depth of the sphere (scene), and the defocus blur increases away from the desired depth like a regular photo. Different depths of focus can be selected to focus on different targets in paired scenes.

Figure 12 shows several images 1200-1240 refocused at different depths, calculated from a single light field measured according to another example embodiment of the invention. The method shown in FIG. 12 may be implemented, for example, using an imaging device such as that shown in FIG. 1 .

13A and 13B illustrate a refocusing method according to another example embodiment of the present invention. The method may be implemented, for example, by computing/processing components of an imaging system such as sensor data processing circuit 140 in FIG. 1 . Each output pixel (eg, 1301 ) from the imaging device corresponds to a three-dimensional (3D) point (eg, 1302 ) on a virtual film plane 1310 . The virtual film plane 1310 is located behind the main lens 1330, where the plane 1310 is optically conjugate to the desired focal plane in the sphere (not shown). That is, the virtual film plane 1310 is located at a location where the film plane is expected to be positioned to capture a simple two-dimensional (2D) image (comparable, for example, to where photo film is positioned for a conventional camera to capture a 2D image). By separating light by direction (eg, using microlens array 120 of FIG. 1 ), light reaching virtual film plane 1310 can be selectively counted. In this way, the value of the output pixel 1301 may be calculated by summing the cones of rays 1320 converged on the corresponding 3D point 1302 . Values for these light rays may be gathered from data collected by light sensor 1350 . For ease of viewing, Figure 13A shows a 2D imaging configuration. In FIG. 13B , the ray cones 1330 from the 3D point 1340 are summed for the same pixel 1301 using the selected global depth of focus when closer to the main lens.

In some implementations, the desired light values do not correspond exactly to the discrete sample locations captured by the light sensors. In some implementations, ray values are estimated as a weighted sum of selected tight sample locations. In some implementations, the weighting method corresponds to a four-dimensional filter kernel that reconstructs a continuous four-dimensional light field from discrete sensor samples. In some implementations, the 4D filter is implemented by a 4D tent function corresponding to a quadrilinear interpolation of the 16 nearest neighbor samples in 4D space.

35 is a flowchart for calculating a refocus image from received light sensor data according to another example embodiment of the present invention. At block 3520 , a set of sub-aperture images is extracted from the light sensor data 3510 , where each sub-aperture image consists of a single pixel under each microlens image of the pixel at the same relative position under its microlens. At block 3530, the sub-aperture image sets are combined to generate the final output image. The sub-aperture images can optionally be translated relative to each other and composited to bring the desired plane into focus.

In another example implementation, darkening associated with pixels near the edges of the output image is mitigated. For example, for pixels near the edges of the output image, some desired light rays are not captured in the measurement light field (they may be beyond the spatial or directional boundaries of an imaging device such as microlens array 120 and photosensor array 130 in FIG. 1 ). For applications where this dimming is undesirable, pixel values are normalized by dividing the value associated with the pixel (eg captured by the light sensor array) by the fraction of light actually found in the measured light field.

As mentioned above, various calculation methods are selected for different applications. The following discussion sets forth a variety of such approaches. In some applications, reference is made to the figures, and in other applications, methods are generally described. In each of these applications, certain methods may be implemented using computational components such as sensor data processing circuit 140 shown in FIG. 1 .

In another example embodiment, the imaging method of each output pixel of a particular imaging system corresponds to a virtual camera model, where the virtual film is arbitrarily and/or selectively rotated or deformed, and the virtual main lens aperture is moved accordingly and as needed. change its size. As an example, FIG. 25 shows an example embodiment in which if a virtual image has appeared behind a virtual lens aperture 2520 of any size on a virtual lens plane 2530 that is allowed to be inconsistent with the physical master lens plane 2510, then it would appear as if it had appeared on a virtual film 2560 Calculate the virtual image as above. The pixel value corresponding to point 2550 on virtual film 2560 is calculated by summing light rays traveling through virtual aperture 2520 and converging at point 2550 , specified according to their intersection point and direction of incidence on light sensor 2570 .

FIG. 26 illustrates a method of manipulating a virtual lens plane according to another example embodiment of the present invention. Virtual lens plane 2630 and/or virtual film plane 2660 are optionally tilted with respect to a physical master lens or other reference. Images computed using this method have a resulting global focal plane that is not parallel to the imaging plane.

In another example embodiment, as illustrated in Figure 27, the virtual film 2560 need not be flat, but may take any shape.

Various approaches involve the selective realization of different apertures. In one example embodiment, the virtual aperture in the plane of the virtual lens is generally a circular aperture, while in other example embodiments the virtual aperture is generally non-circular and/or implemented as multiple distinct regions of any shape. In these or other embodiments, the concept of a "virtual aperture" may be generalized and, in some applications, correspond to a method involving light data processing to correspond to light receivable through a selected "virtual" aperture.

In various embodiments, the virtual aperture method is realized by a predetermined but arbitrary function on the virtual lens plane. 31A-31D illustrate different scalar functions selectively implemented as virtual aperture functions in connection with one or more example embodiments. Various functions include, for example, varying values smoothly (as exemplified in FIG. 31B ), implementing multiple different regions (as exemplified in FIG. 31A ), and taking negative values (as exemplified in FIG. 31D ). To calculate the value of a point on the virtual film, all rays originating from different points on the virtual lens and converging on a point on the virtual film are weighted and summed by the virtual aperture function. In various other embodiments, the final value is calculated by any calculation function that depends on the ray value. For example, the computation function may not correspond to the weighting through the virtual aperture function, but may contain discrete program branches depending on the value of the test function computed on the ray values.

In other example embodiments, the method of computing output pixels may be chosen independently, as may be implemented in conjunction with other embodiments described herein. For example, in an example embodiment, parameters including the orientation of the virtual lens plane and the size of the virtual aperture are varied continuously for each output pixel. In another example, as shown in FIG. 32 , the virtual aperture function used to integrate the rays of each output pixel varies on a pixel-by-pixel basis. In output image 3200 , pixel 3201 uses virtual aperture function 3210 and pixel 3251 uses virtual aperture function 3250 .

In another example embodiment, the virtual aperture function varies on a pixel-by-pixel basis. In a particular embodiment, the function is chosen to block out light from undesired parts of a particular scene, such as undesired objects in the foreground.

In yet another example embodiment, a user interactively selects a virtual aperture parameter, and the light data is processed according to the selection. Figure 33 is a process flow diagram illustrating one such example implementation. In a first block 3310, the process receives data from a light sensor. In block 3320, the user selects an area of the output image; in block 3330, the user selects an image forming method; and in block 3340, the user changes the parameters of the selected method and visually inspects the scene image computed in block 3350 ( such as on a computer monitor). Block 3360 checks to see if the user has finished editing the image portion and returns to block 3330. Block 3370 checks whether the user has finished selecting the image portion to be edited and returns to frame 3320. If editing is complete, block 3380 saves the final edited image.

In another example embodiment, an image with an extended depth of field is computed by simultaneously focusing on more than one object. In one implementation, the depth of field of the output image is extended by conventional pattern imaging with a down-stopped (reduced in size) main lens aperture. For each output pixel, the evaluation is performed using rays that converge on the output pixel through a smaller aperture (on the virtual lens plane) than that used in light sensing.

In one implementation involving the example system 100 shown in FIG. 1 , the depth of field is extended by extracting light sensor values under each lenticule image where each light sensor is placed at the same relative position within each lenticule image. 1, the extension of the depth of field produces an image in which not only the target 105 is in focus (due to the correlation between distances "d" and "s") but also objects such as objects 106 and 107 that may be blurred due to defocus Objects at different depths are also in focus. The method of extending the depth of field combined with optional subsampling of the resulting image is computationally efficient. The method is selectively implemented in applications where noise generated with the image is tolerable, for example where the image is generated for preview purposes (eg for display on preview screen 170 of FIG. 1 ). Figure 4, discussed below, further relates to the method of generating the preview image.

14A and 14B illustrate a method of computing images with different depths of field in conjunction with one or more example embodiments. Figure 14A shows images and close-ups calculated by refocusing. Notice that the face in the close-up is blurred due to the shallow depth of field. The middle row of Figure 14B shows the final image computed with an extended depth-of-field method such as that described in the previous paragraph.

FIG. 8 is a flowchart of another calculation method for extending the depth of field in an image according to another exemplary embodiment of the present invention. In block 810, a set of images refocused at all depths of focus for a particular scene are refocused. At block 820, for each pixel, a pixel is determined from a set of images with the highest local contrast. At block 830, the pixels with the highest local contrast are composed into a final virtual image. Using this approach, a desired signal-to-noise ratio (SNR) can be achieved using a relatively large number of pixels (eg, relative to selecting a single pixel (light sensor) for each microlens in a microlens array). Referring to Figure 14C, the example image shown was generated using a method similar to that described in connection with Figure 8, and exhibits relatively low image noise.

In an alternative embodiment, according to the distance between the virtual film plane of each refocused image and the principal plane of the main lens through which light of the image travels to the virtual film plane, the minimum set of refocused images to be calculated is defined as follows. The minimum distance is set at the focal length of the main lens, while the maximum distance is set at the conjugate depth of the closest object in the scene. The spacing between the individual virtual film planes is no greater than Δx _m f/A, where Δx _m is the width of the microlens, f is the spacing between the main lens and the microlens array, and A is the width of the lens aperture.

In another example embodiment, multiple refocused images are combined to produce an extended depth-of-field image at each final pixel to preserve the best focused pixel in any one of the refocused image sets. In another embodiment, pixels to be retained are selected by enhancing local contrast and coherence with neighboring pixels. For general information on imaging and specific information on imaging methods involving enhanced local contrast, reference may be made to ACM Transactions on Graphics, Vol. 23, No. 3, 2004, pp. 292-300, A. Agarwala, M. Dontcheva, M. Agrawala , S. Drucker, A. Colburn, B. Curless, D. Salesin, M. Cohen, "Interactive Digital Photomontage," which is hereby incorporated by application in its entirety.

In another example embodiment of the invention, the extended depth-of-field image is calculated as follows. For each output image pixel, refocus calculations are performed at the pixel to focus at different depths. At each depth, a measure of the uniformity of the collected rays is computed. The depth that yields the (relatively) greatest uniformity is chosen and maintained for that pixel value. Using this method, when an image pixel is in focus, all its light rays originate from the same point in the scene, and thus are likely to be of similar color and intensity.

Although the uniformity metric can be defined in different ways, for many applications the following uniformity metric is used: for each color component of each ray, according to the corresponding The color component computes the variance of the color intensity. Sum all these variances and take the uniformity as the inverse of this sum.

FIG. 34 is a flowchart of extending the depth of field in an image according to another example embodiment of the present invention. At block 3410, pixels are selected in the virtual image to be computed. At block 3420, the pixel is refocused at multiple focal lengths, and the uniformity of the combined rays refocused at each depth is calculated. At block 3430, the refocused pixel value associated with the greatest uniformity of the combined rays is retained as the final output image pixel value. The process continues at block 3440 until all pixels are processed.

In another example embodiment, the above-described process is tuned such that the selection of the final pixel value takes into account the uniformity of neighboring pixel values, and associated rays that are combined to compute those pixel values.

In yet another example embodiment of the invention, the depth of field is extended by focusing each pixel at the depth of the closest object in that direction. Figure 9 is a flowchart for extending depth of field in an image, according to one such example implementation. At block 910, pixels in the final virtual image to be calculated are selected. At block 920, for a ray (or set of rays) transmitted from the selected pixel into the scene through the center of the lens, the depth of the nearest object is estimated.

At block 930, the value of the selected pixel is calculated in the image refocused on the estimated depth. If the additional pixel is processed as expected at block 940 , another pixel is selected at block 910 and the process continues at block 920 with the newly selected pixel. When no additional pixels are processed as intended at block 940, the calculated values for each selected pixel are used to create the final virtual image.

In some embodiments involving depth of field extension, light rays originating from depths closer than the desired object depth are mitigated or eliminated by disregarding light rays originating at depths closer to the lens, such as commonly referred to as "blooming" or "Halo" artifacts. As an example, FIG. 23 shows a set of rays traveling through a desired focal point 2301 of a universe on an object of interest 2310 . Some of these rays are blocked from the main lens by object 2320 and these rays correspond to rays 2340 detected by light sensor 2350 but not considered in calculating the image value of point 2301 . In some embodiments, the resulting pixel values are normalized by dividing by the disregarded fraction of rays. These embodiments may be used alone or in combination with each other, as well as with any other embodiment including those involving extended depth of field.

As described above, light data is processed to focus and/or correct images according to various example embodiments. Various methods involving post-correction methods are described below. In some of these implementations, the ray is obtained by tracing the ray through the actual optical element (such as a lens or lens group) of the optics used in capturing the ray, and mapping the traced ray to the specific light sensor that captures the light. Corrects aberrations. The light data is rearranged using known imperfections exhibited by the optical components and the known positions of the sensors that detect the light.

In a corrective implementation, the gamut of rays contributing to each pixel formed by the idealized optics is calculated for each pixel on the film of the composite print. In one implementation, these rays are computed by tracing rays from the virtual film position through ideal optics back to the universe. FIG. 15 illustrates a method of tracing rays from a 3D point 1501 on a virtual film plane 1510 through an ideally thin main lens 1520 to a global ray cone 1530 in conjunction with one such example implementation. In some implementations, the desired set of rays 1525 may not necessarily correspond to a direction through an actual lens, but may correspond to any set of rays to be weighted and summed to produce a desired image value.

FIG. 16 illustrates a method of obtaining values for light traveling along ideal rays for a particular application in conjunction with another example embodiment. These values are calculated by tracing the desired ideal global ray 1630 through the actual main lens 1650, which is a single element with a spherical interface and used to physically direct the actual global ray to the light sensor 1640 when measuring (detecting) this ray. In this embodiment, rays that would ideally converge at a single 3D point (1601) do not converge, which represents a defect of lenses with spherical interfaces called spherical aberration. Ray sensor 1640 provides individual values for each of the aberrated rays (such as 1651 ) for correcting spherical aberration.

17A-17C show example results of computer simulations using the through-the-lens correction method. The image in Figure 17A is an ideal 512x512 photograph (visible through ideal optics). The image in Figure 17B is an image produced using an actual f/2 bi-convex spherical lens, which suffers from loss of contrast and blurring. The image in Figure 17C is a photograph computed using the image correction method described above, using optics and sensor arrangements that facilitate 10x10 directional (u,v) resolution at each 512x512 microlens.

In another example embodiment of the present invention, chromatic aberration of a main lens used to capture an image is corrected. Chromatic aberration is caused by the divergence of light rays as they are physically guided through an optical component due to differences in physical orientation depending on the wavelength of the light. Consider the wavelength-dependent refractive index of light that occurs in real optical components through which incident rays are traced. In some applications, the individual color components of the system are tracked separately according to the dominant wavelength.

In another example embodiment, as shown in Figure 18A, the individual red, green and blue components that coexist in a color imaging system are tracked separately. The green sphere rays are computationally traced back to the imaging system to generate green rays 1830 and determine where and in what direction they intersect the color light sensor 1810 . Similarly, Figure 18B shows computationally tracing desired blue sphere rays 1820, which are refracted to a greater extent than green rays. Figure 18C shows computationally tracing desired red sphere rays 1830, which are refracted to a smaller extent than green rays. Values for each light are calculated from values from light sensor 1810 using methods such as those described in connection with other example embodiments described herein. The light field values for each ray are integrated to calculate the corrected image value for each particular film pixel. For some applications, chromatic aberration is improved by focusing the individual color channels on a plane where their wavelengths are best focused.

The desired ray may not converge exactly on one of the discrete ray values sampled by the light sensor. In some implementations, the values for these rays are calculated as a function of discrete ray values. In some embodiments, the function corresponds to a weighted sum of discrete ray values in the desired ray domain. In some embodiments, the weighted sum corresponds to a 4D convolution of discrete sampled values with a predetermined convolution kernel function. In other embodiments, the weighting may correspond to quad-linear interpolation based on 16 nearest neighbors. In further embodiments, the weighting may correspond to cubic or bicubic interpolation from the 16 nearest neighbors.

It is worth noting that the example correction process has been described in terms of ray tracing for conceptual simplicity; various other methods can also be implemented with correction. In one embodiment, for each desired output pixel, the set of contributing light sensor values and their relative weights are precomputed. As mentioned above, these weights are properties of many factors including the optics, the sensor, the desired set of rays to be weighted and summed for each output pixel, and the desired light field reconstruction filter. These weights are optionally precomputed using ray tracing, and stored. A corrected image is formed by weighting and summing the appropriate sensed light field values for each output pixel.

Figures 29 and 30 illustrate other example embodiments used in conjunction with the calibration methods described above. 29 is a process flow diagram for precomputing a database of weights associated with light (light) sensors and output pixel values associated with each light sensor. In the first two blocks 2910 and 2920, for a desired image formation process, a data set (e.g., in a database) is received (for example, in a database) consisting of the ideal global set of rays to be summed to produce an output image pixel value (for each output image pixel), and A specification for the actual main lens optics used to physically conduct light to the light sensor. At block 2925, an image pixel is selected. For the output value of the pixel, at block 2930 the global ray correlation set is computationally traced through the virtual representation of the master lens optics to the ray sensor. This results in a set of weights applied to the individual light sensor values to compute the output pixel value. These values are stored in the output database in block 2940. Block 2950 checks to see if all pixels have been processed, and returns to block 2925 if not. If all pixels have been processed, final block 2960 secures the completed database.

FIG. 30 is a flowchart of a process of calculating an output image using the weight database calculated by the process as in FIG. 29 . In blocks 3010 and 3020, the process receives the database and passes through the collection of light sensor values captured by the master lens optics used in computing the database. At block 3025, pixels in the output image are selected such that their final image values can be calculated. For the selected pixel, block 3030 uses the database to find the contributing light sensor set and its weight. At block 3040, the individual sensor values given in 3020 are weighted and summed for the image pixel value. At block 3050, a check is made to see if all image pixels have been processed. If not, the process returns to block 3025, and if so, the output image is saved at block 3060.

In various example implementations, the light data is processed in the frequency domain using certain methods involving refocus calculation methods operating in the Fourier domain. Figure 20 is a flowchart illustrating one such method in connection with another example embodiment. The input to the algorithm is a discrete 4D light field 2010 denoted L(s,t,u,v), which represents rays starting at (u,v) on the master lens and ending at (s,t) on the microlens plane (eg from the main lens 110 of FIG. 1 and terminate at the plane of the microlens array 120). The first step is to calculate the discrete 4D Fourier transform 2020 of the light field. The 4D Fourier transform value on (k _s , k _t , k _u , k _v ), called M(k _s , k _t , k _u , k _v ), is defined by the following equation:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; \&Integral;</span><span\; class="notranslate">\; \&Integral;</span><span\; class="notranslate">\; \&Integral;</span><span\; class="notranslate">\; \&Integral;</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}\sqrt{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; sk</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; tk</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; uk</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; vk</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; dsdtdudv</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Wherein the exp function is an exponential function, exp(x)=e ^x . In some embodiments, the discrete light field is sampled on a rectilinear grid in 4D space, and the Fourier transform is computed using a Fast Fourier Transform (FFT) algorithm.

The next step, performed once for each depth of the desired refocused image, is to extract the appropriate 2D slices 2030 of the 4D Fourier transform, and to compute the inverse 2D Fourier transform of the extracted slices, which are photographs 2040 focused at different depths. For a function G(k _x , _ky ), the 2D inverse Fourier transform g(x, y) is defined by the following equation:

$\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; \&Integral;</span><span\; class="notranslate">\; \&Integral;</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}\sqrt{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; xk</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; yk</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; dk</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; dk</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

The values extracted on the 2D slice are determined according to the desired depth of focus. Consider the conjugate plane (on the image side of the lens) of the desired global focal plane, when the distance between this conjugate plane and the main lens is D and the distance between the microlens plane and the main lens is F, then the coordinates (k The value of the extracted 2D slice in _x , _ky ) is given by

G(k _x , k _y )=1/F ² M(k _x (1-D/F), k _y (1-D/F), k _x D/F, k _y D/F) (3 )

Artifacts due to discretization, resampling and Fourier transform are selectively improved using different methods. During normal signal processing, when a signal is sampled, it is replicated periodically in a dual domain. When this sampled signal is reconstructed by convolution, it is multiplied in the dual domain with the Fourier transform of the convolution filter. In this way, the original, central replica (replica) is separated, thereby eliminating all other replicas. The desired filter is the 4D sinc function, sinc(s)sinc(t)sinc(u)sinc(v), where sinc(x)=sin(πx)/(πx); however, this function has infinite extension.

In various approaches, finite extension filters are used for frequency domain processing; such filters can exhibit drawbacks that can be selectively mitigated. Figure 21A illustrates these deficiencies with respect to a particular 1D filter, in conjunction with the corresponding discussion below relating to mitigation or such deficiencies. Figure 21A shows a triangular filter implemented with 1D linear interpolation (or as the basis for a 4D quad-linear filter). Figure 21B shows the Fourier transform of the triangular filter method, which is non-unity in band-limit (see 2010), and tapers off to smaller fractional values with increasing frequency. Furthermore, the filter is not truly band-limited, it includes energy at frequencies outside the desired rejection band (2020).

The first drawback described above results in "rolloff artifacts" that can cause darkening of computed photo borders. The attenuation of the filter spectrum with increasing frequency means that the spatial light field values modulated by this spectrum also "tilt" towards the boundary to fractional values.

The second drawback described above concerns aliasing artifacts in the computed photographs related to energy at frequencies above the band limit. Nonzero energies for extensions beyond the band limit indicate that periodic replicas are not fully eliminated, leading to two confounds. First, replicas that appear parallel to the slicing plane appear as 2D replicas encroaching on the boundaries of the final photo. Second, replicas positioned perpendicular to this plane are projected and summed onto the image plane, creating ghosting and loss of contrast.

In an example embodiment, the correction for the aforementioned tilt-type defects is eliminated by multiplying the input optical field by the inverse of the filter's inverse Fourier spectrum to counteract the effects introduced during the resampling process. In this example implementation, the multiplication is performed prior to the 4D Fourier transform in the preprocessing step of the algorithm. While it corrects for tilt errors, premultiplication weights the energy of the lightfield near its boundaries, maximizing the energy that overlaps back to the desired field of view as aliasing.

Three methods of suppressing aliasing artifacts—oversampling, superior filtering, and zero-padding—are used alone or in combination in various example embodiments described below. Oversampling to extract 2D slices increases the replication period in the spatial domain. This means that the smaller energy in the tail of the in-plane replica falls within the boundaries of the final photo. Increasing the sampling rate in one domain results in an increase in the field of view in the other domain. Confounding energy in neighboring replicas falls in these peripheral regions and is pruned out to isolate the initial central image of interest.

Another approach to aliasing mitigation involves a finite extension filter that approximates as closely as possible a perfect frequency spectrum (which can be represented by using an ideal filter). In an example implementation, the 4D Kaiser-Bessel separable function kb4(s,t,u,v)=kb(s)kb(t)kb(u)kb(v) is used as filter, where

$\mathrm{<span\; class="notranslate">\; kb</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; \&CenterDot;</span>\sqrt{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

In this equation, _I0 is the standard zero-order corrected Kaiser-Bessel function of the first kind, W is the width of the desired filter, and P is a parameter that depends on W. In this example embodiment, the values of W are 5, 4.5, 4.0, 3.5, 3.0, 2.5, 2.0, and 1.5, and the values of P are 7.4302, 6.6291, 5.7567, 4.9107, 4.2054, 3.3800, 2.3934, and 1.9980, respectively. For general information on obfuscation, and for specific information on methods for mitigating obfuscation in conjunction with one or more example embodiments of the present invention, reference may be made to JL Jackson, CH Meyer, 1997, IEEE Transactions on Medical Imaging, Vol. DGNishimura and A. Macovski, "Selection of convolution function for Fourierinversion using gridding," which is hereby incorporated by reference in its entirety. In one implementation, a width "W" of less than about 2.5 is achieved to achieve desired image quality.

In another example embodiment, aliasing is mitigated by padding the light field with smaller zero-valued boundaries and Fourier transforming prior to pre-multiplication. This pushes the energy slightly out of bounds and minimizes the amplification of aliased energy caused by tilt-corrected premultiplication.

FIG. 22 is a flowchart illustrating a method of refocusing in the frequency domain using the above-mentioned various corrections according to another exemplary embodiment of the present invention. At block 2210, a discrete 4D light field is received. In a preprocessing stage once per input lightfield, block 2215 checks whether aliasing reduction is desired, and if so performs padding of the lightfield with small zero-valued boundaries (eg, 5% of width in that dimension) Box 2220. At block 2225, a check is made to determine if tilt correction is desired, and if so, the light field is modulated at block 2230 by the inverse of the Fourier transform of the resampling filter. In the final block of the preprocessing stage, the 4D Fourier transform of the light field is computed at block 2240 .

In a refocusing phase performed once per desired focal length, the process receives desired focal lengths for the refocused image at block 2250 , such as through user guidance. At block 2260, a check is made to determine if aliasing reduction is desired. If not, block 2270 extracts a 2D slice of the Fourier transform of the light field using the desired 4D resampling filter, where the trajectory of the 2D slice corresponds to the desired focal length; and block 2275 computes the inverse 2D Fourier transform of the extracted slice and proceeds to block 2290. If at block 2260 aliasing reduction is desired, the process proceeds to block 2280 where 2D slices are extracted using the desired 4D resampling filter and oversampling (eg, 2x oversampling in each of the two dimensions). At block 2283, the 2D inverse Fourier transform of the patch is computed and the resulting image is cropped to the original size without oversampling at block 2286, after which the process proceeds to block 2290. At block 2290, a check is made to determine if refocusing is complete. If not, another focal length is selected at block 2250 and the process proceeds as described above. If refocusing is complete, the process exits at block 2295.

By summing the rays explicitly as described in the alternative embodiment above, the asymptotic computational complexity of the frequency domain algorithm is less than refocusing. Suppose the input discrete light field has N samples in each of its four dimensions. The computational complexity of the algorithm explicitly summing the rays is then O( ^N4 ) for refocusing at each new depth. For refocusing at each new depth, the computational complexity of the frequency domain algorithm is O(N ² logN), mainly the cost of the 2D inverse Fourier transform. However, the preprocessing step consumes O(N ⁴ logN) for each new light field dataset.

In another example embodiment, the captured light is optically filtered. While not limited to this application, some examples of such filters are neutral density filters, color filters, polarizing filters. Any existing filter or filter to be developed can be used to perform the desired filtering of the light. In one implementation, the light rays are optically filtered in groups or individually such that each group or individual light is filtered differently. In another implementation, filtering is achieved by using a spatially varying filter attached to the main lens. In an example application, a gradient filter such as a neutral density gradient filter is used to filter the light. In another implementation, a spatially varying filter is used prior to one or more of the light sensor, microlens array, or light sensor array. Referring to FIG. 1 , one or more such filters are selectively placed before one or more of main lens 110 , microlens array 120 , and photosensor array 130 , as an example.

In another example embodiment of the invention, a computational component, such as a processor, is programmed to selectively select the rays combined in a computational output pixel to facilitate a desired net filtering of the pixel value. As an example, consider an optically neutral gradient density filter at the master lens, with each lens aperture image appearing under the microlens weighted according to the filter gradient across its range. In one implementation, the output image is computed by selecting the photosensors under each microlens at a point on a gradient that matches the desired level of neutral density filtering of the output image pixels. For example, to produce an image in which individual pixels are filtered over a larger range, each pixel value is set to the sensor value at the extreme of the gradient corresponding to maximum filtering under the corresponding lenticule.

FIG. 2 is a data flow diagram illustrating a method of processing an image in connection with other example embodiments of the invention. Image sensor device 210 acquires image data using microlens/light sensor chip device 212 in a manner similar to microlens array 120 and photosensor array 130 shown, for example, in FIG. 1 and described above. Image sensor device 210 optically includes integrated processing circuitry 214 carrying some processing circuitry in preparation for acquiring image data for transmission.

The sensor data generated on image sensor device 210 are transmitted to signal processor 220 . The signal processors include a low-resolution image processor 222 and one or both of a compression processor 224 and a (light) ray direction processor 226; each of these processors is implemented individually or using a common processor function depending on the application accomplish. In addition, each of the processors shown in FIG. 2 is selectively programmed with one or more processing functions described in connection with other figures or other paragraphs herein. The signal processor 220 and the image sensor arrangement 210 are optionally implemented on a common device or component, eg on a common circuit and/or on a common image device.

Low-resolution image processor 222 uses the sensor data received from image sensor device 210 to generate low-resolution image data, which is sent to view-through display 230 . An input device 235 such as a camera or a button on a camcorder sends an image capture request to the signal processor 220 to request, for example, to capture a particular image displayed in the view-through display 230 and/or to initiate video imaging when so implemented.

In response to an image capture request or other guidance, signal processor 220 uses sensor data captured by image sensor device 210 to generate processed sensor data. In some applications, compression processor 224 is implemented to generate compressed raw data that is transferred to data storage 240 (eg, memory). This raw data is then optionally processed at signal processor 220 and/or external computer 260 or other processing means to enable ray direction processing, such as with ray direction processor 226 as described below.

In some applications, light direction processor 226 is implemented to process processor data received at signal processor 220 to rearrange sensor data for use in generating focused and/or corrected image data. Light direction processor 226 uses one or both of sensor data received from image sensor device 210 and raw data transmitted to data storage device 240 . In these applications, the ray direction processor 226 uses the ray mapping characteristics of the particular imaging device (such as a camera, video camera, or mobile phone) where the image sensor device 210 is implemented to determine the Rearrangement of measured rays. Image data generated with ray direction processor 226 is sent to data storage device 240 and/or to communication link 250 for various applications, such as streaming or sending the image data to a remote location.

In some applications, integrated processing circuitry 214 includes some or all of the processing functionality of signal processor 220 by implementing, for example, a CMOS-type processor or other processor of suitable functionality. For example, low resolution image processor 222 optionally includes integrated processing circuit 214 and low resolution image data is sent directly from image sensor device 210 to view-through display 230 . Similarly, compression processor 224 or similar functionality is optionally implemented with integrated processing circuitry 214 .

In some applications, the calculation of the final image may be performed on the integrated processing circuit 214 (eg, in some digital still cameras that only output the final image). In other applications, image sensor device 210 may simply send raw light data or a compressed version of such data to an external computing device such as a desktop computer. Computation of the final image from these data is then performed on an external device.

FIG. 3 is a flowchart of a method of processing image data according to another example embodiment of the present invention. At block 310, image data is captured on a camera or other imaging device using a master lens or a lens group with a microlens/light sensor array such as that shown in FIG. If a preview image is desired at block 320 , a preview image is generated at block 330 using, for example, a viewfinder or other type of display. The preview image is displayed on, for example, a camera or a viewfinder of a video camera using a subset of the captured image data.

At block 340, raw data from the photosensor array is processed and compressed for use. At block 350, ray data is extracted from the processed and compressed data. This extraction involves, for example, detecting a bundle or collection of light rays incident on a particular photosensor in the photosensor array. Light mapping data is retrieved at block 360 for the imaging device in which the image data was captured. At block 370, the raymap data and the extracted ray data are used to synthesize a rearranged image. For example, the extraction, mapping, and synthesis blocks 350-370 are selectively implemented by determining a ray for a particular pixel of the scene from which the ray is captured, and integrating the ray energy to synthesize a value for the particular pixel. In some applications, ray mapping data is used to trace rays for each specific pixel through the actual lens used to acquire the image data. For example, by determining at block 370 an appropriate set of rays that add together to facilitate focusing on a selected object at a particular focal length, the rays may be rearranged to arrive at a focused image. Similarly, by determining an appropriate arrangement of rays to correct conditions such as lens aberrations in an imaging device, the rays can be rearranged to produce an image that is relatively free of characteristics associated with the aberrations or other conditions.

Various methods are optionally used to generate preview images for camera-type and other applications. FIG. 4 is a flowchart of a process for generating such a preview image according to another example embodiment of the present invention. The method shown in FIG. 4 and described below may be implemented in conjunction with generating a preview image, eg, at block 330 of FIG. 3 .

A preview instruction for raw sensor data is received at block 410 . At block 420, a center pixel is selected from each microlens image in the raw sensor image data. The selected center pixels are collected at block 430 to form a high depth-of-field image. At block 440, the high depth-of-field image is down-sampled at a resolution suitable for the view-through display. Referring to FIG. 2 , such downsampling is selectively performed on one or more image sensor devices 210 or signal processors 220 , as examples. The generated preview image data is sent to the viewfinder at block 450 and the viewfinder displays an image using the preview image data at block 460 .

FIG. 5 is a flowchart of processing and compressing image data according to another example embodiment of the present invention. The method shown in FIG. 5 and described below may be implemented in conjunction with processing and compressing image data at block 340 in FIG. 3 . When implemented with the device shown in FIG. 2 , the method shown in FIG. 5 may be implemented in, for example, one or both of the image sensor device 210 and the signal processor 220 .

At block 510, raw image data is received from a sensor array. If colorization is desired at block 520, the color filter array values are demosaiced at block 530 to produce color at the sensor. If alignment and alignment is desired at block 540 , the microlenses are adjusted and aligned with the photosensor array at block 550 . If interpolation is desired at block 560, the pixel values are interpolated at 570 to an integer number of pixels associated with each microlens. At block 580, the processed raw image data is compressed and rendered for compositing processing (eg, forming a refocused and/or corrected image).

FIG. 6 is a flowchart of image synthesis according to another example embodiment of the present invention. The method shown in FIG. 6 and described below may be implemented in conjunction with the image synthesis method shown at block 370 of FIG. 3 and described further below.

At block 610, raw image data is received from the light sensor array. If refocusing is desired at block 620, the image data is refocused at block 630 using methods such as those described herein to selectively rearrange light represented by the original image data. If image correction is desired at block 640 , the image data is corrected at block 650 . In various applications, where both refocusing and image correction are desired, this may be done before the image correction at block 650 or concurrently with the refocusing at block 630 . At block 660, a resulting image is generated using the processed image data (where available) including refocused and corrected data.

FIG. 7A is a flowchart of image refocusing using a lens arrangement according to another example embodiment of the present invention. The method shown in FIG. 7 and described hereinafter may be implemented, for example, in conjunction with image data refocusing at block 630 in FIG. 6 .

At block 710, a virtual focal plane for refocusing the image portion is selected. At block 720, the virtual image pixels of the virtual focal plane are selected. If correction is desired at block 730 (eg, for lens aberrations), then at block 740 values for virtual lines (or sets of virtual lines) passing between the selected pixel and each particular lens position are calculated. In one application, this calculation is facilitated by calculating the conjugate ray falling on the selected pixel and tracing the ray through a path in the lens arrangement.

At block 750, the sum of the ray values (or set of virtual lines) for each lens position for a particular focal plane is summed to determine a total value for the selected pixel. In some applications, the sum accumulated at block 750 is a weighted sum, where certain rays (or sets of rays) are given greater weight than others. If there are additional pixels for refocusing at block 760, another pixel is selected at block 720 and the process continues until no pixels need to be refocused. After the pixels have been refocused, the pixel data is combined at block 770 to produce a refocused virtual image at the virtual focal plane selected at block 710 . Some or all of the refocusing methods involving blocks 720, 730, 740 and 750 in FIG. 7 are implemented by more specific functions of the various applications.

Sensor data processing circuitry implemented using one or more of the example embodiments described herein may, depending on the implementation, include one or more microprocessors, application specific integrated circuits (ASICs), digital signal processors (DSPs), and/or programmable Gate arrays such as Field Programmable Gate Arrays (FPGAs). As such, the sensor data processing circuitry may be of any type or form of circuitry now known or later developed. For example, a sensor data processing circuit may comprise a single component or multiple components (microprocessor, ASIC and DSP) coupled together to implement, provide and/or perform a desired operation/function/application, active and/or passive.

In various applications, sensor data processing circuitry implements or executes one or more applications, routines, programs and/or data structures that implement particular methods, tasks or operations described and/or illustrated herein. Functions of applications, routines, or programs are selectively combined or allocated in some applications. In some applications, applications, routines or programs are implemented by sensor (or other) data processing circuitry using one or more of a variety of programming languages known or later developed. Such programming languages include, for example, FORTRAN, C, C++, Java, and BASIC, optionally implemented in conjunction with one or more aspects of the invention, compiled or uncompiled.

The various embodiments described above are provided for illustration only and should not be construed as limiting the present invention. From the above description and illustrations it will be apparent to those skilled in the art that various modifications and changes can be made in the present invention without strictly following the exemplary embodiments and applications shown and described herein. For example, such changes may include implementing various optical imaging applications and devices in different types of applications, increasing or decreasing the number of rays collected per pixel (or other selected image area), or implementing different Algorithms and/or equations to acquire or process image data. Other variations may include the use of coordinate representations in addition to or in addition to Cartesian coordinates, such as polar coordinates. Such modifications and changes are made without departing from the true spirit and scope of the invention.

Claims (12)

1. one kind is used for from the digital imaging system of the light set composograph of being caught, and said system comprises:

Main lens, said main lens are used for light is guided to the physics focal plane;

Be used to catch the photosensor array of light set;

Microlens array on the plane between said main lens and the said photosensor array, said light set physically is directed to said photosensor array from said main lens through said microlens array;

Data processor uses the virtual redirection information of the said light set of being caught by said photosensor array to calculate synthetic focusedimage again, and said focusedimage again is focused on the virtual focal plane different with said physics focal plane.

2. the system of claim 1 is characterized in that, said data processor comes computed image through the light of being caught that optionally makes up said light set.

3. the system of claim 1 is characterized in that, said data processor comes computed image through the light of the said light set that optionally adds up.

4. the system of claim 1 is characterized in that, said data processor comes computed image through the light of the optionally weighting and the said light set that adds up.

5. the system of claim 1; It is characterized in that spatial distribution and the set of (ii) said light that said data processor uses (i) said light to be integrated on the said photosensor array come computed image from said main lens through the physical direction that said microlens array arrives said photosensor array.

6. digital imaging system as claimed in claim 1; It is characterized in that said data processor is redirected the set of said light the part of said image is focused on different with the plane of placing said microlens array and different with the plane of the placing said photosensor array virtual focal planes again virtually virtually.

7. digital imaging system as claimed in claim 1 is characterized in that, said data processor uses the said virtual redirection information of the light set of being caught by said photosensor array to come computed image with the correcting lens aberration.

8. digital imaging system as claimed in claim 1 is characterized in that said focusedimage again comprises the depth of field of extension.

9. digital imaging system as claimed in claim 1 is characterized in that, for each lenticule in the said microlens array, said photosensor array comprises a plurality of optical sensors.

10. digital imaging system as claimed in claim 9; It is characterized in that; Said main lens focuses on the two dimensional image of scene on the said microlens array; And each lenticule in the wherein said microlens array is adjusted to disperse by said main lens and focuses on light above that, and the light of being dispersed is directed to said lenticular a plurality of optical sensors.

11. digital imaging system as claimed in claim 1; It is characterized in that said data processor uses data from said photosensor array, creates the focuson image through the different depths of focus of resolving each subimage from the different piece that scene is shown by the light of being caught.

12. digital imaging system as claimed in claim 11 is characterized in that, said data processor is through the synthetic final image of the said focuson image of combination.

Images