JP6009099B2 - Apparatus, program and system for improving 3D images - Google Patents

Description

  The present invention relates generally to improving a plurality of three-dimensional digital images.

  In addition to multiple 3D printers and 3D display devices, the proliferation of three-dimensional (3D) stereo cameras and associated depth sensing methods has generated a new set of multiple challenges for processing and enhancing multiple 3D images. More specifically, the quality of captured 3D images can suffer from the effects of suboptimal lighting conditions that exist at the time of capture, such as glare or faint areas. Multiple standard computer graphics methods can be used to enhance multiple 3D images and change lighting effects, but provide 3D depth information from the surfaces of multiple objects in multiple images, and so on. There is currently no suitable standard way that multiple captured images can be improved to change 3D lighting or to reduce or eliminate the effects of suboptimal lighting conditions.

  There are various techniques that can be used during image capture that also capture depth information associated with image texture information. The depth information is processed to generate a point cloud or depth map using a three-dimensional (3D) polygonal mesh that defines the shape of multiple 3D objects in the image. After processing, the depth information can be represented in several formats including, but not limited to, 3D space, or X, Y, and Z point clouds in multiple triangle or quadrilateral 3D surface meshes. Other formats for representing depth information can include an XML encoding format, a texture format, or a graphical format such as openGL.

6 illustrates an example 3D polygon mesh with multiple surfaces to which lighting and multiple textures are applied, according to embodiments. FIG. 6 illustrates an example surface mesh of a sphere that is subdivided into a plurality of triangular surfaces, according to embodiments. FIG. 5 is a process flow diagram illustrating a method for enhancing a plurality of 3D images according to embodiments. FIG. 4A generally shows multiple light vectors for light, viewer / camera, reflection, specular highlight, and surface normal, according to multiple embodiments. FIG. 4B generally illustrates multiple light vectors for light, viewer / camera, reflection, specular highlight, and surface normal, according to multiple embodiments. FIG. 4C generally shows multiple light vectors for light, viewer / camera, reflection, specular highlight, and surface normal, according to multiple embodiments. FIG. 6 is a block diagram illustrating a tangible persistent computer readable medium storing code for enhancing a plurality of 3D images according to embodiments. 1 is a block diagram of an electronic device that can be used in accordance with embodiments. FIG. 3 is a block diagram of an example system that encodes multiple images using a 3D mesh of multiple polygons and corresponding multiple textures, according to embodiments. FIG. 8 is a schematic diagram of a small form factor device in which the system of FIG. 7 may be implemented, according to embodiments. Multiple identical numerals are used throughout this disclosure and multiple figures to refer to similar components and features. The numbers in the 100s refer to the first features in FIG. 1, the numbers in the 200s refer to the first features in FIG. 2, and so on.

  As generally described above, 3D images can be converted into 3D stereo cameras, depth sensors, multi-view cameras, stereo satellites, other depth capture imaging devices, or, for example, simultaneous localization and mapping (SLAM). ) Or light (often laser) detection and depth capture methods such as ranging, depth information can be extracted and processed along with associated texture information. For example, multiple structured light patterns may be scattered in a scene, and multiple structured light patterns vary with distance, so depth information can be reconstructed by detecting the size of the multiple patterns. In other examples, a time of flight (TOF) sensor can be utilized to gather information by measuring the round trip time of the flight of infrared light back from the sensor to the object. The extracted depth information can be encoded with associated texture information, utilized in any media codec format, or encoded in a new codec format.

  The terms image texture and texture are used interchangeably to describe a 2D image array that represents multiple colors on a surface. A plurality of image textures are texture mapped using a plurality of computer graphics techniques. Texture mapping is the process of fitting an image onto a polygonal mesh that includes surface geometry. Thus, during texture mapping, a set of textures are geometrically matched and warped onto a polygonal mesh, similar to a rubber sheet that is stretched over an irregular surface and adhered to the surface. In this way, the surface shape is represented as a polygonal mesh that is separate from the actual colors or image textures on the surface.

  Further, as shown in embodiments, the terms surface mesh section, surface patch, surface mesh, surface polygon, and polygon patch are a plurality of triangles that are connected into a mesh that defines the entire surface, It can be used interchangeably to describe a surface shape with respect to a quadrilateral or other polygon. For example, FIG. 2 shows a triangular mesh display of surface shapes.

  By adopting the above display of multiple images by separating the surface shape from the surface color or image texture, the present technology uses the vector analysis and color space intensity ranking for each image texture, Multiple original light source position vectors and light source colors in the scene can be provided, recovered, and determined. The techniques also reduce or change the original lighting and change the lighting in the scene in subsequent renderings by adding new light sources with new colors to the rendering. sell. As noted above, the terms image and scene can be used interchangeably.

  The extracted depth information can be obtained using a plurality of currently known or later developed methods, a plurality of 3D points, a plurality of surface vectors, or a plurality of geometric shapes such as triangles, quadrilaterals, etc. Is sufficient to allow the creation of a 3D surface mesh or depth map composed of However, the format of depth information and associated image texture information is not standardized. Since there is no standard way to transmit depth information and associated image texture information, lighting information can be used to improve depth information in various applications, such as allowing multiple depth-based lighting corrections or enhancements in multiple scenes. Use can be avoided. Accordingly, embodiments of the present disclosure relate to improving lighting information in multiple 3D images using a standard depth map format and associated texture information, which are multiple in multiple standard computing systems. It can be rendered using multiple graphics processing units and standard computer graphics methods, such as openGL or Microsoft DirectX® for rendering polygonal meshes and associated image textures. The standard depth map format and texture information can be used with any media codec format or encoded with a new codec format. Multiple colorimetric adjustments can be made to an image or scene by allowing multiple 3D depth maps to include a light surface color intensity ranking for each triangle or quadrilateral in a standard format mesh. Those of ordinary skill in the art will be able to limit any number of standard computer graphics methods to textured 3D polygon meshes given surface color intensity ranking and light source determination made using the methods of the present invention. Will recognize that this can be embodied on a standard computer system, such as a tablet, phone, eye-mounted display computer, laptop, or desktop system. Let's go. Thus, multiple lighting adjustments can be made to multiple 3D depth images and corresponding multiple textures.

  In the following description and claims, the terms “connection” and “coupling” may be used in conjunction with their derivatives. It should be understood that these terms are not intended as synonyms for each other. Rather, particularly in embodiments, “coupled” can be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Connected” can mean that two or more elements are in direct physical or electrical contact. However, “connection” may also mean that two or more elements are not in direct contact with each other but still operate or interact in cooperation with each other.

  Some embodiments may be implemented in one or a combination of hardware, firmware, and software. Some embodiments may also be implemented as a plurality of instructions stored on a machine-readable medium that are read and executed by a computing platform to perform the operations of the present disclosure. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine, eg, a computer. For example, a machine-readable medium includes, among other things, read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; or electrical, optical, acoustic, or other forms of propagation It may include a signal, eg, a carrier wave, an infrared signal, a digital signal, or an interface that transmits and / or receives signals.

  Embodiments are implementations or examples. References in the specification to “embodiments,” “one embodiment,” “some embodiments,” “various embodiments,” or “other embodiments” are described with respect to a plurality of embodiments. A particular feature, structure, or characteristic specified is included in at least some embodiments of the inventions, but not necessarily in all embodiments. The various appearances of “an embodiment”, “one embodiment”, or “some embodiments” do not necessarily all refer to the same embodiments. Multiple elements or aspects from an embodiment may be combined with multiple elements or aspects of other embodiments.

  Not all components, features, structures, characteristics, etc. described and shown in this disclosure need to be included in a particular embodiment or embodiments. Where the description may include a component, feature, structure, or property ("may", "might", "can", or "could"), for example, that particular component, feature, structure, or property That is not essential. When a specification or claim refers to one (“a” or “an”) element, it does not mean that there is only one of the elements. Where the specification or claims refer to “an additional” element, it does not exclude that there are more than one additional element.

  Although some embodiments have been described with reference to particular implementations, it should be noted that other implementations are possible with some embodiments. Further, the arrangement and / or order of multiple circuit elements or other features shown in the drawings and / or described in the present disclosure is not necessarily arranged in the particular manner shown and described. Absent. Many other configurations are possible with some embodiments.

  In each system shown in the figures, the elements in some cases have the same or different reference numbers to suggest that the elements shown may be different and / or similar. . However, an element may have multiple different implementations and be sufficiently flexible to operate with some or all of the multiple systems shown or described in this disclosure. The various elements shown in the figures may be the same or different. Which is referred to as the first element and which is referred to as the second element is arbitrary.

  FIG. 1 illustrates an example 3D polygonal mesh 102 having multiple surfaces to which lighting and textures are applied, according to multiple embodiments. The 3D polygon mesh 102 forms a human face structure. The polygon mesh includes a plurality of vertices, lines, edges, and faces that are used to define the shape of the 3D object, specifically the human face. Any type of mesh can be used by multiple techniques. For example, the mesh can be a quadrilateral mesh or a triangular mesh. Further, alternative multiple depth formats can also be used by multiple embodiments. For example, depth information can also be considered a 3D point cloud because the mesh is composed of multiple points in 3D space. Further, the mesh may be encoded as a depth map in a 2D array where multiple array values indicate the depth of each point.

  Polygon mesh 102 may include a plurality of control points. A control point is a position within the polygon mesh 102 that includes corresponding information such as color, multiple normal vectors, and multiple texture coordinates. Multiple texture coordinates can be used to link control points to texture information such as a texture map. Texture information adds multiple details, colors, or image texture information to a triangular mesh.

  The image may be applied or texture mapped onto the surface polygon mesh 102, thereby generating the image 104. When multiple textures are applied on each surface polygon, the surface polygon mesh 102 exhibits the shape of a human face. While polygon mesh 102 and the corresponding textures that are applied to yield image 104 have been described in the context of rendering 3D images, multiple 2D images can also be rendered using multiple techniques. In any case, rendering an image using a plurality of polygonal meshes and corresponding textures can be performed using openGL, DirectX, H.264, or the like. It can be achieved using a graphics pipeline in conjunction with standard graphics or media encoding formats such as H.264, MPEG4, and VC1 and any other media codec format.

  Additional lighting and multiple textures may be applied to further refine the image 104, resulting in the image 108. Lighting is applied to the image 108 to produce a faint shadow on the neck from the chin of the human face on the image 108. Unwrapped texture map 106 shows texture information from image 108 without any lighting applied. In embodiments, the lighting on the image 108 may be varied to reduce or eliminate shadows on the human face from the chin to the neck in the image 108.

  FIG. 2 illustrates an example surface mesh 200 that is subdivided into a plurality of triangular surfaces 202, 204, 206,..., 214, etc., according to embodiments. However, it should be understood that the surface mesh may be subdivided using multiple surfaces having, for example, multiple 3D points, surface vectors, or other geometric shapes. Each of the plurality of surfaces (eg, 202, 204, 206,..., 214, etc.) of the surface mesh 200 corresponds to a section of an example sphere texture image (not shown). The color intensity of each of the plurality of texture image sections is, for example, HSL (hue, saturation, and lightness), HSV (hue, saturation, and value), YUV, or color intensity. Can be determined and ranked by intensity in color space, such as other suitable methods of determining and ranking. By sorting multiple surfaces according to color intensity and ranking them, the actual scene lighting is restored using a vector analysis of the surface normal of each surface by the observation position surface normal, Is useful for relighting the mesh. A plurality of texture image sections having a plurality of high levels of color intensity correspond to a plurality of sections of the surface mesh 200 having a high level of specular reflection from the light source to the viewing position, thus determining the light source vector direction and color. Illustrated are multiple areas of bright lines that are useful to do, and light source information is useful for changing lighting effects using multiple standard computer graphics methods. Conversely, a plurality of texture image sections having a plurality of low level color intensities correspond to a plurality of sections of the surface mesh 200 having a plurality of low level specular reflections from the light source to the viewing position, and therefore, low or Can show multiple areas of dim lighting. Thus, multiple texture image sections can be ranked from high color intensity to low color intensity to determine the actual scene lighting direction vectors and light source colors.

  Maximum light reflection occurs when the surface normal vector N is intermediate between the observation (image capture device) vector V and the light source vector L. The surface intermediate normal vector information can be used to calculate a light source vector that is given recognition of the view vector V that defines the position of the camera. The higher ranking color intensity surfaces and their corresponding surface normal vectors from the polygon mesh are used together as a set to calculate an approximation of the light source vector L of a given image. In an embodiment, the light source vector L may be recovered and determined from the original 3D surface shape data and a plurality of image texture patches corresponding to each surface, and the best estimate of the original light source vector is the most Selected based on a plurality of surface normals N from mesh polygons corresponding to high color intensity, or else the light source vector L is an average of a plurality of L vectors, a median L vector, a plurality of L vectors It can be derived from a weighted or calculated mathematical combination of a set of multiple light source vectors L calculated for each surface normal N, such as linear regression or other mathematical methods. For this reason, the light source can be located in 3D space. As described more specifically in this disclosure below, a set of surfaces ranked by color intensity, and their corresponding normal vectors, are further used to capture the lighting of the captured multiple 3D images. Can be used to adjust, add or remove, enhance, change the color of lighting, or otherwise change the lighting, set of multiple surface normal vectors ranked by color intensity, multiple The image texture and associated polygons may be encoded in a file, communication stream, memory, or other data structure in a standard code format or other standard data format.

  FIG. 3 is a process flow diagram illustrating a method 300 for enhancing multiple 3D images, according to multiple embodiments. In various embodiments, the method 300 provides an enhanced 3D image. In some embodiments, the method 300 may be performed on a computing or other electronic device, as described more specifically in this disclosure. The method 300 includes capturing a 3D image 302, extracting depth information and texture information 304 from the 3D image, generating a surface mesh 306, determining a plurality of texture sections corresponding to the plurality of surface mesh sections. 308, ranking a plurality of surfaces 310, determining a plurality of light source vectors 312, adjusting lighting 314, and outputting an enhanced image 316.

  At block 302, a 3D image is converted into a 3D stereo camera, depth sensor, multi-view camera, stereo satellite, other depth capture imaging device, or, for example, simultaneous self localization and mapping (SLAM) or Captured using depth capture methods such as light (typically laser) detection and ranging, or otherwise provided with source 3D images. Extracting depth information and texture information 304 may include extracting or otherwise determining a depth or texture map from the captured 3D image data.

  At block 306, a surface mesh, such as the example surface mesh 200, can be generated using the extracted depth information and the surface mesh can then be distributed into multiple surface mesh sections as previously described. As described above, the plurality of surface mesh sections may be any desired geometric shape including triangles, quadrilaterals, etc., which are a plurality of 3D points and a plurality of surface vectors.

  At block 208, a texture section corresponding to each surface mesh section is determined. For example, for each individual surface mesh section 202-214 of the surface mesh 200, the corresponding texture image section of the 3D image depth or texture map is determined.

  At block 310, the plurality of texture image sections are ranked with respect to color intensity, which corresponds to ranking the specular reflection of each of the plurality of individual surfaces of the surface mesh 200, as described above. Therefore, the plurality of surfaces can be ranked from the highest color intensity to the lowest color intensity.

  At block 312, a plurality of light source vectors is determined, including determining a plurality of intermediate normal vectors for each or a desired subset of the plurality of individual texture image sections. In one embodiment, the highest ranked individual texture image section and its surface corresponding intermediate normal vector may be used to determine the light source vector via known multiple vector methods, thereby providing a 3D Determining the position of the light source in space, which for each surface texture lighting to a new set of multiple light sources using multiple standard computer graphics methods for surface lighting, Enable change. In other embodiments, a plurality of texture image sections having a color intensity with a rank exceeding a predetermined threshold, and a plurality of normal vectors corresponding to their surfaces are mathematically combined together as a plurality of groups. The best estimate of the correct light source vector is made, and the best estimate of the intermediate normal vector result may be used to determine the position of the light source vector via known multiple vector methods, thereby providing a 3D Determine the position of the light source in space. In yet other embodiments, the plurality of light source vectors determined for each of the plurality of texture image sections having a rank that exceeds a predetermined threshold are mathematically combined together into a plurality of groups, Can determine the best estimate of the light source vector, thereby determining the position of the light source in 3D. In an embodiment, the light source vector L may be selected based on the surface normal N from the mesh polygon corresponding to the highest color intensity, or else the light source vector L is an average of a plurality of L vectors, From a weighted or calculated combination of sets of multiple light source vectors L calculated for each surface normal N, such as median L vector, linear regression of multiple L vectors, or other mathematical methods May be derived.

  At block 314, a plurality of attributes or characteristics of the light source are adjusted, which generally include, but are not limited to, adjusting light brightness, color (ie, hue, saturation, chroma, etc.), color intensity or luminance, and It may include adding and removing multiple light sources. More specifically, in one example embodiment, lighting to one or more of the plurality of texture image sections that have a color intensity rank that exceeds a predetermined threshold should effectively remove the light. Or adjusted to reduce the color intensity of one or more surfaces to provide anti-light, which in one embodiment is to remove glare or from a new location using standard computer graphics methods. It can be useful to increase the lighting of dark surfaces by adding light. In another example embodiment, the one or more lightings of the plurality of texture image sections having a color intensity rank below a predetermined threshold are adjusted by increasing the color intensity of the one or more surfaces. In yet another example embodiment, the above-described embodiments may be combined, and one or more lightings of a texture image section having a color intensity rank that exceeds a predetermined threshold may be combined with the one or more of them. Add lighting to one or more of the texture image sections that have a color intensity rank that is less than or equal to a predetermined threshold by reducing the color intensity of the surface and increasing the color intensity of those one or more surfaces . In yet another example embodiment, which may be combined similarly to the previous or later embodiments of the present disclosure, one or more light sources are added by adding a position vector with an associated light source color The added light source is generally directed to all surfaces, or a predetermined plurality of surfaces, such as a plurality of surfaces having greater or lesser color intensity equal to a predetermined threshold, for example. It can be directed selectively to the surface. Thus, an improved image can be generated. All of the embodiments can be implemented in and using standard computer graphics methods.

  The adjustment of multiple characteristics or properties of the light source performed at block 314 also removes any light source present in the 3D image when captured by projecting “anti-light” in embodiments. Can include. As used in this disclosure, “anti-light” includes changing the color intensity of all or selected texture image sections. Adding “anti-light” is removing light from multiple texture image sections. The adjustment of the plurality of characteristics or characteristics of the light source performed in block 314 may be interactive in the same or other embodiments, further adding a plurality of new light sources to the 3D image, adjusting a plurality of lighting characteristics. Using multiple controls, adding synthetic light source information to multiple standard video encoding formats, adjusting multiple lighting characteristics by using multiple standard GPU graphics pipelines to add / remove lighting effects, and Using digital light measurement and adjustment in multiple camera systems.

  At block 316, the enhanced image is output, which may include rendering the enhanced image on a display device or via 2D or 3D printing. In another embodiment, the enhanced image is a computer readable file, such as a media format file, for example, a hard disk drive, memory stick, SD card, random access memory, read only memory, or presently known or later devised. Can be output by storing the enhanced image in a computer readable memory, such as any other form of electronic storage. In addition, the output of the enhanced image is also a standard depth map format such as a surface mesh composed of a plurality of triangles or other geometric shapes with a corresponding plurality of textured surfaces, a plurality of point clouds Ranked in color intensity in a predetermined or standard format, such as multiple Z images or 2D pixel arrays with pixel values corresponding to depth or Z dimension, or standard camera image sensor output formats such as a subset of MIPI standards A plurality of sets of surfaces, thereby allowing an initial or additional lighting adjustment to be performed downstream prior to or in conjunction with enhanced image rendering.

  The process flow diagram of FIG. 3 shows that multiple blocks of the method 300 should be executed in any particular order, or that all of the blocks should be executed or included in all embodiments. Not intended to show. Moreover, any number of additional blocks may be included in method 300 depending on the particular implementation details.

  4A-C generally illustrate multiple light vectors, viewer / camera, reflection, specular highlight, and surface normal for light, according to multiple embodiments. More specifically, FIG. 4A shows that the intermediate normal vector “n” is determined to be in the middle between the source vector L and the view vector “w” of the light source. As previously mentioned, the intermediate vector is the vector where the maximum reflection of the light source is observed. FIG. 4B shows a view vector V, a surface normal vector N, a light source vector L, and a reflection vector R for a non-planar surface. FIG. 4C shows the relationship between the light source vector L, the view vector V, the mirror reflection vector R of the light source vector L, the smooth surface normal vector N, and the half-angle vector H which is an intermediate vector between the vectors L and V. .

  FIG. 5 is a block diagram illustrating a tangible persistent computer readable medium 500 that stores code for enhancing a plurality of 3D images, according to embodiments. Tangible persistent computer readable medium 500 may be accessed by processor 502 via signal bus 504. Moreover, medium 500 can include code configured to cause processor 502 to perform the methods and particularly method 300 of the present disclosure. Media 500 may store various software components or modules, including a plurality of components or modules that correspond to the processing of method 300 shown in FIG. For example, the medium 500 can include a 3D image capture module 512 that can be configured to capture, import, or otherwise obtain or receive 3D images. The medium 500 may also include a depth information and texture information extraction module 514 configured to extract or otherwise determine the depth and texture map of the 3D image. The medium 500 can also include a surface mesh generation module 516 that can be configured to generate a surface mesh of the 3D image. The medium 500 can further include a texture section determination module 518 corresponding to the surface mesh section that can be configured to determine a corresponding texture section for each surface mesh section. The medium 500 can still further include a surface ranking module 520 that can be configured to rank the plurality of surfaces with respect to at least one of color intensity and specular reflection. The medium 500 can still further include a light source vector determination module 522 that can be configured to determine a light source vector and thereby indicate the position of the light source in 3D space. The medium 500 may also include a lighting adjustment module 524 that may be configured to adjust a plurality of characteristics of the light source of the 3D image, as described more specifically above with respect to the method 300. The medium 500 can also be configured to output an enhanced image and associated data, such as a set of multiple surfaces ranked by color intensity and associated multiple textures. 526 may be included.

  The block diagram of FIG. 5 is not intended to indicate that the tangible persistent computer readable medium 500 should include all of the components shown in this disclosure. Further, tangible persistent computer readable medium 500 may include any number of additional components not shown in FIG. 5, depending on the particular implementation details.

  FIG. 6 is a block diagram of an electronic device that can be used in accordance with embodiments. The electronic device 600 can be virtually any electronic device including, for example, a printing device, camera, laptop, or other portable or desktop computer, cellular or other mobile phone, and the like. The electronic device 600 may include a central processing unit (CPU) 602, a memory 604, a signal bus 606, a graphics processing unit (GPU) 608, an image capture mechanism 610, a sensor 612, and a display 614.

  The CPU 602 may be configured to read and execute a plurality of applications 620 that may be stored in the memory 604 via the signal bus 606. CPU 602 may be a single, dual, or multi-core processor or any other suitable processing device. Memory 604 may include random access memory (RAM), read only memory (ROM), hard disk or solid state drive, flash memory, or other suitable type of memory. The memory 604 stores a plurality of applications 620.

  A signal bus 606 interconnects the CPU 602, memory 604, GPU 608, image capture mechanism 608, image capture mechanism 610, sensor 612, and display 614 to allow the transfer of electronic signals and data between and within the same. To do.

  The GPU 608 may be any suitable GPU processor, for example, rendering or processing of multiple graphics images, frames, or videos, etc., for display on a display 614 or other display to a user of the device 600. Can be configured to perform any number of graphics operations. In some embodiments, GPU 608 may include a number of graphics engines (not shown) that may be configured to perform a plurality of specific graphics tasks.

  Image capture mechanism 610 may be configured to capture 3D images. The image capture mechanism 610 may be configured as a camera, stereoscope camera, infrared sensor, or other image capture mechanism. Image capture mechanism 610 is configured to capture depth and image texture information associated with the captured 3D image. In some examples, the sensor 612 may alternatively be configured to capture captured image depth and image texture information, so that a charge coupled device image sensor, a system on chip sensor image sensor, or other It may be configured as a suitable sensor or a combination of sensors. Multiple applications 620 encode depth and image texture information in a surface mesh, such as a 3D depth map or surface mesh 200, in a standardized media codec that currently exists or will be developed in the future.

  The display 614 may be configured as a flat panel liquid crystal or other display device configured to display captured images to the user, and further a graphical or human machine interface to the user for control and operation of the device 600. May be configured to display. Display 614 may be integral with or separate from device 600.

  The plurality of applications 620 includes a plurality of instructions sufficient to perform the method 300. For example, multiple applications 620 may extract depth information and texture information or otherwise capture 3D images to capture, acquire, or otherwise receive 3D images, as described more specifically above with respect to method 300. A plurality of at least one of color intensity and specular reflection to determine a depth and texture map, to generate a surface mesh module of a 3D image, to determine a plurality of texture sections of the image corresponding to a plurality of surface mesh sections Multiple instructions may be included to rank the surface of the image, to determine the light source vector and thereby indicate the position of the light source in 3D space, and to adjust multiple characteristics of the light source in the 3D image. The plurality of applications 620 may also output a plurality of enhanced images and associated data, such as sets of textures associated with a plurality of surfaces and 3D images ranked by color intensity. Instructions can be included.

  The electronic device 600 can in some embodiments print an encoded 3D surface mesh, such as the surface mesh 200, and a corresponding plurality of texture sections, such as a printer, fax machine, and other printing devices. A print device 622. The print device 622 can be integral with or external to the electronic device 600. The plurality of applications 620 may include a plurality of instructions necessary to interface and operate with the print device 622.

  The block diagram of FIG. 6 is not intended to indicate that the device 600 should include all of the components shown in FIG. Further, device 600 may include any number of additional components not shown in FIG. 6, depending on the particular implementation details.

  FIG. 7 is a block diagram of an example system 700 that encodes multiple images using a 3D mesh of polygons and corresponding multiple textures, according to multiple embodiments. A plurality of items with similar numbers are described with respect to FIG. In some embodiments, system 700 is a media system. In addition, the system 700 is a personal computer (PC), laptop computer, ultra laptop computer, tablet, touchpad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), mobile phone, mobile phone / It can be incorporated into a combination of PDAs, televisions, smart devices (eg, smart phones, smart tablets, or smart TVs), mobile internet devices (MID), messaging devices, data communication devices, and the like.

  In various embodiments, the system 700 includes a platform 702 connected to a display 704. Platform 702 may receive content from a content device, such as content service device 706 or content distribution device 708, or other similar content sources. A navigation controller 710 that includes one or more navigation functions may be used to interact with the platform 702 and / or the display 704, for example. Each of these components is described in more detail below.

  Platform 702 may include any combination of chipset 712, central processing unit (CPU) 602, memory device 604, storage device 713, graphics subsystem 714, multiple applications 620, and wireless device 716. Chipset 712 may provide intercommunication between CPU 602, memory device 604, storage device 713, graphics subsystem 714, multiple applications 620, and wireless device 714.

  The CPU 602 may be implemented as a compound instruction set computer (CISC) or reduced instruction set computer (RISC) processor, x86 instruction set compatible processor, multi-core, or any other microprocessor or central processing unit (CPU). In some embodiments, the CPU 602 includes a dual core processor, dual core mobile processor, or others.

  Memory device 604 may be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM). Storage device 713 includes, but is not limited to, a magnetic disk drive, optical disk drive, tape drive, internal storage device, connected storage device, flash memory, battery backup SDRAM (synchronous DRAM), and / or network access. It may be implemented as a non-volatile storage device such as a possible storage device. In some embodiments, the storage device 713 includes technology to increase storage performance enhanced protection for important digital media, for example when multiple hard drives are included.

  Graphics subsystem 714 may perform processing of multiple images, such as still or video for display. Graphics subsystem 714 may include a graphics processing unit (GPU) such as GPU 608 or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple graphics subsystem 714 and display 704. For example, the interface can be any of a high definition multimedia interface, a display port, wireless HDMI® and / or wireless HD compliant technology. Graphics subsystem 714 can be integrated into CPU 602 or chipset 712. Alternatively, graphics subsystem 714 may be a stand-alone card that is communicatively coupled to chipset 712.

  The graphics and / or video processing techniques described in this disclosure may be implemented in various hardware architectures. For example, graphics and / or video functions may be integrated within chipset 712. Alternatively, separate graphics and / or video processors can be used. As yet another embodiment, graphics and / or video functionality may be implemented by a general purpose processor including a multi-core processor. In further embodiments, multiple functions may be implemented in a consumer electronic device.

  Wireless device 716 may include one or more radios that can transmit and receive signals using a variety of suitable wireless communication technologies. Such techniques may involve multiple communications over one or more wireless networks. Exemplary wireless networks include a wireless local area network (WLAN), a wireless personal area network (WPAN), a wireless metropolitan area network (WMAN), a cellular network, a satellite network, or the like. In communication across such multiple networks, the wireless device 716 may operate according to any version of one or more applicable standards.

  Display 704 may include any television type monitor or display. For example, display 704 may include a computer display screen, touch screen display, video monitor, television, or the like. Display 704 may be digital and / or analog. In some embodiments, the display 704 is a holographic display. The display 704 may also be a transparent surface that can receive visual projections. A plurality of such projections may convey various forms of information, images, objects or the like. For example, a plurality of such projections can be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 620, the platform 702 may display a user interface 718 on the display 704.

  The content service device 706 may be hosted by any national, international, or independent service and thus may be accessible to the platform 702 via, for example, the Internet. Content service device 706 may be coupled to platform 702 and / or display 704. Platform 702 and / or content service device 706 may be coupled to network 720 to communicate (eg, send and / or receive) media information from and to network 720. Content distribution device 708 may also be coupled to platform 702 and / or display 704.

  Content service device 706 may include a cable television box, personal computer, network, telephone, or Internet-enabled device capable of distributing digital information. In addition, the content service device 706 can communicate content between multiple content providers and platforms 702 or displays 704 over a network 720 or directly, in other similar plurality of any one-way or two-way. Devices may be included. It will be appreciated that content may be communicated unidirectionally and / or bidirectionally via the network 720 to and from any one of a plurality of components in the system 700 and content providers. Like. Examples of content may include any media information including, for example, video, music, medical, game information, and others.

  Content service device 706 may receive content such as cable television programs that include media information, digital information, or other content. Examples of multiple content providers may include, among other things, any cable or satellite television, or a wireless or internet content provider.

  In some embodiments, platform 702 receives a plurality of control signals from navigation controller 710 that includes one or more navigation functions. The navigation function of the navigation controller 710 may be used to interact with the user interface 718, for example. The navigation controller 710 may be a pointing device that may be a computer hardware component (specifically a human interface device) that allows a user to enter spatial (eg, continuous and multi-dimensional) data into a computer. . Many systems, such as multiple graphical user interfaces (GUIs), televisions, and monitors, allow users to control and provide data to a computer or television using physical gestures. The plurality of physical gestures include, but are not limited to, facial expressions, facial movements, various limb movements, body movements, gestures, or any combination thereof. Such multiple physical gestures can be recognized and converted into multiple commands or multiple commands.

  The movement of the navigation function of the navigation controller 710 is reflected on the display 704 by the movement of a pointer, cursor, focus ring, or several other visual indicators displayed on the display 704. For example, under the control of a plurality of applications 620, a plurality of navigation functions located on the navigation controller 710 can be mapped to a plurality of visual navigation functions displayed on the user interface 718. In some embodiments, navigation controller 710 may not be a separate component, but rather may be integrated within platform 702 and / or display 704.

  The system 700 includes a plurality of drivers (not shown), including, for example, a technique that allows multiple users to immediately turn the platform 702 on and off by touching a button after initial activation when multiple users are enabled. Can be included. Program logic may allow platform 702 to stream content to multiple media adapters or other content service devices 706 or content distribution devices 708 when the platform is off. In addition, the chipset 712 may include hardware and / or software support for 6.1 surround sound audio and / or high resolution 7.1 surround sound audio, for example. The plurality of drivers may include a graphics driver for the integrated graphics platform. In some embodiments, the graphics driver includes a peripheral component interconnect express (PCIe) graphics card.

  In various embodiments, one or more of any of the components shown in system 700 may be integrated. For example, platform 702 and content service device 706 may be integrated, platform 702 and content distribution device 708 may be integrated, or platform 702, content service device 706 and content distribution device 708 may be integrated. In some embodiments, platform 702 and display 704 are an integrated unit. For example, the display 704 and the content service device 706 may be integrated, or the display 704 and the content distribution device 708 may be integrated.

  System 700 can be implemented as a wireless system or a wired system. When implemented as a wireless system, the system 700 is suitable for communicating over a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, and the like. Components and interfaces. An example of a wireless shared medium may include a portion of the radio spectrum, such as the RF spectrum. When implemented as a wired system, system 700 includes a plurality of input / output (I / O) adapters, a plurality of physical connectors for connecting the I / O adapters to corresponding wired communication media, a network interface card (NIC), a disk controller. A plurality of components and interfaces suitable for communicating over a wired communication medium, such as a video controller, an audio controller, or the like. Examples of wired communication media may include wires, cables, multiple metal leads, printed circuit boards (PCBs), backplanes, switch fabrics, semiconductor materials, twisted pair wires, coaxial cables, fiber optics, and the like.

  Platform 702 may establish one or more logical or physical channels to communicate information. The information can include media information and control information. The media information may refer to arbitrary data representing content for the user. Examples of content include, for example, voice conversations, video conferencing, streaming video, email (email) messages, voicemail messages, multiple alphanumeric symbols, graphics, images, videos, text, and the like Data from may be included. Data from a voice conversation can be, for example, speech information, silence periods, background noise, comfort noise, tones, and the like. Control information may refer to any data representing multiple commands, multiple instructions, or multiple control words for an automated system. For example, the control information may be used to route media information through the system or to instruct a node to process media information in a predetermined manner. However, embodiments are not limited to the elements or context shown or described in FIG.

  FIG. 8 is a schematic diagram of a small form factor device 800 in which the system 700 of FIG. 7 may be implemented, according to embodiments. Items labeled with similar numbers are the same as described with respect to FIG. In some embodiments, for example, device 700 is implemented as a mobile computing device having multiple wireless functions. A mobile computing device can refer to any device having a processing system and a mobile power source or supply, such as one or more batteries, for example.

  As mentioned above, several examples of mobile computing devices include personal computers (PCs), laptop computers, ultra laptop computers, tablets, touchpads, portable computers, handheld computers, palmtop computers, personal digital assistants (PDAs). ), Mobile phones, cell phone / PDA combinations, televisions, smart devices (eg, smart phones, smart tablets, or smart TVs), mobile internet devices (MID), messaging devices, data communication devices, and the like Good.

  Examples of mobile computing devices are also like wrist computers, finger computers, ring computers, eyeglass computers, belt clip computers, armband computers, shoe computers, clothing computers, or any other suitable type of wearable computer May include a computer arranged to be worn by a person. For example, a mobile computing device can be implemented as a smartphone capable of executing multiple computer applications with voice and / or data communications. Some embodiments may be described by way of example as a mobile computing device implemented as a smartphone, although other embodiments are implemented using other wireless mobile computing devices as well. It will be understood that this may be done.

  As shown in FIG. 8, the device 800 may include a housing 802, a display 804, an input / output (I / O) device 806, and an antenna 808. Device 800 may also include a plurality of navigation functions 810. Display 804 may include any suitable display unit for displaying information suitable for a mobile computing device. I / O device 806 may include any I / O device suitable for entering information into a mobile computing device. For example, the I / O device 806 includes an alphanumeric keyboard, a numeric keypad, a touch pad, a plurality of input keys, a plurality of buttons, a plurality of switches, a plurality of rocker switches, a plurality of microphones, a plurality of speakers, a voice recognition device, and software. Or the like. Information can also be input to the device 800 using a microphone. Such information may be digitized by a voice recognition device.

  In embodiments, the image capture mechanism may be a camera device that interfaces with a host processor using an interface developed in accordance with specifications from the Mobile Industry Processor Interface (MIPI) Camera Serial Interface (CSI) Alliance. For example, the camera serial interface can be a MIPI CSI-1 interface, a MIPI CSI-2 interface, or a MIPI CSI-3 interface. Thus, the camera serial interface can be any camera serial interface that is currently being developed or will be developed in the future. In embodiments, the camera serial interface may include a data transmission interface that is a one-way differential serial interface with data and clock signals. Furthermore, the camera interface with the host processor can also be any camera parallel interface (CPI) currently developed or developed in the future.

  In embodiments, the image capture mechanism may be a component of a mobile computing device. For example, a camera device developed in accordance with the MIPI CSI Alliance standard may be integrated with at least one or more of the device 600 of FIG. 6, the system 700 of FIG. 7, the device 800 of FIG. 8, or any combination thereof. It can be a mechanism. The image capture mechanism may include various sensors, such as a depth sensor, an image sensor, an infrared sensor, an x-ray photon counting sensor, or any combination thereof. The plurality of image sensors may be a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a system on chip (SOC) image sensor, an image sensor having a photosensitive thin film transistor, or any combination thereof. Can be included.

  Example 1

  An apparatus for enhancing 3D images is described in this disclosure. A 3D image includes depth information and texture information associated with it. The apparatus corresponds to the plurality of surface mesh sections depending at least in part on the depth information and logic for generating a 3D mesh having the plurality of surface mesh sections and at least in part on the texture information. Logic for determining a plurality of texture sections. The apparatus also adjusts the logic for ranking the texture section and the corresponding plurality of surface mesh sections and the characteristics of the one or more light sources, depending at least in part on the color intensity of the plurality of texture sections. Logic for changing one or more of the plurality of texture sections ranked by color intensity and thereby generating an improved image.

  The position of the one or more light sources can be determined and the one or more light sources can be added to the captured 3D image. One or more light sources may also be present in the 3D image when captured. The logic for determining the location of the one or more light sources depends on the plurality of image textures and the corresponding plurality of surface mesh normal vectors at least partially ranked by color intensity, and the one or more light sources The method may include determining the position of a plurality of light sources in the 3D image using the vector. The logic for determining one or more light source vectors may also determine a corresponding light source vector for each of the plurality of texture sections ranked by color intensity and mathematically combine the set of light source vectors. Can be included. The color intensity of the plurality of texture sections ranked by color intensity may include a plurality of texture sections having a color intensity that is one of less than and greater than a predetermined threshold. The logic for determining the light source vector may include determining an intermediate vector that is perpendicular to the view vector and each of the one or more texture sections and surface mesh sections ranked by color intensity. Between the correct vector and the corresponding surface vector. Logic for adjusting the plurality of characteristics of the at least one light source includes at least one of brightness, color, hue, saturation, chroma, lightness, color intensity, and luminance of one or more of the at least one light source. Can include changing. The logic for adjusting the plurality of characteristics of the at least one light source may also include one of adding and removing at least one light source. The surface mesh and corresponding multiple textures may be re-rendered using multiple new light sources, and the resulting image may also be output. Re-rendering the surface mesh and corresponding multiple textures and outputting the resulting image can be done with multiple standard ranked by color intensity in a predefined standard format, standard depth map format, or standard camera output format. Providing at least one of the surfaces may be included. Logic for adjusting the plurality of characteristics of the at least one light source may be performed prior to or in conjunction with enhanced image rendering. Further, the apparatus can be a printing device or an image capture mechanism.

  Example 2

  At least one non-transitory machine readable medium having a plurality of instructions stored thereon is described. The instructions, when executed on the device, cause the device to extract depth information and texture information associated with the 3D image and to generate a surface mesh having a plurality of surface mesh sections from the depth information. The plurality of instructions are executed by the device, causing the device to determine a plurality of texture sections corresponding to the plurality of surface mesh sections from the texture information, the plurality of texture sections and the corresponding plurality of surface mesh sections. Are ranked with respect to their respective color intensity. In addition, the instructions may cause the device to adjust the characteristics of the one or more light sources in response to being executed on the device, thereby providing a plurality of texture sections ranked by color intensity. One or more of the color intensities are changed to generate an improved image.

  The instructions for adjusting the characteristics of the at least one light source may include instructions for adding or removing one or more light sources to the 3D image. Multiple instructions for adjusting, adding or removing multiple characteristics of one or more light sources also adjust, add or remove multiple characteristics of the light sources present in the 3D image when captured A plurality of instructions can be included. The position of at least one light source can be determined. The plurality of instructions for determining the position of the one or more light sources is executed on the device, and a corresponding plurality of surfaces for each of the plurality of texture sections ranked by color intensity. A plurality of instructions for determining a corresponding light source vector for the mesh section and combining the plurality of light source vectors together mathematically may be further included. The plurality of texture sections ranked by color intensity may include a plurality of texture sections having a color intensity that is one of less than and greater than a predetermined threshold. A plurality of instructions for adjusting, adding, or removing a plurality of characteristics of at least one light source, when executed on the device, the brightness, color, hue, chrominance of one or more of the at least one light source. A plurality of instructions for changing at least one of degree, chroma, lightness, color intensity, and luminance may be further included. The instructions for adjusting the characteristics of the at least one light source may further include instructions for adding or removing at least one light source when executed on the device.

  The non-transitory machine-readable medium also uses a plurality of new light sources to re-render the surface mesh and corresponding textures when executed on the device, and output an enhanced image. Can include the following instructions. Multiple instructions for outputting an enhanced image are executed on the device and between multiple cameras, sensors, 2D or 3D printers, 2D or 3D displays, computer graphics systems or computer systems. A plurality of ranked by color intensity in a predetermined standard format, standard depth map format, standard camera output format, standard media codec format, standard computer graphics format, standard 3D depth file format, or standard protocol format. A plurality of instructions for providing at least one of the surfaces of the display;

  Example 3

  A system is described in this disclosure. The system includes a central processing unit configured to execute a plurality of stored instructions and at least one memory device storing a plurality of instructions readable by the central processing unit, wherein the plurality of instructions are central When executed by the processing unit, it is configured to extract depth information and texture information from the 3D image and generate a surface mesh having a plurality of surface mesh sections from the depth information. When executed, the system also determines a plurality of texture sections corresponding to the plurality of surface mesh sections from the texture information and ranks the plurality of texture sections with respect to their respective color strengths. Adjust, add, or remove multiple characteristics of the light source, thereby changing one or more color intensities of the multiple texture sections ranked by color intensity, thereby improving the image A plurality of instructions configured to generate may be included.

  The system may further include at least one sensor configured to capture 3D images and associated depth and texture information. The plurality of instructions is further configured to determine a position of the one or more light sources. The instructions also determine a corresponding light source vector for each of the plurality of texture sections ranked by color intensity having a color intensity that is one of less than and greater than a predetermined threshold, A plurality of light source vectors may be mathematically combined together to thereby determine the position of one or more light sources. The instructions also adjust a plurality of characteristics of the at least one light source to provide brightness, color, hue, saturation, chroma, lightness, color intensity, and one or more of the one or more light sources. It can be configured to modify at least one of the luminances.

  Further, the instructions may be configured to adjust a plurality of characteristics of the one or more light sources, and further configured to add additional light sources and remove existing light sources. Multiple commands include multiple standard cameras, sensors, 2D or 3D printers, 2D or 3D displays, computer graphics systems or computer systems, a standard format, standard depth map format, standard camera output format, standard It may be further configured to output multiple surfaces ranked by color intensity in a media codec format, a standard computer graphics format, a standard 3D depth file format, or a standard protocol format.

  Although multiple flow diagrams and / or state diagrams have been used in this disclosure to describe multiple embodiments, it is understood that the inventions are not limited to those diagrams or the corresponding description of the disclosure. Should be. For example, the flow need not proceed through each illustrated box or state, nor does it need to proceed in exactly the same order as shown and described herein.

  The inventions are not limited to the specific details listed in this disclosure. Indeed, those skilled in the art will appreciate that many other variations may be made within the scope of the present invention from the foregoing description and drawings. Accordingly, what defines the scope of the invention is the following claims, including any modifications.

Claims (28)

An apparatus for enhancing the 3D image having depth information and texture information associated with the 3D image, comprising:
Means for generating a 3D mesh having a plurality of surface mesh sections, depending at least in part on the depth information;
Means for determining a plurality of texture sections corresponding to the plurality of surface mesh sections, depending at least in part on the texture information;
Means for ranking the plurality of texture sections and the corresponding plurality of surface mesh sections, depending at least in part on the color intensity of the plurality of texture sections;
Adjust multiple characteristics of one or more light sources to alter the color intensity for one or more of the texture sections ranked by the color intensity, thereby producing an improved image Means to
E Bei the <br/> and means for determining the position of the one or more light sources,
The means for determining the position of the one or more light sources depends on the ranked color intensity of the plurality of image textures and the corresponding plurality of surface mesh normal vectors, Determining the position of a plurality of light sources in the 3D image using a light source vector . The apparatus of claim 1, wherein the one or more light sources are added to the captured 3D image. The apparatus according to claim 1 or 2 , wherein the one or more light sources are in a 3D image when captured. The means for determining the one or more light source vectors includes determining a corresponding light source vector for each of the plurality of texture sections ranked by color intensity and mathematically combining the set of light source vectors together. An apparatus according to any one of claims 1 to 3 . The apparatus of claim 4 , wherein the color intensity of the plurality of texture sections ranked by color intensity includes a plurality of texture sections having a color intensity that is one of less than and greater than a predetermined threshold. The means for determining the one or more light source vectors includes a view vector, a vector perpendicular to each of the one or more texture intensity and surface mesh sections ranked by the color intensity, and the corresponding plurality of 6. The apparatus according to any one of claims 1 to 5, comprising determining an intermediate vector that is approximately in the middle between the surface vectors. Means for adjusting said plurality of characteristics of the one or more light sources, wherein at least one of brightness for one or more of the light source, color, hue, saturation, chroma, lightness, color intensity, of the luminance 7. The apparatus according to any one of claims 1 to 6 , comprising changing at least one. It means for adjusting said plurality of characteristics of the one or more light sources, according to any one of claims 1 7 comprising one of addition and removal of the at least one light source. The apparatus according to any one of claims 1 to 8 , further comprising means for re-rendering the surface mesh and a plurality of corresponding textures using a plurality of new light sources and outputting the resulting image. . Re-rendering the surface mesh and corresponding textures and outputting the resulting image is ranked by the color intensity in a predetermined standard format, standard depth map format, or standard camera output format. 10. The apparatus of claim 9 , further comprising providing at least one of the plurality of surfaces. 11. The apparatus according to any one of claims 1 to 10 , wherein the means for adjusting the plurality of characteristics of the at least one light source is performed prior to or in conjunction with the enhanced rendering of the image. The apparatus according to any one of claims 1 to 11 which is a printing device. The apparatus according to any one of claims 1 to 11 is an image capture mechanism. A program having a plurality of instructions, wherein the plurality of instructions are executed on the device in response to being executed on the device.
Extracting depth information and texture information associated with the 3D image;
Generating a surface mesh having a plurality of surface mesh sections from the depth information;
Determining a plurality of texture sections corresponding to the plurality of surface mesh sections from the texture information;
Ranking the plurality of texture sections and corresponding plurality of surface mesh sections with respect to their respective color strengths;
Adjust multiple characteristics of one or more light sources, thereby changing the color intensity for one or more of the plurality of texture sections ranked by the color intensity to generate an improved image the method comprising,
Determining the position of the one or more light sources;
Was executed,
The plurality of instructions for determining the position of the one or more light sources is executed on the device in response to being executed on the device,
Depending on the ranked color intensity of the plurality of image textures and the corresponding plurality of surface mesh normal vectors at least in part, the position of the plurality of light sources in the 3D image using one or more light source vectors. A program further comprising a plurality of instructions that cause the determining step to be executed . Wherein one or more of said plurality of instructions for adjusting the plurality of characteristics of the light source, in claim 14 including a plurality of instructions for adding or removing one or more light sources with respect to the 3D image The listed program. Adjusting said plurality of characteristics of the one or more light sources, additional, or wherein the plurality of instructions for removing, said plurality of characteristics of one or more light sources present in the 3D image when captured The program according to claim 14 or 15 , comprising a plurality of instructions for adjusting, adding or removing. The plurality of instructions for determining the position of the one or more light sources is executed on the device in response to being executed on the device,
Determining, for each of a plurality of texture sections ranked by color intensity, a corresponding light source vector for a corresponding plurality of surface mesh sections;
The program according to any one of claims 14 to 16, further comprising: a plurality of instructions for executing a step of mathematically combining the plurality of sets of light source vectors. The program of claim 17 , wherein the plurality of texture sections ranked by color intensity includes a plurality of texture sections having a color intensity that is one of less than and greater than a predetermined threshold. Adjusting said plurality of characteristics of the one or more light sources, additional, or wherein the plurality of instructions for removing, when executed on the device, the device, of the at least one light source 1 or brightness for multiple, color, hue, saturation, either chroma, brightness, color intensity, claims 14, further comprising a plurality of instructions for executing the steps of changing at least one of luminance 18 1 The program described in the section. The instructions for adjusting the plurality of characteristics of the one or more light sources, when executed on the device, cause the device to perform the steps of adding or removing at least one light source. The program according to any one of claims 14 to 19 , further comprising: When executed on the device, the device causes the device to perform a step of re-rendering the surface mesh and corresponding textures using a new light source to output the enhanced image. The program according to any one of claims 14 to 20 , further comprising an instruction. When executed on the device, the plurality of instructions for outputting the enhanced image includes a plurality of cameras, a plurality of sensors, a plurality of 2D or 3D printers, a plurality of 2D or 3D, Predefined standard formats, standard depth map formats, standard camera output formats, standard media codec formats, standard computer graphics formats, standard 3D depth files between displays, multiple computer graphics systems, or multiple computer systems The program according to claim 21 , further comprising: a plurality of instructions for executing a step of further providing at least one of a plurality of surfaces ranked by color intensity in a format or standard protocol format. A system comprising a central processing unit for executing a plurality of stored instructions and at least one memory device for storing a plurality of instructions readable by the central processing unit,
When the plurality of instructions are executed by the central processing unit,
Extracting depth information and texture information from the 3D image;
Generating a surface mesh having a plurality of surface mesh sections from the depth information;
Determining a plurality of texture sections corresponding to the plurality of surface mesh sections from the texture information;
Ranking the plurality of texture sections with respect to their respective color strengths;
Adjusting, adding, or removing characteristics of one or more light sources, thereby changing the color intensity for the one or more of the plurality of texture sections ranked by the color intensity, thereby and generating an enhanced image,
Determining the position of the one or more light sources;
It is configured to run,
The plurality of instructions are:
Depending on the ranked color intensity of the plurality of image textures and the corresponding plurality of surface mesh normal vectors at least in part, the position of the plurality of light sources in the 3D image using one or more light source vectors. A system further configured to cause the determining step to be performed . 24. The system of claim 23 , further comprising at least one sensor that captures a 3D image and the associated depth and texture information. The plurality of instructions are:
Determining a corresponding light source vector for each of a plurality of texture sections ranked by color intensity having a color intensity that is one of less than and greater than a predetermined threshold;
25. The system of claim 23 or 24 , further configured to mathematically combine a plurality of the light source vectors together to thereby determine the position of each of the light sources. The one or more light sources wherein the plurality of instructions configured to adjust a plurality of said characteristics of said one or more light sources of brightness, color, hue, saturation, chroma, lightness, color intensity, luminance The system according to any one of claims 23 to 25 , wherein at least one of them is modified. The plurality of instructions configured to adjust a plurality of the characteristics of the one or more light sources is further configured to add additional light sources and remove existing light sources. 27. The system according to any one of 23 to 26 . The plurality of instructions may be a predetermined standard format between a plurality of cameras, a plurality of sensors, a plurality of 2D or 3D printers, a plurality of 2D or 3D displays, a plurality of computer graphics systems, or a plurality of computer systems. Output a plurality of surfaces ranked by the color intensity in a standard depth map format, standard camera output format, standard media codec format, standard computer graphics format, standard 3D depth file format, or standard protocol format. The system according to any one of claims 23 to 27 , further configured.

Images