CN116708996B

CN116708996B - Photographing method, image optimization model training method and electronic equipment

Info

Publication number: CN116708996B
Application number: CN202310983622.XA
Authority: CN
Inventors: 田贤浩; 王瑞星; 王宇
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-08-07
Filing date: 2023-08-07
Publication date: 2023-11-17
Anticipated expiration: 2043-08-07
Also published as: CN116708996A

Abstract

The embodiment of the application provides a photographing method, an image optimization model training method and electronic equipment, and relates to the technical field of image processing, wherein the technical scheme of the embodiment of the application comprises the following steps: if the user triggering photographing operation is detected in the appointed photographing mode, the current illumination of the current photographing scene of the electronic equipment is obtained, and then the target frame reduction number corresponding to the target illumination range to which the current illumination belongs is determined according to the preset corresponding relation between the illumination range and the frame reduction number. And shooting a first number of frames RAW images, wherein the first number is the difference value between the default shooting frame number of the appointed shooting mode and the target frame number. And then, optimizing the first number of frames of RAW images by using a pre-trained image optimization model to obtain optimized images, and displaying the optimized images in a photographing interface. Thereby achieving a reduction in shooting time.

Description

Photographing method, image optimization model training method and electronic equipment

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a photographing method, an image optimization model training method, and an electronic device.

Background

Under the condition that the shooting scene is not ideal, the imaging effect of the camera is poor, so that the image quality of the shot image is low. For example, when a mobile phone shoots a night scene, because the illuminance of the night scene is low and the photosensitivity of a camera sensor (camera sensor) of the mobile phone is limited, the original (RAW) image shot by the mobile phone contains more noise and has low signal to noise ratio, so that the RAW image is directly optimized only by means of a traditional image signal processing (image signal processing, ISP) algorithm, and a high-quality night scene image is difficult to obtain.

In order to improve the image quality of the RAW image, a multi-frame denoising method is generally adopted. That is, RAW maps of a default photographing frame number are photographed by a camera, wherein exposure parameters used for photographing each RAW map are different, and the RAW maps are input to a deep learning network (deep nerual network), and then an optimized image output from the deep learning network is obtained. However, the more the number of frames of the RAW image input into the deep learning network, the more the reference basis is used when the deep learning network optimizes the image, so that the higher the image quality of the output optimized image, so that the mode requires the more frames of the RAW image to be shot at a time, for example, a night scene shooting scene, and generally requires a mobile phone to shoot ten frames of RAW images. But taking too many RAW maps takes a long time, for example, it is generally required to continuously take 3-4 seconds for the same scene.

Disclosure of Invention

The embodiment of the application aims to provide a photographing method, an image optimization model training method and electronic equipment, so as to reduce the photographing time required to be consumed in a multi-frame denoising mode. The specific technical scheme is as follows:

in a first aspect of an embodiment of the present application, a photographing method is provided, where the method may be applied to an electronic device, and the photographing method may include: if the shooting operation triggered by the user is detected in the appointed shooting mode, the current illumination of the current shooting scene of the electronic equipment is obtained; determining a target frame reduction number corresponding to a target illuminance range to which the current illuminance belongs according to a preset corresponding relation between the illuminance range and the frame reduction number; shooting a first number of frames RAW images, wherein the first number is the difference value between the default shooting frame number of the appointed shooting mode and the target frame reduction number; optimizing the first number of frames of RAW images by using a pre-trained image optimization model to obtain optimized images; and displaying the optimized image in a photographing interface.

It can be appreciated that the specified photographing mode may be a noise reduction photographing mode such as a night view photographing mode, a defogging photographing mode, or a raining photographing mode. In the appointed photographing mode, in the prior art, a RAW image with a default photographing frame number is obtained in each photographing, and the first number of the RAW images with the default photographing frame number is obtained in each photographing, namely, the number of the RAW images photographed at a time in the embodiment of the application is smaller, so that the photographing time is shorter.

The specific implementation of the photographing method may be described below with reference to fig. 1.

In the scheme provided by the application, the electronic equipment can determine the target frame reduction number corresponding to the current illuminance according to the current illuminance of the photographed scene, photograph RAW images of difference frames between the default photographing frame number and the target frame reduction number, and optimize the RAW images by utilizing a pre-trained image optimization model to obtain an optimized image. The more the illuminance of the environment is, the more the information carried in the RAW image is obtained, the higher the image quality is, and the fewer the RAW image needs to be input for optimization by the image optimization model is, so that the embodiment of the application can determine the target frame reduction number according to the current illuminance, dynamically reduce the frame number of the RAW image to be shot on the basis of guaranteeing the quality of the optimized image, and save the shooting time.

With reference to the first aspect, in one possible implementation manner, performing optimization processing on the first number of frame RAW graphs by using a pre-trained image optimization model to obtain an optimized image, including: and inputting the first number of frames RAW images and the zero value RAW images of the target reduced frame number frames into the image optimization model to obtain the optimized image output by the image optimization model.

It can be understood that the image size of the zero-value RAW image is the same as the size of the RAW image obtained by shooting, and the pixel value of each pixel included in the zero-value RAW image is 0.

Because the frame numbers of the RAW images shot each time may be different, and the image optimization model limits the total dimension of the input image to be a fixed dimension, the embodiment of the application inputs the RAW images of the first number of frames and the zero-value RAW images of the target frame reduction number of frames into the image optimization model together, so that the total dimension of the images of the input image optimization model is the same as the total dimension of the images of the RAW images of the default shooting frame number. That is, in the embodiment of the present application, after the number of shooting frames is reduced, the electronic device may still perform image optimization using the image optimization model.

With reference to the first aspect, in one possible implementation manner, the preset correspondence between the illuminance range and the reduced frame number is obtained by: acquiring a test RAW (random access) atlas corresponding to each test illuminance, wherein the test RAW atlas comprises a RAW image of a default shooting frame number obtained by shooting a test scene by the electronic equipment in a specified shooting mode; for a test RAW atlas corresponding to each test illuminance, respectively setting zero of each RAW image of the preset frame reduction number in the test RAW image set corresponding to the test illuminance to obtain a plurality of test samples corresponding to the test RAW atlas; inputting each test sample into the image optimization model to obtain a candidate image which is output after the image optimization model optimizes each test sample; determining the image quality of each candidate image; according to the image quality of candidate images corresponding to various preset frame reduction numbers under various test illumination, determining the corresponding relation between each illumination range and the frame reduction numbers; and determining the preset corresponding relation based on the corresponding relation between the illumination range and the reduced frame number obtained in each test scene included in the plurality of test scenes.

It will be appreciated that zeroing the test RAW atlas may be implemented as: randomly deleting the test RAW graphs with the preset frame reduction number from the group of test RAW atlases, and filling zero-value RAW graphs with the preset frame reduction number into the group of test RAW atlases; or randomly selecting a test RAW diagram with the preset frame reduction number from the group of test RAW diagrams, and setting the pixel value of each pixel included in the selected test RAW diagram to be 0.

According to the embodiment of the application, the image quality of the optimized candidate image corresponding to the different preset frame reduction numbers under different test illumination can be obtained for each test scene, so that the influence of the different preset frame reduction numbers under different test illumination on the image quality is obtained, and the preset frame reduction number with the largest frame reduction number under the condition of less influence on the image quality is selected in each illumination range to be used as the frame reduction number corresponding to the illumination range under the test scene. And then obtaining a preset corresponding relation by combining the number of reduced frames corresponding to each illumination range under each test scene. When photographing in an actual application scene, the target frame reduction number can be determined by utilizing the preset corresponding relation, so that the influence on the image quality of an optimized image is reduced under the condition of reducing the number of frames of a photographed RAW image. That is, the image quality of the optimized image is ensured while reducing the total time taken to take the RAW image.

The specific implementation process of determining the preset correspondence relationship may refer to the description below with respect to fig. 6.

With reference to the first aspect, in a possible implementation manner, before the determining the image quality of each candidate image, the method further includes: acquiring a reference image obtained by photographing the test scene with reference illuminance by the electronic equipment in a conventional photographing mode, wherein the reference illuminance is higher than the test illuminance;

the determining the image quality of each candidate image includes: determining, for each candidate image, an image gap between the candidate image and the reference image; and determining the image quality of the candidate image according to the image gap.

It can be understood that the reference illuminance is higher than each test illuminance, that is, the reference illuminance is higher, so that more information is included in one frame of image is obtained by shooting the test scene of the reference illuminance in the conventional shooting mode, less noise is included in the image, and the image quality is higher. Therefore, the smaller the image difference between the candidate image and the reference image is, the higher the image quality of the candidate image is, and on the contrary, the larger the image difference between the candidate image and the reference image is, the lower the image quality of the candidate image is. The embodiment of the application can determine the image quality of the candidate image by comparing the image difference between the candidate image and the reference image.

With reference to the first aspect, in a possible implementation manner, the determining an image gap between the candidate image and the reference image includes: extracting edges of the candidate images to obtain candidate edge images; performing edge extraction on the reference image to obtain a reference edge image; and determining edge detail difference parameters between the candidate edge image and the reference edge image to obtain an image difference between the candidate image and the reference image.

It can be understood that there is a difference between the image quality of the images photographed at different illuminances, i.e., the lower the illuminance, the lower the image quality of the images photographed. I.e. the illumination of the test scene affects the image quality of the candidate image as well, in addition to the number of frames. Therefore, in order to reduce the influence of illumination on the image quality, the candidate image and the reference image can be subjected to edge extraction respectively, and the two edge extraction results can be compared to obtain the image quality of the candidate image because the influence of illumination on the edge position of the object is small. That is, the embodiment of the application can reduce the influence of illumination on the image quality of the determined candidate image, thereby more accurately obtaining the influence of different reduced frame numbers on the image quality of the candidate image.

With reference to the first aspect, in one possible implementation manner, the edge detail difference parameter is a mean square error, a structural similarity or a peak signal-to-noise ratio between the candidate edge image and the reference edge image.

It can be appreciated that the specific form of the edge detail difference parameter can be selected according to the actual application scenario. For example, in the night scene photographing mode, the mean square error between the candidate edge image and the reference edge image is selected as the edge detail difference parameter. Therefore, the image quality can be more accurately determined under different application scenes.

With reference to the first aspect, in one possible implementation manner, the determining, according to image quality of candidate images corresponding to various preset reduced frame numbers under various test illuminances, a correspondence between each illuminance range and the reduced frame number includes: aiming at each preset illumination range, acquiring the image quality of candidate images under various test illumination in the preset illumination range; the image quality of the candidate images obtained through screening is higher than the preset frame reduction number of the designated image quality; and taking the largest preset frame reduction number in the screened preset frame reduction numbers as the frame reduction number corresponding to the preset illumination range.

It can be understood that, for the preset reduced frame numbers with the image quality higher than the designated image quality of the candidate images, the image quality of the obtained optimized image is higher by taking pictures in the illumination range by using the preset reduced frame numbers. Therefore, the maximum preset frame reduction number can be selected from the preset frame reduction numbers, so that the time consumption of single shooting is reduced as much as possible while the image quality of an optimized image is ensured.

With reference to the first aspect, in one possible implementation manner, the determining the preset correspondence based on the correspondence between the obtained illuminance range and the reduced frame number in each test scenario included in the multiple test scenarios includes: aiming at each illumination range, acquiring the frame reduction number corresponding to the preset illumination range under various test scenes; and taking the average value of the frame reduction numbers corresponding to the preset illumination ranges in the various test scenes as the frame reduction number corresponding to the illumination ranges.

It can be understood that taking the average value of the frame reduction numbers corresponding to each preset illumination range under various test scenes as the frame reduction number corresponding to the illumination range can reduce the situation that the corresponding relationship between the illumination range and the frame reduction number obtained based on a single test scene is inaccurate, thereby improving the accuracy of the preset corresponding relationship.

In a second aspect of the embodiment of the present application, there is provided an image optimization model training method, including: acquiring a plurality of groups of sample RAW images and training labels of each group of sample RAW images, wherein each group of sample RAW images is a RAW image of a default shooting frame number frame obtained by shooting a sample scene by electronic equipment in a specified shooting mode, and the training labels of each group of sample RAW images are standard optimized images corresponding to the group of sample RAW images; setting zero of RAW images of a random preset number of frames in the set of sample RAW images aiming at each set of sample RAW images, and optimizing the set of sample RAW images by utilizing an image optimization network to obtain sample optimization images; training the image optimization network according to the sample optimization image and the training labels of the group of sample RAW images, and taking the trained image optimization network as an image optimization model, wherein the image optimization model is used for optimizing a first number of frame RAW images shot in a specified shooting mode, the first number is the difference value between the default shooting frame number of the specified shooting mode and the target frame reduction number, and the target frame reduction number is the frame reduction number corresponding to the illumination range of a shooting scene when the first number of frame RAW images are shot.

It can be understood that the embodiment of the application can enable the prediction result of the image optimization network to be closer to the standard optimization result of each group of sample RAW graphs, and enable the prediction accuracy of the image optimization model obtained by training to be higher. Moreover, as the pixel value of each pixel point included in the zero-set RAW graph is 0, and the 0 value does not generate gradient in gradient propagation, so that the zero-set RAW graph is equivalent to training and prediction without participating in an image optimization network, the embodiment of the application carries out random zero setting on each group of sample RAW graphs, and the image optimization network can identify whether each frame in a default shooting frame number is deleted or not in training, so that the image optimization network can learn to distribute more weights on the RAW graph which is not deleted in the training process, thereby improving the optimization capability of the RAW graph. Therefore, the image optimization model obtained by training in the embodiment of the application can optimize the RAW atlas with the zero RAW image, and can obtain more accurate optimized images, so that the frame number of the RAW image shot at a time can be reduced when the electronic equipment is in actual shooting, and the time consumption for shooting is saved.

Specific implementations of image optimization model training may be found in the description below with respect to fig. 10.

In a third aspect of the embodiment of the present application, there is provided an electronic device, including: one or more processors and memory;

the memory is coupled with the one or more processors, the memory for storing computer program code comprising computer instructions that the one or more processors invoke the computer instructions to cause the electronic device to perform the method of any of the first or second aspects.

In a fourth aspect of embodiments of the present application, there is provided a computer readable storage medium comprising a computer program which, when run on an electronic device, causes the electronic device to perform the method according to any one of the first or second aspects.

In a fifth aspect of embodiments of the present application, there is provided a chip system for application to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause performance of the method according to any of the first or second aspects.

In a sixth aspect of embodiments of the application, there is also provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any of the first or second aspects described above.

It will be appreciated that the electronic device provided in the third aspect, the computer readable storage medium provided in the fourth aspect, the chip system provided in the fifth aspect, and the computer program product provided in the sixth aspect are all configured to perform the method provided by the present application. Therefore, the advantages achieved by the method are similar to those achieved by the method, and reference may be made to the advantages of the corresponding method, which will not be described here.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

Fig. 1 is a flowchart of a photographing method provided in an embodiment of the present application;

FIG. 2 is an exemplary schematic diagram of a first mobile phone display interface provided in an embodiment of the present application;

FIG. 3 is an exemplary diagram of a second mobile phone display interface according to an embodiment of the present application;

FIG. 4 is an exemplary schematic diagram of a third mobile phone display interface provided in an embodiment of the present application;

FIG. 5 is an exemplary schematic diagram of a fourth mobile phone display interface provided in an embodiment of the present application;

FIG. 6 is a flowchart of a method for determining a preset correspondence provided in an embodiment of the present application;

FIG. 7 is a graph of image quality versus luminance provided in an embodiment of the present application;

FIG. 8 is an exemplary schematic diagram of a photographing process provided in an embodiment of the present application;

fig. 9 is an effect diagram of a photographing process according to an embodiment of the present application;

FIG. 10 is a flowchart of an image optimization model training method provided in an embodiment of the present application;

FIG. 11 is an exemplary schematic diagram of a model input provided in an embodiment of the present application;

FIG. 12 is a schematic diagram of an electronic device according to an embodiment of the present application;

fig. 13 is a software architecture block diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

Image optimization is performed by using a deep learning network, and end-to-end multi-exposure fusion (multi-exposure fusion), demosaicing (demosaic), denoising (denoise) and other effects can be realized. However, when the deep learning model performs image optimization, the number of frames of the RAW image required to be input is fixed, and the number of frames of the RAW image required to be input is large, so that the shooting time required by the current image optimization is long, and a very obvious shooting dullness is brought to a user.

In order to reduce photographing time, the embodiment of the application provides a photographing method, which can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented Reality (Augmented Reality, AR)/Virtual Reality (VR) devices, notebook computers, ultra-Mobile Personal Computer, UMPC, netbooks, personal digital assistants (Personal Digital Assistant, PDA) or special cameras (e.g., single-lens reflex camera, card-type camera), and the embodiment of the application does not limit the specific types of the electronic devices.

As shown in fig. 1, the photographing method provided by the embodiment of the application includes the following steps:

s101, if the user triggering photographing operation is detected in the appointed photographing mode, the current illumination of the current photographing scene of the electronic equipment is obtained.

The shooting mode is designated as a shooting mode for image optimization by using a multi-frame RAW image. For example, the specified photographing mode is: and the noise reduction photographing modes such as a night scene photographing mode, a defogging photographing mode, a raining photographing mode and the like.

Alternatively, the electronic device may initiate the specified photographing mode when detecting that the user selects to initiate the specified photographing mode. Or if the electronic equipment detects that the illuminance of the current photographing scene is lower than the preset illuminance in the conventional photographing mode, starting the appointed photographing mode.

The triggering mode of the photographing operation comprises the following steps: clicking a shutter control displayed in a screen, pressing a button configured on an electronic device, or voice triggering, etc., as embodiments of the present application are not particularly limited.

The electronic device can detect illumination of a shooting scene through the built-in photosensitive module.

S102, determining a target frame reduction number corresponding to a target illuminance range to which the current illuminance belongs according to a preset corresponding relation between the illuminance range and the frame reduction number.

Wherein, the preset correspondence between the illuminance range and the reduced frame number may be manually preset or may be predetermined by a method described below.

S103, shooting a RAW image of a first number of frames. The first number is the difference between the default shooting frame number of the appointed shooting mode and the target frame number of the target frame reduction.

In the prior art, in the specified photographing mode, a RAW map with a default photographing frame number is obtained every time photographing. In the embodiment of the application, the first number of frames of RAW images are shot in the appointed shooting mode, and the number of frames shot each time is less than the number of frames shot in the prior art, so that the time consumption of single shooting is reduced.

S104, optimizing the RAW images of the first number of frames by using a pre-trained image optimization model to obtain optimized images.

S105, displaying the optimized image in the photographing interface.

According to the photographing method provided by the embodiment of the application, the target frame reduction number corresponding to the current illuminance can be determined according to the current illuminance of the photographed scene, RAW images of difference frames between the default photographing frame number and the target frame reduction number are photographed, and then the RAW images are optimized by utilizing a pre-trained image optimization model to obtain an optimized image. The more the illuminance of the environment is, the more the information carried in the RAW image is obtained, the higher the image quality is, and the fewer the RAW image needs to be input for optimization by the image optimization model is, so that the embodiment of the application can determine the target frame reduction number according to the current illuminance, dynamically reduce the frame number of the RAW image to be shot on the basis of guaranteeing the quality of the optimized image, and save the shooting time.

The photographing method according to the embodiment of the application is described below by taking an electronic device as a mobile phone and designating a photographing mode as a night scene photographing mode as an example:

it will be appreciated that the terms "interface" and "user interface" in the description and claims of the application and in the drawings are media interfaces for interaction and exchange of information between an application or operating system and a user, which enable conversion between an internal form of information and a form acceptable to the user. A commonly used presentation form of the user interface is a graphical user interface (graphic user interface, GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It may be an interface element such as an icon, a window, a control, etc. displayed in a display screen of the electronic device, where the control may include a visual interface element such as an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc.

Referring to fig. 2, fig. 2 is a default display interface of the mobile phone, where the interface includes a plurality of application icons, such as the office application icon, the taxi taking application icon, the camera application icon, the address book application icon, the phone application icon, and the album application icon in fig. 2. Other information may also be presented within the default display interface, such as the current time and date may also be presented within the default display interface of FIG. 2. When the mobile phone detects that the user clicks the camera application icon, the camera application is started in response to clicking operation of the user, and a photographing interface in a default photographing mode is displayed.

Referring to fig. 3, fig. 3 is a photo interface in a default photo mode, and in fig. 3, the default photo mode is a normal photo mode. As shown in fig. 3, the photo interface 300 may include a preview window 301, a shutter control 302, an album shortcut control 303, a camera flip control 304, and a camera mode option 305.

Preview frames may be shown within preview window 301. The preview frame displayed in the preview window 301 is a RAW image acquired by the camera of the mobile phone based on the view range.

The shutter control 302 may be used to trigger the shooting of an image.

Album shortcut control 303 may be used to open an album application. After the user triggers the mobile phone to start the album application program through the album shortcut control 303, the photographed image and video can be checked. In addition, thumbnail images of the captured images or videos may also be displayed on the album shortcut 303.

The camera flip control 304 may be used to select the preview frames shown within the preview window 301 as: the front camera of the mobile phone is based on the preview frames acquired in the view finding range, or the rear camera of the mobile phone is based on the preview frames acquired in the view finding range.

One or more shooting mode options may be displayed in the camera mode options 305. The one or more photography mode options may include: night scene mode options, portrait mode options, photo mode options, video mode options, and more options. It will be appreciated that more or fewer shooting mode options may also be included in the camera mode options 305.

After detecting that the user clicks the night scene mode option in the camera mode option 305, the mobile phone starts a night scene photographing mode in response to the clicking operation of the user. Or if the illuminance of the current photographing scene is detected to be lower than the preset illuminance in the conventional photographing mode, starting a night scene photographing mode. The normal photographing mode is a photographing mode corresponding to the photographing mode option in fig. 3.

And then, the mobile phone can acquire the illumination of the current photographing scene in real time in the preview process. Referring to fig. 4, in the night scene photographing mode, after detecting that a user clicks the shutter control 302, the mobile phone responds to the clicking operation of the user to obtain the current illuminance, determines the target frame reduction number corresponding to the target illuminance range to which the current illuminance belongs according to the preset correspondence between the illuminance range and the frame reduction number, and photographs a first number of frames of RAW images of the current photographing scene, wherein the first number is the difference between the default photographing frame number and the target frame reduction number of the night scene photographing mode. The exposure parameters used when the electronic device captures each RAW image are different.

And then, the mobile phone inputs the first number of frames of RAW images and the zero value RAW images of the target reduced frame number of frames into an image optimization model to obtain an optimized image which is output after the image optimization model optimizes the input RAW images. The zero-value RAW graph is the same as the RAW graph shot by the electronic equipment in size, and the pixel value of each pixel point is 0.

Thereafter, as shown in fig. 5, the mobile phone displays the optimized image in the preview window 301 of the photographing interface 300. It can be understood that compared with the image obtained by shooting the scene in the conventional shooting mode, the optimized image obtained in the night scene shooting mode is closer to the image obtained by shooting the scene in the daytime, so that noise reduction of the night scene image is realized, and the image quality of the image shot at night is improved.

It can be appreciated that, besides night scene shooting scenes, the embodiment of the present application may also be used for other scenes that need to implement noise reduction according to a multi-frame RAW image, for example, defogging, rain removing or other noise removing scenes, where the embodiment of the present application is not specifically limited to the application scene.

The embodiment of the application can also pre-determine the preset corresponding relation between the illumination range and the frame reduction number before shooting by using the dynamic frame reduction strategy, wherein the electronic equipment for determining the preset corresponding relation between the illumination range and the frame reduction number can be the same as or different from the electronic equipment applied to the shooting method.

For example, a preset correspondence between an illuminance range and a reduced frame number may be determined by using a mobile phone, and then the preset correspondence between the illuminance range and the reduced frame number determined by the mobile phone is synchronized to electronic devices such as other mobile phones, tablet computers, wearable devices and the like, so that each electronic device that obtains the preset correspondence may execute the photographing method provided by the embodiment of the present application.

Referring to fig. 6, the method for determining the preset correspondence relationship includes the steps of:

s601, acquiring a reference image obtained by photographing a test scene with reference illuminance under a conventional photographing mode by the electronic equipment.

The test scene is a scene with a large influence of illumination conditions on an imaging picture. For example, the test scene may be a park, a road, an underground parking garage, or a scene containing a building, etc., which is not particularly limited by the embodiments of the present application.

The reference illuminance is higher than the preset illuminance, that is, the reference illuminance is higher, so that the obtained reference image contains less noise and has higher image quality, and can be used as a fact library (groudtluth) for testing the RAW image.

S602, acquiring a test RAW atlas corresponding to each test illuminance. The test RAW atlas comprises a RAW chart with a default shooting frame number, wherein the electronic equipment shoots a test scene in a specified shooting mode.

Wherein the reference illuminance is higher than the various test illuminations. For example, the reference illuminance may be an illuminance of a test scene at any one of the daytime, the various test illuminations may be illuminations of the test scene at different moments from evening to late night, or the respective test illuminations may be illuminations of the test scene at different illuminations at night.

When one test RAW atlas is obtained through each shooting, the test illumination of the current test scene can be recorded, and the recorded test illumination is stored corresponding to the test RAW atlas.

When the electronic device shoots a test RAW image with a default shooting frame number each time, exposure parameters used for shooting the test RAW image of each frame are different.

S603, setting zero of each RAW graph of the preset frame reduction number in the RAW graph corresponding to each test illumination to obtain a plurality of test samples corresponding to the RAW graph.

Wherein, the number of various preset reduced frames is denoted as k,f is a default shooting frame number in a specified shooting mode.

For example, in the night scene photographing mode, the default photographing frame number is 10, and the various preset reduced frame numbers are respectively: 0,1, 2, 3, 4, 5. The method for obtaining the test samples based on each preset frame reduction number is the same, the description is made with the preset frame reduction number being 1, and 1 frame of test RAW image is randomly selected to be zero in 10 frames of test RAW images corresponding to each test illuminance, and the test RAW image after zero setting is used as one test sample. And so on, 6 test samples corresponding to each test RAW atlas are obtained.

In the embodiment of the present application, the method for setting zero for the test RAW graph with the preset reduced frame number in the set of test RAW graphs may be: randomly deleting the test RAW graphs with the preset frame reduction number from the group of test RAW atlases, and filling zero-value RAW graphs with the preset frame reduction number into the group of test RAW atlases; or randomly selecting a test RAW diagram with the preset frame reduction number from the group of test RAW diagrams, and setting the pixel value of each pixel included in the selected test RAW diagram to be 0.

S604, respectively inputting each test sample into an image optimization model to obtain candidate images which are output after the image optimization model optimizes each test sample.

S605 determines the image quality of each candidate image.

An image gap between the candidate image and the reference image may be determined for each candidate image, and an image quality of the candidate image may be determined based on the image gap.

In the embodiment of the application, for each candidate image, edge extraction can be performed on the candidate image to obtain a candidate edge image, and edge extraction can be performed on a reference image to obtain a reference edge image. Edge detail difference parameters between the candidate edge image and the reference edge image are then determined. The edge extraction may be performed on the image by using a laplacian-gaussian filter operator, or other edge extraction algorithms may also be used, which is not particularly limited in the embodiment of the present application.

Alternatively, the edge detail difference parameter may be a mean square error (Mean Squared Error, MSE), a structural similarity (Structural Similarity, SSIM), or a Peak Signal-to-Noise Ratio (PSNR) between the candidate edge image and the reference edge image. The specific form of the edge detail difference parameter can be selected according to the actual application scene.

S606, according to the image quality of candidate images corresponding to various preset frame reduction numbers under various test illumination, determining the corresponding relation between each illumination range and the frame reduction numbers.

The electronic device may acquire, for each preset illuminance range, image quality of candidate images under various test illuminations within the preset illuminance range, and then screen the acquired candidate images for a preset reduced frame number, where the image quality of the candidate images is higher than the specified image quality. And taking the largest preset frame reduction number in the screened preset frame reduction numbers as the frame reduction number corresponding to the preset illumination range.

Each preset illumination range can be obtained by manual analysis according to experience or according to the image quality of each candidate image. The specified image quality may be: and the image quality which can meet the service requirement is preset according to the actual scene.

During screening, the image quality of candidate images corresponding to the same preset reduced frame number under various test illumination can be fitted into a quality curve. For example, referring to fig. 7, fig. 7 is an image quality-illuminance graph, and the X-axis in fig. 7 represents Lux index (Lux-index) of the test environment, wherein the higher the Lux-index, the lower the illuminance of the test environment, and the lower the Lux-index, the higher the illuminance of the test environment; the Y-axis represents the image quality of the candidate image. In fig. 7, each curve is a quality curve, and the preset frame reduction numbers corresponding to each quality curve from top to bottom are 0, 1, 2, 3, and 4, respectively, and the image quality standard line represents the designated image quality under each illumination.

Then, for each preset illumination range, whether the image quality of each quality curve in the illumination range is higher than the designated image quality is judged. Screening out the preset frame reduction number corresponding to the quality curve with the judging result being yes, and selecting the largest preset frame reduction number from the screened preset frame reduction number as the frame reduction number corresponding to the preset illumination range. For example, referring to fig. 7, each preset illumination range in fig. 7 is: [0,300 ], [300,340 ], [340,400 ], [400,460 ]、. For the preset illumination range [0,300), the quality range of each quality curve is higher than the specified image quality in [0,300), so that the largest preset frame reduction number of the preset frame reduction numbers corresponding to each quality curve, namely 4, is taken as the frame reduction number corresponding to [0,300 ]. For the preset illumination range [300, 340), the quality ranges of the quality curves corresponding to the preset frame reduction numbers of 0-3 are higher than the specified image quality in the range [300, 340), so that the maximum preset frame reduction number, namely 3, is selected from 0-3 and is used as the frame reduction number corresponding to the range [300, 340). And so on, as shown in table one, the number of reduced frames corresponding to each illumination range is obtained.

List one

S607, determining a preset corresponding relation based on the corresponding relation between the obtained illumination range and the reduced frame number in each test scene included in the plurality of test scenes.

For each illumination range, an average value of the number of frames subtracted corresponding to the illumination range in various test scenes can be used as the number of frames subtracted corresponding to the illumination range, so that a preset corresponding relation between the illumination range and the number of frames subtracted is obtained.

Wherein, since the calculated average value may not be an integer, after the average value is calculated, the average value may be rounded up or rounded down, and the rounded result is taken as the number of frames subtracted corresponding to the illuminance range.

Through the method, the embodiment of the application can obtain the influence of different frame reduction numbers on the image quality under different test illumination through the test in advance, so that a preset frame reduction number with more frame reduction numbers and smaller influence on the image quality is selected as the frame reduction number corresponding to each illumination range. When photographing in an actual application scene, the dynamic frame reduction strategy can be executed by utilizing the preset corresponding relation between each illumination range and the frame reduction number, so that the influence on the image quality of an optimized image is reduced under the condition of reducing the frame number of a photographed RAW image. That is, the image quality of the optimized image is ensured while reducing the total time taken to take the RAW image.

Referring to fig. 8, the following describes the overall flow of the photographing method provided by the embodiment of the present application:

under the condition that the electronic equipment detects that a user triggers photographing operation in a specified photographing mode, the current illumination of a current photographing scene is obtained through a camera, and a target frame reduction number corresponding to a target illumination range to which the current illumination belongs is determined according to a dynamic frame reduction strategy and is recorded as k. And then shooting an (F-k) frame RAW image in a specified shooting mode, and inputting the (F-k) frame RAW image and the k frame zero value RAW image into an image optimization model to obtain an optimized image output by the image optimization model. The image optimization model is trained based on an image optimization network.

Referring to fig. 9, the following describes a photographing effect of the photographing method according to the embodiment of the present application in combination with an actual application scenario:

in fig. 9, the three images on the left represent the same shooting scene at different illuminations, and the numbers on the left of the images represent Lux-index of the shooting scene. For a shooting scene with the Lux-index of 400, determining the frame reduction number under the illumination based on a dynamic frame reduction strategy, namely, determining that the frame reduction number is 4 when the Lux-index is 400 based on a preset corresponding relation between the illumination range and the frame reduction number. Each square rectangle in fig. 9 represents a frame of RAW image obtained by photographing, and each black rectangle represents a frame of zero-value RAW image. And under a specified shooting mode, shooting 5 frames of RAW images of the scene, and inputting the 5 frames of RAW images and 4 frames of zero-value RAW images into an ISP preprocessing module to obtain a processing result of the ISP preprocessing module, wherein the ISP preprocessing module is used for carrying out white balance, dead point detection, black level correction and the like on the images. And then inputting the processing result of the ISP preprocessing module into an image optimization model to obtain an optimized image output by the image optimization model. And inputting the optimized image into an ISP post-processing module to obtain an output result of the ISP post-processing module, wherein the ISP post-processing module is used for performing gamma (gamma) correction, tone mapping (tone mapping) and/or dynamic range correction (Dynamic Range Correction, DRC) and the like on the image. And carrying out edge extraction on the output result of the ISP post-processing module to obtain an edge image.

And two edge images are obtained by the above process for the shooting scenes with the Lux-index of 450 and 500 respectively.

By comparing the three edge images, the three edge images have higher similarity, namely the embodiment of the application can obtain the optimized image with higher image quality for shooting scenes with different illumination.

Since the image optimization model needs to be trained in advance before image optimization is performed using the image optimization model. Therefore, based on the same inventive concept, the embodiment of the application also provides an image optimization model training method, which is applied to electronic equipment. The electronic device applied by the photographing method provided by the embodiment of the application and the electronic device applied by the image optimization model training method can be the same or different.

Referring to fig. 10, the image optimization model training method provided by the embodiment of the application includes the following steps:

s1001, acquiring a plurality of groups of sample RAW graphs and training labels of each group of sample RAW graphs.

Each group of sample RAW images is a RAW image of a default shooting frame number frame obtained by shooting a sample scene by the electronic equipment in a specified shooting mode. In order to improve generalization and accuracy of the image optimization model, sample scenes corresponding to each group of sample RAW graphs can be set to be different. Wherein, different sample scenes include the shooting scene that illuminance is different, and optionally, different sample scenes can also include: different shooting scenes of the included objects and/or different shooting angles, etc.

The training labels of each set of sample RAW maps are standard optimized images of the set of sample RAW maps. For example, for a sample scene corresponding to each group of sample RAW images, when the illuminance of the sample scene is higher than a preset illuminance, the electronic device takes a reference image obtained by photographing the sample scene in a conventional photographing mode as a standard optimized image of the group of RAW images.

Taking a night scene shooting scene as an example, the electronic device can shoot a sample scene by adopting different exposure parameters in a night scene shooting mode at night 8 to obtain an F frame RAW image, and take the F frame RAW image as 1 group of sample RAW images. F represents a default shooting frame number of a night scene shooting mode, and F can be preset according to the model of the electronic equipment and shooting software used. Moreover, the electronic device may take a photograph of the sample scene in the normal photographing mode at 9 a.m., and use the obtained image as a training tag of the set of sample RAW graphs.

S1002, setting zero of RAW images of a random preset number of frames in each group of sample RAW images, and optimizing the group of sample RAW images by using an image optimization network to obtain sample optimization images.

For each set of sample RAW patterns, k frames of sample RAW patterns may be randomly deleted from the set of sample RAW patterns, where Q represents a positive integer set, and F represents a default photographing frame number designating a photographing mode. And inputting the deleted sample RAW graph and the k-frame zero-value RAW graph into an image optimization network.

Or, for each group of sample RAW graphs, k frames of sample RAW graphs can be randomly selected from the group of sample RAW graphs, pixel values of all pixel points included in the selected sample RAW graphs are set to be 0, and the sample RAW graphs after zero setting are input into an image optimization network.

The image optimization network may be a convolutional neural network (Convolutional Neural Network, CNN), among others. For example, the image optimization network may be a visual geometry group (Visual Geometry Group, VGG) network, alexan network (AlexNet), google network (google net), or residual network (ResNet), among others.

Because the cyclic neural network (Recurrent Neural Network, RNN) is generally used for processing time sequence data, it is difficult to apply to various tasks in the computer vision field, and CNN can be widely applied to the computer vision field, so the embodiment of the application selects CNN as an image optimization network.

In addition, since CNN limits the dimension of input data to a fixed dimension, referring to fig. 11, each white diamond in fig. 11 represents one sample RAW graph, and the black diamond represents one zero-set RAW graph, so that the upper F frame is a set of sample RAW graphs, and the lower F frame represents (F-2) frame sample RAW graphs and 2-frame zero-set RAW graphs. With reference to fig. 11, in the embodiment of the present application, when the zero setting is performed on each set of sample RAW graphs, the total dimension of each set of sample RAW graphs is unchanged before and after the zero setting, so that the image optimization process can be performed on each set of sample RAW graphs before and after the zero setting by using an image optimization model.

S1003, training an image optimization network according to the sample optimization image and training labels of the set of sample RAW images, and taking the trained image optimization network as an image optimization model.

The image optimization model is used for performing optimization processing on a first number of frames of RAW images shot in a specified shooting mode, wherein the first number is a difference value between a default shooting frame number of the specified shooting mode and a target frame reduction number, and the target frame reduction number is a frame reduction number corresponding to an illumination range to which illumination of a shooting scene belongs when the first number of frames of RAW images are shot.

During training, a loss value can be calculated according to the sample optimization image of each group of sample RAW images and the training label of the group of sample RAW images by using a preset loss function, and whether the image optimization network is converged or not is judged based on the loss value. When the image optimization network is not converged, adopting a gradient descent mode, adjusting network parameters of the image optimization network by using the loss value, and performing the next training. And when the image optimization network converges, taking the current image optimization network as an image optimization model.

Optionally, it may be determined whether the calculated loss value is smaller than a preset loss value, if yes, the image optimization network is determined to converge, otherwise, it is determined that the image optimization network is not converged. Or, whether the difference between the calculated loss value and the calculated loss value in the previous training is smaller than the preset difference or not can be judged, if yes, the convergence of the image optimization network is determined, and if not, the non-convergence of the image optimization network is determined. Or, whether the image optimization network converges or not may also be determined by other manners, which is not particularly limited in the embodiment of the present application.

Through the method, the prediction result of the image optimization network can be more similar to the standard optimization result of each group of sample RAW graphs, namely, the prediction accuracy of the image optimization model obtained through training is higher. In addition, with reference to fig. 11, since the pixel values of each pixel point included in the zero-set RAW image are all 0, and no gradient is generated in the gradient propagation of the 0 values, the zero-set RAW image is equivalent to the RAW image which does not participate in the training and prediction of the image optimization network, so that the embodiment of the application randomly zero-sets each group of sample RAW images, and the image optimization network can recognize whether each frame in the default shooting frame number is pruned in the training, so that the image optimization network can learn to distribute more weights on the non-pruned RAW image in the training process, thereby improving the optimizing capability of the RAW image. Therefore, the image optimization model obtained by training in the embodiment of the application can optimize the RAW atlas with the zero RAW image, and can obtain more accurate optimized images, so that the frame number of the RAW image shot at a time can be reduced when the electronic equipment is in actual shooting, and the time consumption for shooting is saved.

Next, the form and the software and hardware architecture of the electronic device provided in the embodiment of the present application will be described.

Fig. 12 exemplarily shows a hardware structure of an electronic device according to an embodiment of the present application. As shown in fig. 12, the electronic device may include: processor 110, external memory interface 120, internal memory 126, camera 130, display 140, audio module 150, speaker 150A, receiver 150B, microphone 150C, headset interface 150D, and sensor module 160. Wherein the sensor module 160 may include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, etc.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller can be a neural center and a command center of the electronic device. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universalasynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

In the embodiment of the present application, the processor 110 may be configured to run a camera application, and the processor 110 is configured to control the camera application to perform the photographing function described in the foregoing embodiment, so as to implement the photographing method provided by the present application.

The internal memory 126 may include one or more random access memories (random access memory, RAM) and one or more non-volatile memories (NVM).

The random access memory may include a static random-access memory (SRAM), a dynamic random-access memory (dynamic random access memory, DRAM), a synchronous dynamic random-access memory (synchronous dynamic random access memory, SDRAM), a double data rate synchronous dynamic random-access memory (double data rate synchronous dynamic random accessmemory, DDR SDRAM, such as fifth generation DDR SDRAM is commonly referred to as DDR5 SDRAM), etc.;

the nonvolatile memory may include a disk storage device, a flash memory (flash memory).

The FLASH memory may include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. divided according to an operation principle, may include single-level memory cells (SLC), multi-level memory cells (MLC), triple-level memory cells (TLC), quad-level memory cells (QLC), etc. divided according to a storage specification, may include universal FLASH memory (english: universal FLASH storage, UFS), embedded multimedia memory cards (embedded multi media Card, eMMC), etc. divided according to a storage specification.

The random access memory may be read directly from and written to by the processor 110, may be used to store executable programs (e.g., machine instructions) for an operating system or other on-the-fly programs, may also be used to store data for users and applications, and the like.

The nonvolatile memory may store executable programs, store data of users and applications, and the like, and may be loaded into the random access memory in advance for the processor 110 to directly read and write.

The external memory interface 120 may be used to connect external non-volatile memory to enable expansion of the memory capabilities of the electronic device. The external nonvolatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and video are stored in an external nonvolatile memory. In some embodiments, the electronic device may include 1 or N external memory interfaces 120, N being a positive integer greater than 1.

In an embodiment of the present application, computer program code corresponding to the photographing function may be stored in the NVM. After the photographing function is turned on, the corresponding program code may be loaded into the RAM. The processor 110 can directly read the program codes in the RAM to realize the photographing function. In addition, the RAW map obtained by the photographing function can also be written into the NVM for storage for viewing by the user.

The electronic device implements display functions through the GPU, the display screen 140, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 140 is used to display images, videos, and the like. The display screen 140 includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD). The display panel may also be manufactured using organic light-emitting diode (OLED), active-matrix organic light-emitting diode (AMOLED), flexible light-emitting diode (flex-emitting diode), mini, micro-OLED, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), or the like. In some embodiments, the electronic device may include 1 or N display screens 140, N being a positive integer greater than 1.

The electronic device may implement shooting functions through an ISP, a camera 130, a video codec, a GPU, a display screen 140, an application processor, and the like.

The ISP is used to process the data fed back by the camera 130. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 130.

The camera 130 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device may include 1 or N cameras 130, N being a positive integer greater than 1.

In an embodiment of the present application, the camera 130 may be a tele camera or a main camera, a wide-angle camera, etc.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, and so on.

Video codecs are used to compress or decompress digital video. The electronic device may support one or more video codecs. In this way, the electronic device may play or record video in a variety of encoding formats, such as: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent cognition of electronic devices can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc. Wherein image recognition includes, for example, identifying a person, animal, action, etc., for determining whether conditions for automatically capturing a scene are met.

The electronic device may implement audio functions through an audio module 150, a speaker 150A, a receiver 150B, a microphone 150C, an earphone interface 150D, an application processor, and the like. Such as music playing, recording, etc.

The structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic apparatus. In other embodiments of the application, the electronic device may include more or less components than illustrated, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may also include keys, motors, indicators, and subscriber identity module (subscriber identification module, SIM) card interfaces, etc. For another example, the sensor module may further include: a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a fingerprint sensor, a temperature sensor, a bone conduction sensor, and the like.

Fig. 13 illustrates a software architecture of an electronic device according to an embodiment of the present application. As shown in fig. 13, the layered architecture divides the system into several layers, each with a clear role and division. The layers communicate with each other through a software interface. In some embodiments, the system is divided into five layers, from top to bottom, an application layer, an application framework layer (frame work), a hardware abstraction layer (hardware abstract layer, HAL), a driver layer, and a hardware layer, respectively. Wherein:

The application layer (application) may comprise a series of application packages. For example, the application packages may include applications such as camera applications, office applications, and courier applications. Camera applications may include, but are not limited to: UI module, photographing module, image storage module, etc. The UI module may be responsible for man-machine interaction of the camera application, for example, controlling display of the preview interface and preview screen therein, and monitoring and responding to user operations occurring in the preview interface. The photographing module may include a general photographing module, a night scene photographing module, to provide a general photographing function, a night scene photographing function, etc. The image storage module can be used for storing the photographed images in a file system or a specific database of the electronic equipment for being called by applications such as an image library.

The application framework layer may relate primarily to the camera framework, may include camera management and camera device camera access interfaces, and may function as a top-down interface, may interact with camera applications via an application programming interface (application program interface, API), and may interact with HAL via HAL interface definition language (HAL interfacedefinition language, HIDL).

The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, which aims at abstracting the hardware. The hardware interface details of a specific platform are hidden, a virtual hardware platform is provided for an operating system, so that the operating system has hardware independence, and can be transplanted on various platforms.

The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driving layer may include a display driver, a camera driver, a digital signal processor driver, an image processor driver, and the like. Wherein the display driver is used for driving the display screen to display preview images or optimized images and the like. The camera drives an image sensor for driving one or more cameras in the camera module to acquire images and drives an image signal processor to preprocess the images. The digital signal processor driver is used for driving the digital signal processor to process the image. The image processor driver is used for driving the image processor to process the image.

The hardware layer may include a display screen, a camera module, an image signal processor, a digital signal processor, an image processor, and the like. The camera module may include one or more camera image sensors (e.g., image sensor 1, image sensor 2, etc.) therein. Optionally, a time of flight (TOF) sensor, a multispectral sensor, etc. may also be included in the camera module.

The present application also provides an electronic device, which may include: one or more processors and memory; the memory is coupled to one or more processors and the memory is for storing computer program code that includes computer instructions that the one or more processors call to cause the electronic device to perform any one of the method embodiments described above.

The application also provides a chip system comprising at least one processor for implementing the functions involved in the method performed by the electronic device in any of the above embodiments.

In one possible design, the system on a chip further includes a memory to hold program instructions and data, the memory being located either within the processor or external to the processor.

The chip system may be formed of a chip or may include a chip and other discrete devices.

Alternatively, the processor in the system-on-chip may be one or more. The processor may be implemented in hardware or in software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general purpose processor, implemented by reading software code stored in a memory.

Alternatively, the memory in the system-on-chip may be one or more. The memory may be integral with the processor or separate from the processor, and embodiments of the present application are not limited. The memory may be a non-transitory processor, such as a ROM, which may be integrated on the same chip as the processor, or may be separately provided on different chips, and the type of memory and the manner of providing the memory and the processor are not particularly limited in the embodiments of the present application.

Illustratively, the chip system may be a field programmable gate array (field programmable gate array, FPGA), an application specific integrated chip (application specific integrated circuit, ASIC), a system on chip (SoC), a central processing unit (central processor unit, CPU), a Network Processor (NP), a digital signal processing circuit (digital signal processor, DSP), a microcontroller (micro controller unit, MCU), a programmable controller (programmable logic device, PLD) or other integrated chip.

In still another embodiment of the present application, a computer readable storage medium is provided, where a computer program is stored, where the computer program, when executed by a processor, implements the photographing method and the image optimization model training method according to any one of the foregoing embodiments.

In yet another embodiment of the present application, a computer program product containing instructions that, when run on a computer, cause the computer to perform the photographing method, the image optimization model training method of any of the above embodiments is also provided.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the electronic device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A photographing method, applied to an electronic device, the method comprising:

if the shooting operation triggered by the user is detected in the appointed shooting mode, the current illumination of the current shooting scene of the electronic equipment is obtained;

determining a target frame reduction number corresponding to a target illuminance range to which the current illuminance belongs according to a preset corresponding relation between the illuminance range and the frame reduction number;

shooting a first number of RAW images, wherein the first number is the difference value between the default shooting frame number of the appointed shooting mode and the target frame reduction number;

optimizing the first number of RAW images by using a pre-trained image optimization model to obtain optimized images;

displaying the optimized image in a photographing interface;

the preset corresponding relation between the illumination range and the frame reduction number is obtained through the following steps:

acquiring a test RAW (random access) atlas corresponding to each test illuminance under a test scene, wherein each test RAW atlas comprises a RAW image of a default shooting frame number obtained by shooting the test scene under a specified shooting mode by the electronic equipment;

For a test RAW atlas corresponding to each test illuminance, respectively setting zero of each RAW image of the preset frame reduction number in the test RAW image set corresponding to the test illuminance to obtain a plurality of test samples corresponding to the test RAW atlas;

inputting each test sample into the image optimization model to obtain a candidate image which is output after the image optimization model optimizes each test sample;

determining the image quality of each candidate image;

according to the image quality of candidate images corresponding to various preset frame reduction numbers under various test illumination, determining the corresponding relation between each illumination range and the frame reduction numbers;

determining the preset corresponding relation based on the corresponding relation between the illumination range and the reduced frame number obtained in each test scene included in a plurality of test scenes;

according to the image quality of candidate images corresponding to various preset frame reduction numbers under various test illumination, determining the corresponding relation between each illumination range and the frame reduction numbers comprises the following steps:

aiming at each preset illumination range, acquiring the image quality of candidate images under various test illumination in the preset illumination range;

the image quality of the candidate images obtained through screening is higher than the preset frame reduction number of the designated image quality;

And taking the largest preset frame reduction number in the screened preset frame reduction numbers as the frame reduction number corresponding to the preset illumination range.

2. The method of claim 1, wherein optimizing the first number of RAW maps using a pre-trained image optimization model to obtain an optimized image comprises:

and inputting the first number of RAW graphs and the zero value RAW graphs of the target frame reduction number into the image optimization model to obtain the optimized image output by the image optimization model.

3. The method of claim 1, wherein prior to said determining the image quality of each candidate image, the method further comprises:

acquiring a reference image obtained by photographing the test scene with reference illuminance by the electronic equipment in a conventional photographing mode, wherein the reference illuminance is higher than the test illuminance, and the conventional photographing mode is different from the appointed photographing mode;

the determining the image quality of each candidate image includes:

determining, for each candidate image, an image gap between the candidate image and the reference image;

and determining the image quality of the candidate image according to the image gap.

4. A method according to claim 3, wherein said determining an image gap between the candidate image and the reference image comprises:

extracting edges of the candidate images to obtain candidate edge images;

performing edge extraction on the reference image to obtain a reference edge image;

and determining edge detail difference parameters between the candidate edge image and the reference edge image to obtain an image difference between the candidate image and the reference image.

5. The method of claim 4, wherein the edge detail difference parameter is a mean square error, a structural similarity, or a peak signal-to-noise ratio between the candidate edge image and the reference edge image.

6. The method according to any one of claims 1 to 5, wherein the determining the preset correspondence based on the correspondence between the illuminance range obtained in each of the plurality of test scenes and the reduced number of frames includes:

aiming at each illumination range, acquiring the frame reduction number corresponding to the illumination range under various test scenes;

and taking the average value of the frame reduction numbers corresponding to the illumination range in the various test scenes as the frame reduction number corresponding to the illumination range.

7. A method for training an image optimization model, the method comprising:

acquiring a plurality of groups of sample RAW images and training labels of each group of sample RAW images, wherein each group of sample RAW images is a RAW image of a default shooting frame number obtained by shooting a sample scene by electronic equipment in a specified shooting mode, and the training labels of each group of sample RAW images are standard optimized images corresponding to the group of sample RAW images;

setting zero random preset number of RAW images in the group of sample RAW images aiming at each group of sample RAW images, and optimizing the group of sample RAW images by utilizing an image optimization network to obtain sample optimization images;

training the image optimization network according to the sample optimization image and training labels of the group of sample RAW images, taking the trained image optimization network as an image optimization model, wherein the image optimization model is used for performing optimization processing on a first number of RAW images shot in a specified shooting mode, the first number is a difference value between a default shooting frame number of the specified shooting mode and a target frame reduction number, the target frame reduction number is a frame reduction number corresponding to a determined target illumination range according to a preset corresponding relation between each illumination range and the frame reduction number, and the target illumination range is an illumination range to which illumination of a shooting scene belongs when the first number of RAW images are shot;

The preset corresponding relation between each illumination range and the reduced frame number is obtained through the following steps:

determining the image quality of each candidate image;

8. An electronic device, comprising:

a memory and one or more processors;

the memory is coupled with the one or more processors, the memory for storing computer program code comprising computer instructions that the one or more processors invoke to cause the electronic device to perform the method of any of claims 1-7.

9. A computer readable storage medium comprising a computer program which, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 7.

10. A chip system for application to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause performance of the method of any of claims 1 to 7.