WO2022007862A1

WO2022007862A1 - Image processing method, system, electronic device and computer readable storage medium

Info

Publication number: WO2022007862A1
Application number: PCT/CN2021/105060
Authority: WO
Inventors: 张梦然; 陈泰雨; 薛蓬; 张运超
Original assignee: 华为技术有限公司
Priority date: 2020-07-08
Filing date: 2021-07-07
Publication date: 2022-01-13
Also published as: CN113935898A

Abstract

An image processing method, a system, an electronic device and a computer readable storage medium. The present invention is applicable to the field of artificial intelligence technology. In the image processing method, a first electronic device renders native image data by means of first graphics rendering hardware, to obtain a first image (S301); and the first electronic device performs super-resolution reconstruction on the first image by means of second graphics rendering hardware to obtain a target image (S302), the first graphics rendering hardware and the second graphics rendering hardware being different graphics rendering hardware. Compared with the manner of directly rendering native image data into a target image by using single graphics rendering hardware, said method reduces the rendering power consumption for rendering a high-quality image, reduces the amount of calculation, improves the rendering efficiency, and fully uses hardware resources of heterogeneous graphics rendering hardware in an electronic device, solving the problem in an existing image solution that when the hardware resources of the electronic device are insufficient, the electronic device can only run a high-quality product with a relatively low image quality.

Description

Image processing method, system, electronic device, and computer-readable storage medium

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on July 8, 2020, the application number is 202010653106.7, and the application name is "image processing method, system, electronic device and computer-readable storage medium", all of which are The contents are incorporated herein by reference.

technical field

The present application belongs to the field of artificial intelligence, and in particular, relates to an image processing method, system, electronic device, and computer-readable storage medium.

Background technique

There are more and more products with fine pictures on the market today, such as high-definition games and videos.

When an electronic device renders images of these products, due to high rendering power consumption and a large amount of calculation, the hardware resource requirements of the electronic device are relatively high. However, when the hardware resources of the electronic device are insufficient, the electronic device can only run these products with a lower picture quality, which seriously affects the picture effect of the product.

SUMMARY OF THE INVENTION

The present application provides an image processing method, a system, an electronic device and a computer-readable storage medium, which solves the problem that in the existing image solution, when the hardware resources of the electronic device are insufficient, the electronic device can only run at a low image quality and high Problems with image quality products.

To achieve the above object, the application adopts the following technical solutions:

In a first aspect, an image processing method is provided, applied to a first electronic device, including:

The first electronic device acquires native image data, where the native image data is unrendered image data generated by an application;

The first electronic device renders the native image data through the first graphics rendering hardware to obtain a first image;

The first electronic device performs super-resolution reconstruction on the first image through second graphics rendering hardware to obtain a target image, and the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardware.

It should be noted that the application generates native image data during the running process.

After acquiring the native image data, the first electronic device may render the native image data through the first graphics rendering component, so that the native image data is converted into visible pixels to obtain the first image.

Afterwards, the first electronic device may perform super-resolution reconstruction on the first image through the second graphics rendering component to obtain a high-quality target image.

Compared with the method of directly rendering native image data into a high-quality target image, the image processing method of the present application obtains the target image by combining preliminary rendering and super-resolution reconstruction, which can reduce the time required for electronic devices to render high-quality images. Rendering power and computation.

In addition, the specific types of the first graphics rendering component and the second graphics rendering component may be a central processing unit (CPU), a graphics processing unit (graphics processing unit, GPU), and a neural-network processing unit (neural-network processing units). , NPU) one or more of.

In addition, the first graphics rendering component and the second graphics rendering component are different graphics rendering components, so as to make full use of the heterogeneous graphics rendering components inside the electronic device, and avoid the electronic device with a lower hardware resource due to insufficient hardware resources of a single graphics rendering hardware. Image quality runs high-quality products.

In a possible implementation manner of the first aspect, the first electronic device performs super-resolution reconstruction on the first image through first graphics rendering hardware to obtain a target image, including:

The first electronic device obtains the identifier of the application;

the first electronic device searches for a target super-score model associated with the identifier;

The first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware and the found target super-resolution model to obtain a target image.

It should be noted that, the first electronic device may perform super-resolution reconstruction on the first image by using the super-resolution model.

The super-score model may include a specific over-score model and a general over-score model. The specific super-resolution model is only suitable for some applications, and its applicability is poor, but it has a high image quality optimization ability. The general super-score model has high applicability, but the ability to optimize image quality is limited.

Since the application scope of a specific super-resolution model is limited, the electronic device may pre-establish an association relationship between the specific super-resolution model and its applicable application program.

After acquiring the first image, the first electronic device may acquire the identifier of the application, and search for a target super-score model associated with the identifier.

If the first electronic device can find the target super-score model associated with the above identifier, it means that there is a specific super-score model (ie, the target super-score model) suitable for the above application in the first electronic device. The first electronic device may perform super-resolution reconstruction on the first image by using the above-mentioned first graphics rendering hardware and the above-mentioned target super-resolution model.

The electronic device uses the target super-resolution model to perform super-resolution reconstruction on the first image, which can better improve the image quality of the target image.

In addition, the identifier of the application program may be the package name of the application program, or the identifier of the application program may also be a user-defined identifier.

In a possible implementation manner of the first aspect, after the first electronic device searches for a target super-score model associated with the identifier, the method further includes:

If the target super-resolution model associated with the identifier is not found, the first electronic device performs super-resolution reconstruction on the first image by using the first graphics rendering hardware and a preset general super-resolution model, Get the target image.

It should be noted that, if the first electronic device cannot find the target super-scoring model associated with the above identifier, it means that there is no specific super-scoring model associated with the application.

At this time, the first electronic device may perform super-resolution reconstruction on the first image by using the first graphics rendering hardware and the preset general super-resolution model to obtain the target image.

In a possible implementation manner of the first aspect, after the first electronic device performs super-resolution reconstruction on the first image by using a preset general super-resolution model to obtain the target image, the method further includes:

The first electronic device establishes an association relationship between the identifier and the general super-resolution model.

It should be noted that when the first electronic device uses the general super-resolution model to perform super-resolution reconstruction on the first image, since the first electronic device may be set with multiple general super-resolution models, in order to make the application program next time When activated, the first electronic device may invoke the same general super-resolution model to perform super-resolution reconstruction on the first image, and the first electronic device may establish an association relationship between the above-mentioned identifier and the general super-resolution model.

When the application program is started next time, the first electronic device can find the general super-score model according to the association relationship between the above-mentioned identifier and the general super-score model, determine the general-purpose super-score model as the target The super-resolution model performs super-resolution reconstruction on the first image, so that the first electronic device can maintain the same level of image quality optimization when processing the image of the application.

In a possible implementation manner of the first aspect, the first electronic device renders the native image data by using first graphics rendering hardware to obtain a first image, including:

The first electronic device renders the native image data by using the first graphics rendering hardware and a preset first image resolution to obtain a first image.

It should be noted that, when performing preliminary rendering, the first electronic device may render native image data according to the first image resolution. The first image resolution is preset by the first electronic device.

In a possible implementation manner of the first aspect, the first electronic device performs super-resolution reconstruction on the first image by using second graphics rendering hardware to obtain a target image, including:

The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware and the single-enhanced super-resolution model to obtain a target image, wherein the first image resolution and the The image resolutions of the target images are the same, and the single-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image and the image resolution of the output image are the same.

It should be noted that, if the resolution of the first image is consistent with the image resolution of the target image, the first electronic device may select the single-enhanced super-resolution model when selecting the super-resolution model to perform super-resolution reconstruction on the first image. .

For the haplo-enhanced super-resolution model, the image resolution of the input image of the model is the same as the image resolution of the output image of the model.

The first electronic device performs super-resolution reconstruction on the first image by using the second graphics rendering hardware and the multiple-enhanced super-resolution model to obtain a target image, wherein the resolution of the first image is smaller than the resolution of the first image. The image resolution of the target image. The multiple-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is smaller than the image resolution of the output image.

It should be noted that, if the resolution of the first image is smaller than that of the target image, the electronic device may select a multiple-enhanced super-resolution model to perform super-resolution reconstruction on the first image.

For a multiple-enhanced super-resolution model, the image resolution of the input image of the model is smaller than the image resolution of the output image of the model. That is, the multi-fold enhanced super-resolution model can increase the image resolution of the input image.

The first electronic device renders the native image data through a graphics processor to obtain a first image;

Correspondingly, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image, including:

The first electronic device performs super-resolution reconstruction on the first image through a neural network processor to obtain a target image.

It should be noted that the first electronic device processes the native image data through heterogeneous graphics rendering hardware to obtain the target image. Specifically, the first electronic device may render the original image data through a graphics processor to obtain a first image, and perform super-resolution reconstruction on the first image through a neural network processor.

The first electronic device selects appropriate graphics rendering hardware to perform corresponding operations, which can improve the image processing efficiency of the first electronic device.

In a second aspect, an image processing method is provided, applied to a second electronic device, including:

The second electronic device receives a first image sent by the first electronic device, where the first image is an image obtained by the first electronic device rendering the native image data generated by the application;

The second electronic device performs super-resolution reconstruction on the first image to obtain a target image.

It should be noted that when the user wishes to project the screen display of the first electronic device to the second electronic device, the first electronic device can locally render the native image data of the application program to obtain the first image.

Then, the first electronic device sends the first image to the second electronic device.

The second electronic device receives the first image, and performs super-resolution reconstruction on the first image to obtain a high-quality target image.

In the process of obtaining the target image, the first electronic device and the second electronic device jointly perform image processing, which can make full use of the hardware resources of the graphics rendering hardware of different electronic devices, thereby reducing the need for the first electronic device when rendering high-quality images. The load of the local hardware resources reduces the rendering consumption of the first electronic device for rendering high-quality images, and makes full use of the hardware resources of different electronic devices to better improve the image quality, thereby improving the user experience.

In a possible implementation manner of the second aspect, the second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

acquiring, by the second electronic device, the identifier of the application;

the second electronic device searches for a target super-score model associated with the identifier;

The second electronic device performs super-resolution reconstruction on the first image by using the found target super-resolution model to obtain a target image.

It should be noted that, the second electronic device may perform super-resolution reconstruction on the second image by using the super-resolution model.

After acquiring the second image, the second electronic device may acquire the identifier of the application, and search for the target super-score model associated with the identifier.

If the second electronic device can find the target super-score model associated with the above identifier, it means that there is a specific super-score model (ie, the target super-score model) suitable for the above application in the second electronic device, and the second electronic device can use the graph The rendering hardware and the target super-resolution model described above perform super-resolution reconstruction of the second image.

The electronic device uses the target super-resolution model to perform super-resolution reconstruction on the second image, which can better improve the image quality of the target image.

In a possible implementation manner of the second aspect, after the second electronic device searches for the target super-score model associated with the identifier, the method further includes:

If the target super-resolution model associated with the identifier is not found, the second electronic device performs super-resolution reconstruction on the first image by using a preset general super-resolution model to obtain a target image.

It should be noted that, if the second electronic device cannot find the target super-score model associated with the above identifier, it means that there is no specific super-score model associated with the application.

At this time, the second electronic device may perform super-resolution reconstruction on the second image by using the graphics rendering hardware and the preset general super-resolution model to obtain the target image.

In a possible implementation manner of the second aspect, after the second electronic device performs super-resolution reconstruction on the first image by using a preset general super-resolution model to obtain the target image, the method further includes:

The second electronic device establishes an association relationship between the identifier and the general super-resolution model.

It should be noted that when the second electronic device uses the general super-resolution model to perform super-resolution reconstruction on the second image, since the second electronic device may be set with multiple general super-resolution models, in order to make the application program next time When activated, the second electronic device may invoke the same general super-resolution model to perform super-resolution reconstruction on the second image, and the second electronic device may establish an association relationship between the above-mentioned identifier and the general super-resolution model.

When the application program is started next time, the second electronic device can find the general super-score model according to the association relationship between the above-mentioned identifier and the general super-score model, determine the general super-score model as the target The super-resolution model performs super-resolution reconstruction on the second image, so that the second electronic device can maintain the same level of image quality optimization when processing the image of the application.

In a possible implementation manner of the second aspect, the first image resolution of the first image is consistent with the image resolution of the target image;

The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

The second electronic device performs super-resolution reconstruction on the first image through a single-enhanced super-resolution model to obtain the target image, wherein the single-enhanced super-resolution model is the image resolution of the input image A superresolution model with the same image resolution as the output image.

It should be noted that, if the resolution of the first image is the same as the image resolution of the target image, the second electronic device may select the single-enhanced super-resolution model when selecting the super-resolution model to perform super-resolution reconstruction on the first image. .

The image resolution of the input image of the haplo-enhanced super-resolution model is the same as the image resolution of the output image.

In a possible implementation manner of the second aspect, the first image resolution of the first image is lower than the image resolution of the target image;

The second electronic device performs up-sampling processing on the first image to obtain a second image, and the image resolution of the second image is consistent with the image resolution of the target image;

The second electronic device performs super-resolution reconstruction on the second image through a single-enhanced super-resolution model to obtain the target image, wherein the single-enhanced super-resolution model is the image resolution of the input image A superresolution model with the same image resolution as the output image.

It should be noted that, if the resolution of the first image is smaller than that of the target image, the second electronic device may perform up-sampling processing on the first image to obtain a second image, so that the image resolution of the second image is the same as that of the target image. The image resolution is the same.

Then, the second electronic device uses the single-enhanced super-resolution model to process the second image to obtain the target image.

The algorithm applied for up-sampling can be any one of interpolation algorithms such as the nearest neighbor method, bilinear interpolation method, and cubic interpolation method.

In a possible implementation manner of the second aspect, the first image resolution of the first image is lower than the resolution of the target image;

The second electronic device performs super-resolution reconstruction on the first image through a multiple-enhanced super-resolution model to obtain the target image, wherein the multiple-enhanced super-resolution model is the image resolution of the input image A superresolution model with an image resolution smaller than the output image.

It should be noted that, if the resolution of the first image is smaller than the image resolution of the target image, the second electronic device may use a multiple-enhanced super-resolution model in addition to up-sampling the first image. Perform super-resolution reconstruction on the first image.

The image resolution of the input image of the multiple-enhanced super-resolution model is smaller than the image resolution of the output image. That is, the multi-fold enhanced super-resolution model can increase the image resolution of the input image.

In a third aspect, an electronic device is provided, including:

A native data module, used to obtain native image data, where the native image data is image data generated by an application and not rendered;

a preliminary rendering module, configured to render the native image data through the first graphics rendering hardware to obtain a first image;

The first super-resolution module is configured to perform super-resolution reconstruction on the first image through second graphics rendering hardware to obtain a target image, and the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardware.

In a possible implementation manner of the third aspect, the first super-segmentation module includes:

a first identification submodule, used to obtain the identification of the application;

a first model submodule, used to find the target super-score model associated with the identifier;

The first reconstruction sub-module is configured to perform super-resolution reconstruction on the first image by using the first graphics rendering hardware and the found target super-resolution model to obtain a target image.

In a possible implementation manner of the third aspect, the first super-segmentation module further includes:

The first general sub-module is used to perform super-resolution on the first image through the first graphics rendering hardware and the preset general super-resolution model if the target super-resolution model associated with the identifier is not found. Reconstruct to get the target image.

The first association submodule is used to establish an association relationship between the identifier and the general super-resolution model.

In a possible implementation manner of the third aspect, the preliminary rendering module is specifically configured to render the native image data by using the first graphics rendering hardware and a preset first image resolution to obtain the first an image.

In a possible implementation manner of the third aspect, the first super-resolution module is specifically configured to perform super-resolution on the first image by using the second graphics rendering hardware and the single-enhanced super-resolution model Reconstruction to obtain a target image, wherein the first image resolution is consistent with the image resolution of the target image, and the single-enhanced super-resolution model is that the image resolution of the input image and the image resolution of the output image are the same super-score model.

In another possible implementation manner of the third aspect, the first super-resolution module is specifically configured to perform super-resolution on the first image by using the second graphics rendering hardware and the multiple-enhanced super-resolution model rate reconstruction to obtain a target image, wherein the resolution of the first image is smaller than the image resolution of the target image, and the multiple-enhanced super-resolution model is the image resolution of the input image is smaller than the image resolution of the output image Overscore model.

In a possible implementation manner of the third aspect, the preliminary rendering module is specifically configured to render the native image data through a graphics processor to obtain the first image;

Correspondingly, the first super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through a neural network processor to obtain a target image.

In a fourth aspect, an electronic device is provided, comprising:

an image receiving module, configured to receive a first image sent by a first electronic device, where the first image is an image obtained by the first electronic device rendering native image data generated by an application;

The second super-resolution module is configured to perform super-resolution reconstruction on the first image to obtain a target image.

In a possible implementation manner of the fourth aspect, the second super-segmentation module includes:

The second identification submodule is used to obtain the identification of the application;

The second model submodule is used to find the target super-score model associated with the identifier;

The second reconstruction sub-module is configured to perform super-resolution reconstruction on the first image by using the found target super-resolution model to obtain a target image.

In a possible implementation manner of the fourth aspect, the second super-segmentation module further includes:

The second general sub-module is configured to perform super-resolution reconstruction on the first image by using a preset general super-resolution model to obtain a target image if the target super-resolution model associated with the identifier is not found.

The second association module is configured to establish an association relationship between the identifier and the general super-resolution model.

In a possible implementation manner of the fourth aspect, the first image resolution of the first image is consistent with the image resolution of the target image;

The second super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through a single-enhanced super-score model to obtain the target image, wherein the single-enhanced super-score model is an input A superresolution model where the image resolution of the image is the same as the image resolution of the output image.

In another possible implementation manner of the fourth aspect, the first image resolution of the first image is lower than the image resolution of the target image;

The second super-divided module includes:

an upsampling submodule, configured to perform an upsampling process on the first image to obtain a second image, the image resolution of the second image is consistent with the image resolution of the target image;

An enhancement sub-module, used for the second electronic device to perform super-resolution reconstruction on the second image through a single-enhanced super-resolution model to obtain the target image, wherein the single-enhanced super-resolution model is: A superresolution model where the image resolution of the input image is the same as the image resolution of the output image.

In another possible implementation manner of the fourth aspect, the first image resolution of the first image is lower than the resolution of the target image;

The second super-resolution module is specifically configured to perform super-resolution reconstruction on the first image through a multiple-enhanced super-resolution model to obtain the target image, wherein the multiple-enhanced super-resolution model is an input A superresolution model where the image resolution of the image is smaller than that of the output image.

In a fifth aspect, an image processing system is provided, the system includes a first electronic device and a second electronic device;

The first electronic device is configured to render the native image data generated by the application to obtain a first image, and send the first image to the second electronic device;

The second electronic device is configured to execute the image processing method mentioned in the second aspect above.

In a sixth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the electronic device realizes the steps of the above method.

In a seventh aspect, a computer-readable storage medium is provided, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, enables an electronic device to implement the steps of the above method.

In an eighth aspect, a chip system is provided, the chip system may be a single chip or a chip module composed of multiple chips, the chip system includes a memory and a processor, and the processor executes the storage in the memory. A computer program to implement the steps of the above method.

The beneficial effects that the embodiments of the present application have compared with the prior art are:

In the image processing method of the present application, the electronic device first renders the original image data to obtain the first image, and then performs super-resolution reconstruction on the first image to obtain the target image displayed on the screen. Compared with the method of directly rendering the native image data generated by the application into the target image finally displayed on the screen, generating the target image by means of preliminary rendering and super-resolution reconstruction can reduce rendering power consumption, reduce the amount of calculation, and improve rendering. efficient.

In addition, the electronic device performs the preliminary rendering operation through the first graphics rendering hardware, and performs the super-resolution reconstruction operation through the second graphics rendering hardware, which can make full use of the heterogeneous hardware resources in the electronic device and avoid the lack of hardware resources of a single graphics rendering hardware. Causes electronic devices to run high-quality products at lower image quality.

To sum up, the image processing method of the present application can reduce the rendering power consumption and calculation amount of rendering high-quality images, and can make full use of heterogeneous hardware resources in electronic devices, so as to solve the problem that in the existing image solutions, when the electronic device's When the hardware resources are insufficient, the electronic device can only run high-quality products with lower picture quality, which has strong ease of use and practicability.

Description of drawings

1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

2 is a software structural block diagram of an electronic device provided by an embodiment of the present application;

3 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of an example image provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of another example image provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of another example image provided by an embodiment of the present application;

7 is a schematic diagram of another electronic device provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of an application scenario provided by an embodiment of the present application;

9 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of another application scenario provided by an embodiment of the present application;

11 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

12 is a schematic diagram of another application scenario provided by an embodiment of the present application;

13 is a schematic diagram of another application scenario provided by an embodiment of the present application;

14 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 15 is a schematic diagram of another application scenario provided by an embodiment of the present application;

16 is a schematic diagram of another application scenario provided by an embodiment of the present application;

17 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

FIG. 18 is a schematic diagram of another application scenario provided by an embodiment of the present application;

19 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

FIG. 20 is a schematic diagram of another application scenario provided by an embodiment of the present application;

21 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 22 is a schematic diagram of another application scenario provided by an embodiment of the present application;

23 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 24 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 25 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 26 is a schematic diagram of another electronic device provided by an embodiment of the present application.

detailed description

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It is to be understood that the term "comprising" when used in this specification and the appended claims indicates the presence of the described features, integers, steps, operations, elements and/or hardware, but does not exclude one or more other The presence or addition of features, integers, steps, operations, elements, hardware, and/or sets thereof.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items.

As used in the specification of this application and the appended claims, the term "if" may be contextually interpreted as "when" or "once" or "in response to determining" or "in response to detecting ". Similarly, the phrases "if it is determined" or "if the [described condition or event] is detected" may be interpreted, depending on the context, to mean "once it is determined" or "in response to the determination" or "once the [described condition or event] is detected. ]" or "in response to detection of the [described condition or event]".

In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and should not be construed as indicating or implying relative importance.

References in this specification to "one embodiment" or "some embodiments" and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically emphasized otherwise. The terms "including", "including", "having" and their variants mean "including but not limited to" unless specifically emphasized otherwise.

Before introducing the embodiments of the present application, some terms involved in the embodiments of the present application will be explained first:

Picture quality refers to the picture quality. There are many image quality indicators for evaluating image quality. The more common image quality indicator is image resolution. When other image quality indicators are the same, the higher the image resolution, the higher the image quality and the lower the image resolution. , the lower the image quality.

In addition to image resolution, image quality indicators can also include one of sharpness, sharpness, lens distortion, dispersion, resolution, color gamut range, color purity (color brilliance), and color balance parameters. or more.

Image resolution refers to the amount of information stored in the image, which can be understood as the number of pixels contained in the image. The image resolution can be expressed as "the number of horizontal pixels × the number of vertical pixels". For example, the resolution of the image is 2048×1080, which means that each row of pixels in the image includes 2048 pixels, and each column of pixels includes 1080 pixels.

Rendering in computer terms refers to the process of generating an image from an image model. An image model is a description of a three-dimensional scene using a strictly defined language or data structure, including geometry, viewpoint, texture, lighting information, and rendering parameters. The rendering parameters may include the above-mentioned image quality indicators.

These products have higher requirements on the hardware resources of electronic devices. When an electronic device renders images of these products, the rendering power consumption is high, and if the hardware resources of the electronic device are insufficient, the electronic device can only run these products at a lower image quality, which affects the user experience.

For example, if a user runs a well-crafted game on an electronic device with RTX2080 graphics card-level hardware resources, even if the user adjusts the image resolution to 2048×1080, the frame rate of the electronic device can remain around 90 frames per second .

However, if a user runs a well-crafted game with a higher image resolution on a mobile platform, even if the mobile platform is the current high-level mobile platform Mali-G76, the frame rate of the electronic device can usually only be maintained at 40 frames per second or so. If the user turns down the image resolution, the frame rate of the electronic device is usually only kept at 60 frames per second.

In this regard, a graphics acceleration (GPU-turbo) technology is provided. GPU-turbo technology reconstructs the traditional GPU architecture at the bottom layer of the electronic device system, realizes the coordination of software and hardware, and greatly improves the overall computing efficiency of the GPU. In addition, GPU-turbo technology can detect the similarities and differences of image quality of adjacent frame images through artificial intelligence (AI) technology, render the difference parts of adjacent frames, and retain the same content of adjacent frames. In this way, GPU-turbo can save 80% of the calculation, greatly improving the rendering speed of the GPU.

However, the image rendering method of GPU-turbo technology is consistent with the traditional GPU rendering method. The GPU directly renders the initial image file into the final image displayed on the screen. Therefore, when using the GPU-turbo technology to run high-quality products, it will still generate a large load on the GPU of the electronic device, and the rendering power consumption will be high.

In view of this, embodiments of the present application provide an image processing method, apparatus, electronic device, and computer-readable storage medium, so as to solve the problem of high rendering power consumption and high computational cost when rendering high-quality products in the existing image rendering method. volume problem.

It can be understood that the steps involved in the image rendering method provided in the embodiments of the present application are only used as examples, and not all steps are required to be performed, or not all information or contents in the message are required to be selected. , which can be increased or decreased as needed during use.

The same step or steps or messages with the same function in the embodiments of the present application may refer to each other for reference between different embodiments.

The service scenarios described in the embodiments of the present application are for the purpose of illustrating the technical solutions of the embodiments of the present application more clearly, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. With the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The electronic device described in the embodiments of the present application may be a mobile phone, a tablet computer, a handheld computer, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, a media For devices such as players and wearable devices, the embodiments of the present application do not impose any restrictions on the specific forms/types of the electronic devices.

Exemplarily, FIG. 1 shows a schematic structural diagram of an electronic device 100 .

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195 and so on. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (neural-network processing unit, NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate with each other through the I2C bus interface, so as to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled with the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through a CSI interface, so as to realize the photographing function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to implement the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 charges the battery 142 , it can also supply power to the electronic device through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 and supplies power to the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and then turn it into an electromagnetic wave for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the same device as at least part of the modules of the processor 110 .

The modem processor may include a modulator and a demodulator. Wherein, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and passed to the application processor. The application processor outputs sound signals through audio devices (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110, and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellites System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the electronic device 100 may include one or N display screens 194 , where N is a positive integer greater than one.

The electronic device 100 may implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193 .

Camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy and so on.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos of various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information, and can continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, phone book, etc.) created during the use of the electronic device 100 and the like. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playback, recording, etc.

The audio module 170 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be answered by placing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C through a human mouth, and input the sound signal into the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The earphone jack 170D is used to connect wired earphones. The earphone interface 170D can be the USB interface 130, or can be a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be provided on the display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, and the like. The capacitive pressure sensor may be comprised of at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example, when a touch operation whose intensity is less than the first pressure threshold acts on the short message application icon, the instruction for viewing the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the motion attitude of the electronic device 100 . In some embodiments, the angular velocity of electronic device 100 about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 shakes, calculates the distance that the lens module needs to compensate for according to the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse motion to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenarios.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of the flip holster using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Further, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, characteristics such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

Distance sensor 180F for measuring distance. The electronic device 100 can measure the distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light to the outside through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. Proximity light sensor 180G can also be used in holster mode, pocket mode automatically unlocks and locks the screen.

The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, accessing application locks, taking pictures with fingerprints, answering incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

Touch sensor 180K, also called "touch device". The touch sensor 180K may be disposed on the display screen 194 , and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the location where the display screen 194 is located.

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the pulse of the human body and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined with the bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibration bone block obtained by the bone conduction sensor 180M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M, and realize the function of heart rate detection.

The keys 190 include a power-on key, a volume key, and the like. Keys 190 may be mechanical keys. It can also be a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

Motor 191 can generate vibrating cues. The motor 191 can be used for vibrating alerts for incoming calls, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of the power, and can also be used to indicate a message, a missed call, a notification, and the like.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be contacted and separated from the electronic device 100 by inserting into the SIM card interface 195 or pulling out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as call and data communication. In some embodiments, the electronic device 100 employs an eSIM, ie: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present invention takes an Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 as an example.

FIG. 2 is a block diagram of a software structure of an electronic device 100 according to an embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, and a kernel layer.

The application layer can include a series of application packages.

As shown in Figure 2, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and so on.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer may include window managers, content providers, view systems, telephony managers, resource managers, notification managers, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

Content providers are used to store and retrieve data and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. View systems can be used to build applications. A display interface can consist of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide the communication function of the electronic device 100 . For example, the management of call status (including connecting, hanging up, etc.).

The resource manager provides various resources for the application, such as localization strings, icons, pictures, layout files, video files and so on.

The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can disappear automatically after a brief pause without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also display notifications in the status bar at the top of the system in the form of graphs or scroll bar text, such as notifications of applications running in the background, and notifications on the screen in the form of dialog windows. For example, text information is prompted in the status bar, a prompt sound is issued, the electronic device vibrates, and the indicator light flashes.

Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The Surface Manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

In the following, the workflow of the software and hardware of the electronic device 100 is exemplarily described in conjunction with the capturing and photographing scene.

When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon, for example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. The camera 193 captures still images or video.

Next, an image processing method provided by this embodiment will be described from the perspective of the first electronic device. Please refer to the flowchart of the image processing method shown in FIG. 3, the method includes:

S301. The first electronic device renders the native image data through the first graphics rendering hardware to obtain a first image;

When the user is using the application program of the first electronic device, the application program transmits the native image data to the first graphics rendering hardware of the first electronic device for rendering. Native image data refers to unrendered image files generated by the application.

Rendering refers to a process of converting the image data stored in the first electronic device into visible pixels through techniques such as rasterization.

In the image processing method of this embodiment, the first electronic device may obtain the target image of the target image quality index by means of image super-resolution reconstruction. However, the first electronic device cannot directly perform super-resolution reconstruction on the native image data.

Therefore, the first electronic device needs to perform preliminary rendering on the original image data to obtain a first image that can be reconstructed with super-resolution.

During the preliminary rendering process, the first electronic device may call the first graphics rendering hardware to render the native image data with the first image quality index to obtain the first image.

When the first graphics rendering hardware preliminarily renders the native image data with the first image quality index, the native image data may be rendered at the first image resolution, and the first image resolution is lower than the preset image resolution; or, The first graphics rendering hardware may also adjust other image quality indicators so that the first image quality indicator is lower than the preset image quality indicator, and then render the native image data.

The above-mentioned first image quality index may be a specific type of image quality index, or the above-mentioned first image quality index may be a collection of multiple image quality indexes. When the first image quality index is a set of multiple image quality indexes, it may be understood that the first image quality index is lower than the preset image quality index, and it can be understood that some or all of the image quality indexes in the first image quality index are low at the default image quality index.

For example, in the example images shown in Fig. 4 and Fig. 5, when other image quality indicators except image resolution are the same, and the image resolution of Fig. 4 is lower than that of Fig. 5, it can be considered that the image of Fig. 4 The quality is lower than the picture quality in Figure 5. Please refer to the example images shown in Fig. 5 and Fig. 6. When the resolutions of the images in Fig. 5 and Fig. 6 are the same, since the resolution of Fig. 6 is lower than that of Fig. 5, it can also be considered that the image quality of Fig. 6 is lower than that of Fig. 6. The image quality of Figure 5.

When the first graphics rendering hardware renders an image, the higher the image quality index of the rendered image is, the more hardware resources the first graphics rendering hardware occupies, and the higher the rendering power consumption. Therefore, when the first graphics rendering hardware preliminarily renders the native image data with the first image quality index lower than the preset image quality index, the hardware resources occupied by the first graphics rendering hardware during rendering can be reduced, and the rendering function can be reduced. consumption.

The above-mentioned first graphics rendering hardware can be selected according to the actual situation. In some embodiments, the first graphics rendering hardware may be one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural-network processing unit (NPU). one or more. For example, when the first electronic device uses the GPU-turbo technology to perform preliminary rendering on the native image data, the above-mentioned first graphics rendering hardware may be a combination of a CPU and a GPU.

S302. The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain a target image.

In S301, the first electronic device performs preliminary rendering on the native image data of the application through the first graphics rendering hardware to obtain a first image. However, the image quality of the first image is low, and it is difficult to meet the user's requirements for product image quality.

Super-resolution reconstruction refers to the application of AI technology to map low-resolution images to high-resolution, with the expectation of enhancing image quality.

After the first graphics rendering hardware obtains the first image, the first electronic device may perform super-resolution reconstruction on the first image by using the second graphics rendering hardware.

When the first electronic device performs super-resolution reconstruction on the first image, the first image can be input into a trained super-resolution model, and the image quality of the first image is enhanced by the super-resolution model to obtain the target image.

The type of the above-mentioned super-score model can be selected according to the actual situation. For example, the above-mentioned super-resolution model can be a super-resolution convolutional neural network model (super-resolution convolutional neural network, SRCNN model), a fast super-resolution convolutional neural network model (fast super-resolution convolutional neural network, SRCNN model), Sub-pixel convolutional neural network model (efficient sub-pixel convolutional neural network, ESPCN model), deep recursive model (deeply-recursive convolutional network, DRCN model) and deep super-resolution network model (very deep network for super-resolution, VSDR model) ) and other models.

Moreover, if it takes a long time for the first electronic device to process a single frame of image, the application program cannot maintain a certain frame rate, which causes the screen of the first electronic device to freeze. Therefore, developers should limit the size of the super-resolution model when choosing the super-resolution model, so that the single-frame running time of the super-resolution model meets the requirements of the application's single-frame time for the display frame rate. For example, if an application sends a frame rate of 90 frames per second, the single frame time for an application to send a frame rate is 1/90 of a second. At this time, the scale of the super-scoring model should be limited, so that the single-frame running time of the super-scoring model should be less than 1/90 of a second, so as to ensure that the sending frame rate of the application is kept at 90 frames per second as much as possible, and the occurrence of screen freezes can be reduced. .

When training the above-mentioned super-score model, the first electronic device may acquire at least one set of image sample pairs, and train the super-score model by using the image sample pairs. Image sample pairs refer to pairs of sample images. Each set of image sample pairs includes a first sample image and a second sample image. The contents of the first sample image and the second sample image are consistent, but the image quality of the first sample image is lower than that of the second sample image.

When using the image sample pair to train the super-score model, the first electronic device may input the first sample image in the image sample pair into the super-score model to obtain a first output image.

Then, the first electronic device calculates the loss value according to the first output image, the second sample image and the preset loss function, and updates the super-score model according to the loss value and the preset network update algorithm.

After the super-score model is updated, go back to the previous step, and use the image samples to train the over-score model in a loop until the number of cycles reaches the preset number of times threshold or the loss value is less than the preset loss threshold.

The acquisition method of the above-mentioned image sample pair can be selected according to the actual situation. For example, the above-mentioned image sample pair may be an image dataset from which a pair was originally generated, or the above-mentioned image sample pair may also be a pair of image sample pairs obtained by degrading a high-quality image into a low-quality image.

Moreover, the source of image sample pairs will have a certain impact on the performance of the super-resolution model. If the developer wants to train a general super-score model, the developer can collect samples in an untargeted manner when sampling image sample pairs to obtain general image sample pairs.

At this time, the first electronic device can obtain the general super-score model by using the general image sample pair to train the super-score model. The general super-score model has high applicability and can be applied to many application scenarios. However, the image quality optimization capability of the general super-resolution model is limited, and it is difficult to optimize the image quality of each application scenario to a high degree.

Therefore, if the developer wants the super-score model to be able to optimize the image quality of a certain product or a certain type of product, the developer should only collect the same product or type of product when sampling the image sample pair. Related image sample pairs, get a specific image sample pair.

At this time, the first electronic device uses a specific pair of image samples to train a super-scoring model, and can obtain a specific super-scoring model for a certain product or a certain type of product. The applicability of specific super-score models is poor and can only be applied to specific products. However, the image quality optimization capability of a specific super-scoring model is high, and a higher degree of image quality optimization can be performed on the image of a specific product.

Take gaming applications as an example. When a developer wants to train a general game-like superscore model, the developer can obtain pairs of image samples from various game applications. For example, developers can choose from "Arena of Valor" (Chinese translation "Legendary Showdown"), "PUBG Mobile" (Chinese translation "PUBG"), "Carrot Fantasy" (Chinese translation "Radish Defense"), "Plants vs. .Zombies" (Chinese translation "Plants vs. Zombies"), "Minecraft" (Chinese translation "Minecraft"), "Life After" (Chinese translation "After Tomorrow") and other game applications of the same type or different types Generic game image sample pair.

Then, the first electronic device uses the above-mentioned generic game image sample pair to train a super-score model, so that the trained super-score model can be applied to various game applications.

When a developer wants to train a targeted super-score model for a phenomenal game application, the developer can only acquire images of that game application as a sample pair of specific game images. For example, the current "Arena of Valor" is a phenomenon-level game with a large number of players. In order to enable players to get a smoother game experience when playing "Arena of Valor", developers can train a super-scoring model for "Arena of Valor".

When training a superscore model for Arena of Valor, developers should only use the game images in Arena of Valor as a specific game image sample pair. The above game images can be character images, terrain images, skill images, etc. in "Arena of Valor".

Then, the first electronic device trains the super-scoring model according to the above-mentioned specific game image sample pair, so that the trained super-scoring model can specifically enhance the image quality of the game "Arena of Valor".

When the first electronic device is provided with multiple trained super-score models, the first electronic device can detect the identifier of the application when the user starts the application, and select the corresponding super-score model according to the identifier of the application to perform the above steps Operation of S302.

If the first electronic device does not detect the super-score model corresponding to the identifier of the application, the first electronic device may call the general super-score model to perform the steps of the above-mentioned step S302. In addition, when the first electronic device is provided with a plurality of general super-resolution models, the first electronic device can also use the general super-resolution model to perform super-resolution reconstruction on the first image, and establish the general super-resolution model and the above application. The association relationship between the program identifiers, so that when the first electronic device processes the first image of the application program next time, it can find the same general super-score model to process the first image according to the foregoing association relationship.

In some embodiments, the identifier of the application program may be the package name (packname) of the application program. For example, after training the super-scoring model for "Arena of Valor", the first electronic device may associate the super-scoring model with the package name of "Arena of Valor". When "Arena of Valor" is activated or awakened, the first electronic device obtains the package name of "Arena of Valor", searches for the corresponding super-score model according to the package name, and uses the corresponding super-score model to compare the image of "Arena of Valor" to be processed.

Alternatively, in other embodiments, the identifier of the application program may also be user-defined. For example, the user defines the ID of "Arena of Valor" as 0010. Then, the first electronic device associates "0010" with "Arena of Valor" and a super-resolution model for "Arena of Valor". When "Arena of Valor" is activated or awakened, the first electronic device searches for the logo corresponding to "Arena of Valor", and obtains the logo "0010" of "Arena of Valor". Then, the first electronic device searches for a corresponding super-resolution model according to the identifier "0010", and uses the corresponding super-resolution model to process the image of "Arena of Valor".

In addition, the above-mentioned super-score model may be a single-enhanced super-score model, or, the above-mentioned super-score model may also be a multiple-enhancement type of over-score model. The haplointensive super-resolution model is a super-resolution model in which the image resolution of the input image and the image resolution of the output image are the same. The multiple-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is smaller than that of the output image.

When the above-mentioned super-score model is a single-enhanced super-score model, the image resolution of the first image is consistent with the image resolution of the target image, and the single-enhancement super-score model improves other image quality indicators of the first image by means of The image quality of the first image is enhanced to obtain the target image.

When the above-mentioned super-resolution model is a multiple-enhanced super-resolution model, the image resolution of the first image is smaller than that of the target image. When initially rendering the native image data of the application, the first graphics rendering hardware of the first electronic device may render at a smaller image resolution, thereby reducing the hardware resources occupied by the first graphics rendering hardware in the rendering process , as well as reducing rendering power consumption.

After the first graphics rendering hardware renders a first image with a smaller image resolution, the first electronic device enhances the image quality of the first image through the multiple-enhanced super-resolution model according to the target resolution configured by the user, and converts the The image resolution of the first image is adapted to the target image resolution to obtain the target image.

It should be noted that, when the first electronic device performs the above-mentioned super-resolution reconstruction operation, the second graphics rendering hardware that runs the above-mentioned super-resolution model may be set according to the actual situation. In some embodiments, the first electronic device may run the above-mentioned super-score model on a CPU, and use the CPU to perform image quality enhancement on the above-mentioned first image; in other embodiments, the first electronic device may run the above-mentioned super-resolution model on a GPU The sub-model is used to enhance the image quality of the first image through the GPU; or, the first electronic device may run the super-resolution model on the NPU to enhance the image quality of the first image through the NPU. The present application does not limit the hardware for performing the above-mentioned super-resolution reconstruction operation in the first electronic device.

In order to better describe the image processing method provided by the embodiments of the present application, the following description is made with reference to a specific scenario.

FIG. 7 is a schematic diagram of a first electronic device suitable for scenario 1, scenario 2, scenario 3, and scenario 4 according to this embodiment. As shown in FIG. 7 , a GPU 701 and an NPU 702 may be provided in the first electronic device. In the NPU 702, a general game-type super-score model and a super-score model for game A are preset, and the over-score model for game A is associated with the application identifier of game A.

scene one:

As shown in 8, a plurality of icons can be set on the main page of the first electronic device, including "clock", "calendar", "game B", "memo", "camera", "address book", "phone" , Information and other icons. An icon represents an application.

The user clicks the icon of the game B on the first electronic device, and the first electronic device starts the game B in response to the user's click operation.

As shown in FIG. 9 , after the game B is started, the application program of the game B sends the native image data to the GPU 701 frame by frame.

The GPU 701 performs frame-by-frame rendering on the above-mentioned native image data, obtains a first image corresponding to each frame of native image data, and sends the first image to the NPU 702 frame by frame.

After receiving the first image, NPU 702 obtains the application identifier of game B, and does not find the super-score model corresponding to game B according to the application identifier of game B, then selects the super-score model of the general game class as the target over-score model.

The NPU 702 inputs the first image into the target super-resolution model frame by frame, enhances the image quality of the first image through the target super-resolution model, obtains target images corresponding to the first images of each frame, and sends the target images to the first image frame by frame. A display screen of an electronic device.

As shown in FIG. 10 , after the display screen acquires the target image, the target image is displayed on the screen, so that the screen of the game B is displayed on the display screen.

In addition, if only one super-score model of a general game class is set in the first electronic device, when the user triggers Game B again, the first electronic device still uses the same super-score model of the same general game class to process the image of Game B .

If there are multiple super-score models of general game types in the first electronic device, when the user triggers game B again, the first electronic device can randomly select a super-score model from the multiple general game-type super-score models to match the game The image of B is processed.

Alternatively, if there are multiple super-score models of the general game class in the first electronic device, the first electronic device may also establish the application identifier of the game B after using the super-score model of the general game class to process the image of the game B for the first time. The association with the superscore model for this generic game class. When the user triggers the game B again, the first electronic device can obtain the application identifier of the game B, find the super-score model of the same general game class according to the application identifier of the game B, and process the image of the game B.

Scenario two:

As shown in FIG. 11 , a plurality of icons can be set on the main page of the first electronic device, including “clock”, “calendar”, “game A”, “memo”, “camera”, “contact book”, “phone” ”, “Information” and other icons. An icon represents an application.

As shown in FIG. 9 , the user clicks the icon of the game A on the first electronic device, and the first electronic device starts the game A in response to the user's click operation.

After the game A starts, the application program of the game A sends the native image data to the GPU 701 frame by frame.

After receiving the first image, NPU 702 obtains the application identifier of game A, finds the super-score model for game A according to the application identifier of game A, and selects the super-score model for game A as the target over-score model.

As shown in FIG. 12 , after the display screen acquires the target image, the target image is displayed on the screen, so that the screen of the game A is displayed on the display screen.

Scenario three:

As shown in FIG. 11 , the user clicks the icon of the game A on the first electronic device, and the first electronic device starts the game A in response to the user's click operation.

As shown in Figures 13 and 14, after the game A is started, the application of the game A sends the native image data 1301 to the GPU 701 frame by frame. The image resolution of the native image data 1301 is 480×360.

The GPU 701 performs frame-by-frame rendering on the above-mentioned native image data 1301 to obtain a first image 1302 corresponding to each frame of the native image data 1301, and the image resolution of the first image 1302 is 1920×1080. The GPU 701 sends the initially rendered first image 1302 to the NPU 702 frame by frame.

After receiving the first image 1302, the NPU 702 obtains the application identification of game A, finds the super-score model for game A according to the application identification of game A, and selects the super-score model for game A as the target over-score model. The target super-resolution model is a haplo-enhanced super-resolution model, and the target resolution of the output image of the target super-resolution model is 1920×1080.

The NPU 702 inputs the first image 1302 into the target super-resolution model frame by frame, and enhances the image quality of the first image 1302 through the target super-resolution model to obtain a target image 1303 with an image resolution of 1920×1080.

As shown in FIG. 14 , the image resolution of the target image 1303 is the same as that of the first image 1302 , but the definition of the target image 1303 is higher than that of the first image 1302 , and the image quality of the target image 1303 is higher than that of the first image 1302 picture quality.

After acquiring the target image 1303, the NPU 702 sends the target image 1303 to the display screen of the first electronic device frame by frame.

Scenario four:

As shown in Figure 11, the user clicks the icon of the game A on the first electronic device, and the first electronic device starts the game A in response to the user's click operation.

As shown in FIG. 15 and FIG. 16 , after the game A is started, the application program of the game A sends the native image data 1501 to the GPU 701 frame by frame. The image resolution of the native image data 1501 is 480×360.

The GPU 701 performs frame-by-frame rendering on the above-mentioned native image data 1501 to obtain a first image 1502 corresponding to each frame of native image data, and the image resolution of the first image 1502 is 480×360. The GPU 701 sends the initially rendered first image 1502 to the NPU 702 frame by frame.

After receiving the first image 1502, the NPU 702 obtains the application identifier of game A, finds the super-score model for game A according to the application identifier of game A, and selects the super-score model for game A as the target over-score model. The target super-resolution model is a multiple-enhanced super-resolution model, and the target resolution of the output image of the target super-resolution model is 1920×1080.

The NPU 702 inputs the first image 1502 into the target super-resolution model frame by frame, enhances the image quality of the first image 1502 through the target super-resolution model, and performs resolution adaptation on the first image 1502 to obtain an image resolution of 1920 Target image 1503 of ×1080.

As shown in FIG. 16, the definition of the target image 1503 is the same as that of the first image 1502, but the image resolution of the target image 1503 is higher than that of the first image 1502, and the image quality of the target image 1503 is higher than that of the first image 1502. The quality of an image 1502.

After acquiring the target image 1503, the NPU 702 sends the target image 1503 frame by frame to the display screen of the first electronic device for display on the upper screen.

To sum up, in the image processing method provided in this embodiment, the first electronic device firstly renders the native image data of the application program to obtain the first image. Then, the first electronic device performs super-resolution reconstruction on the first image through the super-resolution model, so as to improve the image quality of the first image, and obtain a target image for display on the upper screen. Compared with the current solution of rendering native image data to directly obtain a high-quality image, the image processing method provided in this embodiment can reduce the requirements on the hardware resources of the first electronic device, and reduce the rendering power consumption, thereby solving the problem. The existing image processing methods have problems of high rendering power consumption and large amount of computation when rendering high-quality products.

In addition, the above-mentioned steps of preliminary rendering and super-resolution reconstruction may be performed by the same graphics rendering hardware in the first electronic device. For example, in some embodiments, the above-mentioned steps of preliminary rendering and super-resolution reconstruction may both be performed by a GPU in the first electronic device; in other embodiments, the above-mentioned steps of preliminary rendering and super-resolution reconstruction may be performed by a GPU. The steps may all be performed by an NPU within the first electronic device.

Alternatively, the above-mentioned steps of preliminary rendering and super-resolution reconstruction may also be performed by different graphics rendering hardware in the first electronic device, that is, the above-mentioned first graphics rendering hardware and second graphics rendering hardware are different graphics rendering hardware. For example, in some embodiments, the foregoing preliminary rendering step may be performed by a GPU of the first electronic device, and the foregoing super-resolution reconstruction step may be performed by an NPU of the first electronic device; in other embodiments, the foregoing preliminary rendering The steps of the super-resolution reconstruction may be performed by the CPU of the first electronic device, and the above-mentioned steps of super-resolution reconstruction may be performed by the NPU of the first electronic device.

When the above steps of preliminary rendering and super-resolution reconstruction are performed by different hardware, a variety of heterogeneous hardware resources in the first electronic device can be fully utilized, and the requirements on the hardware resources of the first electronic device can be reduced. Improve image quality under limited conditions.

When performing super-resolution reconstruction, the super-resolution model selected by the first electronic device may be a single-enhanced super-resolution model, or the super-resolution model selected by the first electronic device may also be a multiple-enhanced super-resolution model. When the super-resolution model selected by the first electronic device is the multiple-enhanced super-resolution model, the hardware resources occupied by the graphics rendering hardware in the preliminary rendering process and the rendering power consumption of the preliminary rendering can be reduced.

In addition, the above-mentioned super-scoring model may be a general super-scoring model, or, the above-mentioned super-scoring model may also be a specific super-scoring model for a certain application or a certain type of application. The applicable range of the specific super-resolution model is smaller than that of the general super-resolution model, but usually the image quality enhancement effect of the specific super-resolution model is better than that of the general super-resolution model.

The above is an image processing method provided by an embodiment of the present application, and the following describes another image processing method provided by this embodiment from the perspectives of the first electronic device and the designated device. Please refer to the flowchart of the image processing method shown in FIG. 8, the method includes:

S1701. The first electronic device renders the native image data to obtain a first image;

In this embodiment, a method for collaboratively performing image rendering by multiple electronic devices will be introduced.

When the first electronic device interacts with other electronic devices on multiple screens, the first electronic device may need to project the display screen of the device to the other electronic devices.

For example, when a user is playing a game on a mobile phone, the user may feel that the screen of the mobile phone is too small and the visual effect is not good. At this point, if the user owns a smart TV, the user can control the mobile phone to interact with the smart TV on multiple screens, establish a communication connection between the mobile phone and the smart TV, and project the game screen on the mobile phone to the smart TV for display, so that the user can Watch the game screen through the smart TV to get a better visual experience.

In the current screen projection solution, the first electronic device of the projection side needs to independently complete the image rendering work, and then transmits the rendered image to the first electronic device of the projection side. Afterwards, the first electronic device of the screen-projected party performs resolution adaptation on the rendered image, and displays the image after resolution adaptation on the screen.

In the current screencasting solution, the rendering of the image is completely handled by the first electronic device of the screencaster. Therefore, this will occupy a lot of hardware resources of the electronic equipment of the screencaster, and the rendering power consumption is high, and the same device is not fully utilized. Hardware resources of multiple electronic devices within a local area network.

In addition, the screen projection technology is usually used to project the display screen of the small-screen device to the screen of the large-screen device. At this time, it is difficult for the image with low image resolution to adapt to the display device with high image resolution, resulting in poor user experience. good.

For this reason, in the image processing method provided in this embodiment, when the user starts the application program on the first electronic device and enables the multi-screen interaction function, the first electronic device can perform preliminary processing on the native image data generated by the application program Render to get the first image.

For the preliminary rendering process, reference may be made to the description of step S301 in the previous embodiment, and details are not repeated here.

S1702. The first electronic device sends the first image to the designated device to instruct the designated device to perform super-resolution reconstruction on the first image to obtain the target image.

The designated device (ie, the aforementioned second electronic device) is another electronic device selected by the user and located in the same local area network as the aforementioned first electronic device.

For example, as shown in FIG. 18, when the user wishes to project the game screen of the mobile phone to the smart TV, the user can enable the "wireless screen projection" function of the mobile phone. After the user turns on the "wireless screen projection" function of the mobile phone, the mobile phone starts to search for available electronic devices in the same local area network. The mobile phone searches for electronic device 1, electronic device 2, and electronic device 3. When the mobile phone detects the user's click operation on the electronic device 1, it means that the electronic device 1 is selected by the user, and the mobile phone determines the electronic device 1 as the designated device.

After obtaining the first image, the first electronic device sends the first image to the designated device. The above designated device is the electronic device to be projected. The number of specified devices can be one, or the number of specified devices can be multiple.

For example, referring to FIG. 18 , it is assumed that the electronic device 1 is a smart TV and the electronic device 2 is a computer. When the user operates the mobile phone, if the user wishes to project the game screen of the mobile phone to the smart TV, the mobile phone is the first electronic device of the screen-casting party, and the smart TV is the electronic device of the screen-casting party (ie the designated device). Users can turn on the "wireless screen projection" function of the mobile phone. After the user enables the "wireless screen projection" function of the mobile phone, the mobile phone starts to search for available electronic devices in the same local area network. The mobile phone searches for electronic device 1, electronic device 2 and electronic device 3. Then the user clicks on the electronic device 1, and the mobile phone sets the electronic device 1 as a designated device in response to the user's operation, and the number of designated devices is one.

When the user operates the mobile phone, if the user wants to project the game screen of the mobile phone to the smart TV and the computer at the same time, the mobile phone is the first electronic device of the screen-casting party, and the smart TV and the computer are the electronic devices of the screen-casting party (that is, the specified device), the user can turn on the "wireless screen projection" function of the mobile phone. After the user enables the "wireless screen projection" function of the mobile phone, the mobile phone starts to search for the first electronic device available in the same local area network. The mobile phone searches for electronic device 1, electronic device 2, and electronic device 3. Then the user clicks on electronic device 1 and electronic device 2, and the mobile phone sets electronic device 1 and electronic device 2 as designated devices in response to the user's operation, and the number of designated devices is 2.

After receiving the first image, the designated device inputs the first image into the trained super-resolution model, performs super-resolution reconstruction on the first image, and obtains and displays the target image.

It can be understood that, in the image processing method of this embodiment, the first electronic device can not only use the local hardware resources to render the image, but also can use the hardware resources of the designated device to optimize the image quality of the image, and make full use of the same hardware resources. hardware resources of multiple electronic devices in the local area network, thereby reducing the load of the first electronic device on local hardware resources when rendering high-quality images, and reducing the rendering power consumption of the first electronic device for rendering high-quality images.

The superscore model on a specified device can be a haplo-enhanced super-score model, or the super-score model on a specified device can also be a multiple-enhanced over-score model. The haplointensive super-resolution model is a super-resolution model in which the image resolution of the input image and the image resolution of the output image are the same. The multiple-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is smaller than that of the output image.

In some embodiments, when the super-resolution model on the specified device is a haplo-enhanced super-resolution model, the first electronic device may acquire the target image resolution configured on the specified device. The target image resolution is the image resolution of the screen display image set on the specified device.

The first electronic device renders the native image data generated by the application according to the target image resolution to obtain a first image. At this time, the image resolution of the first image is the target image resolution.

Then, the first electronic device transmits the first image to the designated device. The designated device inputs the first image into the single-enhanced super-resolution model, improves the image quality of the first image through the single-enhanced super-score model, and obtains the target image and displays it on the screen. The image resolution of the target image is the target image resolution.

In other embodiments, when the super-resolution model on the specified device is a single-enhanced super-resolution model, the first electronic device can also directly perform the processing on the native image data generated by the application program according to the first image resolution configured by the user. Render to get the first image. At this time, the image resolution of the first image is the first image resolution.

Then, the first electronic device transmits the first image to the designated device. Since the super-resolution model on the specified device is a haplo-enhanced super-resolution model, the resolution of the first image cannot be adapted to the target image resolution. The target image resolution is the upper-screen display image resolution set by the specified device. Therefore, after acquiring the first image, the designated device can perform up-sampling processing on the first image, and adapt the resolution of the first image to the target image resolution configured on the designated device to obtain the second image. The image resolution of the second image is the target image resolution.

The algorithm applied for up-sampling can be any one of interpolation algorithms such as the nearest neighbor method, bilinear interpolation method, and cubic interpolation method. The designated device performs up-sampling processing on the first image through a preset up-sampling algorithm to obtain the second image.

After that, the designated device inputs the second image into the single-enhanced super-score network, and enhances the image quality of the second image through the single-enhancement super-score network to obtain the target image. The image resolution of the target image is the target image resolution.

In other embodiments, when the super-resolution model on the specified device is a multiple-enhanced super-resolution model, the first electronic device may render the native image data generated by the application according to the first image resolution configured by the user, Get the first image. At this time, the image resolution of the first image is the first image resolution.

Then, the first electronic device transmits the first image to the designated device. The designated device inputs the first image into the multiple-enhanced super-division network, enhances the image quality of the first image through the multiple-enhanced super-division network, and adapts the resolution of the first image to the target configured on the designated device Image resolution to get the target image. The image resolution of the target image is the target image resolution.

In addition, the super-scoring model on the specified device may be a general super-scoring model, or the above-mentioned super-scoring model may also be a specific super-scoring model trained for a certain product or a certain type of product. For the training methods of the general super-resolution model and the specific super-resolution model, reference may be made to the description of the previous embodiment, and details are not repeated here.

When multiple trained super-score models are set on the designated device, the designated device may, after receiving the first image, obtain the identifier of the application corresponding to the first image, and select the corresponding super-score model according to the identifier of the application. Perform super-resolution reconstruction processing on the first image.

If the specified device does not detect the super-resolution model corresponding to the identifier of the application, the specified device can call the general super-resolution model to perform super-resolution reconstruction processing. In addition, when a plurality of general super-resolution models are set in the designated device, the designated device can also establish a relationship between the general super-resolution model and the identifier of the above-mentioned application after performing super-resolution reconstruction on the first image using the general super-resolution model. association relationship, so that when the specified device processes the first image of the application program next time, the same general super-score model can be found to process the first image according to the foregoing association relationship.

It should be noted that the first electronic device may select appropriate graphics rendering hardware to perform preliminary rendering on the native image data according to the actual situation to obtain the first image. The above-mentioned graphics rendering hardware may be one or more of a central processing unit (central processing unit, CPU), a graphics processing unit (graphics processing unit, GPU), and a neural-network processing unit (neural-network processing units, NPU). For example, when the first electronic device uses the GPU-turbo technology to initially render the native image data, the above-mentioned graphics rendering hardware may be a combination of a CPU and a GPU.

The specified device can choose suitable hardware to run the above super-score model according to the actual situation. In some embodiments, the designated device may run the above-mentioned super-score model on the CPU, and enhance the image quality of the above-mentioned first image through the CPU; in other embodiments, the designated device may run the above-mentioned super-score model on the GPU, through The GPU enhances the image quality of the first image; or, the designated device may run the super-score model on the NPU, and enhance the image quality of the first image through the NPU. The present application does not limit the hardware for performing the above-mentioned super-division reconstruction operation in the specified device.

In the following application scenarios, the first electronic device (that is, the electronic device of the screen-casting party) may be a mobile phone, and a GPU may be provided in the mobile phone. The designated device (ie, the electronic device of the screen-casting party) may be a smart TV and a computer, and an NPU may be provided in the smart TV and the computer. In the NPU, a general game-type super-score model and a super-score model for game A are preset, and the over-score model for game A is associated with the application identifier of game A.

Scenario five:

The user operates the mobile phone and projects the display screen of the mobile phone to the smart TV.

As shown in FIG. 8 and FIG. 19 , the user clicks the icon of the game B on the mobile phone, and the mobile phone starts the game B in response to the user's click operation.

After Game B is started, the application of Game B sends native image data to the GPU of the mobile phone frame by frame.

The GPU of the mobile phone renders the above-mentioned native image data frame by frame, obtains the first image corresponding to each frame of native image data, and sends the first image frame by frame to the smart TV through the wireless communication module of the mobile phone.

After receiving the first image, the smart TV obtains the application identifier corresponding to the first image, and if no super-score model corresponding to game B is found according to the application identifier of game B, a general game-type super-score model is selected as the target over-score model.

The NPU of the smart TV inputs the first image into the target super-resolution model frame by frame, enhances the image quality of the first image through the target super-resolution model, obtains the target image corresponding to each frame of the first image, and sends the target image frame by frame. to the upper screen of the Smart TV's display.

Scenario six:

As shown in FIG. 9 and FIG. 19 , the user clicks the icon of the game A on the mobile phone, and the mobile phone starts the game A in response to the user's click operation.

After Game A starts, the application of Game A sends native image data frame by frame to the GPU of the mobile phone.

The GPU of the mobile phone renders the above-mentioned native image data frame by frame, obtains a first image corresponding to each frame of native image data, and sends the first image to the smart TV through the wireless communication module of the mobile phone.

After receiving the first image, the smart TV obtains the application identifier corresponding to the first image, finds a superscore model for game A according to the application identifier of game A, and selects the superscore model for game A as the target overscore model.

Scenario Seven:

Referring to FIG. 20 and FIG. 21 , the user operates the mobile phone 2001 to project the display screen of the mobile phone 2001 to the smart TV 2002 and the computer 2003 .

The user clicks the icon of the game A on the mobile phone 2001, and the mobile phone 2001 starts the game A in response to the user's click operation.

After the game A is started, the application of the game A sends the native image data 20011 to the GPU of the mobile phone 2001 frame by frame. The image resolution of native image data 20011 is 480×360.

Moreover, the mobile phone 2001 performs data interaction with the smart TV 2002 and the computer 2003 through the wireless communication module, and obtains the image resolution 1920×1080 configured on the smart TV 2002 and the image resolution 2560×1440 configured on the computer 2003 .

The mobile phone 2001 renders the above-mentioned native image data 20011 frame by frame according to the image resolution 1920×1080 configured on the smart TV 2002, and obtains the image 20012. The resolution of the image 20012 is 1920×1080.

At the same time, the mobile phone 2001 renders the above-mentioned native image data 20011 frame by frame according to the image resolution 2560×1440 configured on the computer 2003 to obtain the image 20013, and the resolution of the image 20013 is 2560×1440.

The mobile phone 2001 sends the image 20012 to the smart TV 2002 through the wireless communication module, and sends the image 20013 to the computer 2003 through the wireless communication module.

After receiving the image 20012, the smart TV 2002 obtains the application identifier corresponding to the image 20012, and finds the single-enhanced super-score model for the game A according to the application identifier of the game A, and selects the single-enhanced super-score model for the game A. as the target superscore model.

The NPU of the smart TV 2002 inputs the image 20012 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 20012 through the target super-resolution model to obtain the image 20021. Image 20021 has an image resolution of 1920×1080.

As shown in Figure 21, the image resolution of image 20012 and the image resolution of image 20021 are both 1920×1080, but the resolution of image 20021 is higher than that of image 20012, and the image quality of image 20021 is higher than that of image 20012 quality.

After the smart TV 2002 obtains the image 20012, the image 20012 is transmitted frame by frame to the display screen of the smart TV 2002 for on-screen display.

After receiving the image 20013, the computer 2003 obtains the application identifier corresponding to the image 20013, and finds the haplo-enhanced super-score model for game A according to the application identifier of game A, and selects the haplo-enhanced super-score model for game A as the Goal overscore model.

The NPU of the computer 2003 inputs the image 20013 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 20013 through the target super-resolution model to obtain the image 20031. Image 20031 has an image resolution of 2560×1440.

As shown in Figure 21, the image resolution of image 20013 and the image resolution of image 20031 are both 2560×1440, but the resolution of image 20031 is higher than that of image 20013, and the image quality of image 20031 is higher than that of image 20013 quality.

After the computer 2003 obtains the image 20031, it transmits the image 20031 frame by frame to the display screen of the computer 2003 for display on the upper screen.

Scene Eight:

Referring to FIG. 22 and FIG. 23 , the user operates the mobile phone 2201 to project the display screen of the mobile phone 2201 to the smart TV 2202 and the computer 2203 .

The user clicks the icon of the game A on the mobile phone 2201, and the mobile phone 2201 starts the game A in response to the user's click operation.

After the game A is started, the application program of the game A sends the native image data 22011 to the GPU of the mobile phone 2201 frame by frame. The image resolution of native image data 22011 is 480×360.

The mobile phone 2201 renders the above-mentioned native image data 22011 frame by frame according to the pre-configured image resolution to obtain an image 22012, and the resolution of the image 22012 is 480×360.

The mobile phone 2201 sends the image 22012 to the smart TV 2202 through the wireless communication module, and sends the image 22012 to the computer 2203 through the wireless communication module.

After receiving the image 22012, the smart TV 2202 obtains the application identifier corresponding to the image 22012, and finds the single-enhanced super-score model for the game A according to the application identifier of the game A, and selects the single-enhanced super-score model for the game A. as the target superscore model.

The NPU of the smart TV 2202 upsamples the image 22012 frame by frame to obtain the image 22021, and the image resolution of the image 22021 is 1920×1080. Then, the NUP of the smart TV 2202 inputs the image 22021 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 22021 through the target super-resolution model to obtain the image 22022.

As shown in Figure 23, the image resolution of image 22021 and the image resolution of image 22022 are both 1920×1080, but the resolution of image 22022 is higher than that of image 22021, and the image quality of image 22022 is higher than that of image 22021 quality.

After the smart TV 2202 obtains the image 22022, the image 22022 is transmitted frame by frame to the display screen of the smart TV 2202 for on-screen display.

After receiving the image 22012, the computer 2203 obtains the application identifier corresponding to the image 22012, and finds the haplo-enhanced super-score model for game A according to the application identifier of game A, and selects the haplo-enhanced super-score model for game A as the Goal overscore model.

The NPU of the computer 2203 upsamples the image 22012 frame by frame to obtain the image 22031, and the image resolution of the image 22031 is 2560×1440. Then, the NUP of the computer 2203 inputs the image 22031 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 22031 through the target super-resolution model to obtain the image 22032.

As shown in Figure 23, the image resolution of image 22031 and the image resolution of image 22032 are both 1920×1080, but the resolution of image 22032 is higher than that of image 22031, and the image quality of image 22032 is higher than that of image 22031 quality.

After the computer 2203 obtains the image 22032, it transmits the image 22032 frame by frame to the display screen of the computer 2203 for display on the upper screen.

Scene Nine:

Referring to FIG. 24 and FIG. 25 , the user operates the mobile phone 2401 to project the display screen of the mobile phone 2401 to the smart TV 2402 and the computer 2403 .

The user clicks the icon of the game A on the mobile phone 2401, and the mobile phone 2401 starts the game A in response to the user's click operation.

After the game A is started, the application of the game A sends the native image data 24011 to the GPU of the mobile phone 2401 frame by frame.

The mobile phone 2401 performs frame-by-frame rendering on the above-mentioned native image data 24011 according to the pre-configured image resolution to obtain an image 24012, and the resolution of the image 24012 is 480×360.

The mobile phone 2401 sends the image 24012 to the smart TV 2402 through the wireless communication module, and sends the image 24012 to the computer 2403 through the wireless communication module.

After receiving the image 24012, the smart TV 2402 obtains the application identifier corresponding to the image 24012, and finds a multiple-enhanced super-score model for the game A according to the application identifier of the game A, and selects the multiple-enhanced super-score model for the game A. as the target superscore model.

The NPU of the smart TV 2402 inputs the image 24012 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 24012 through the target super-resolution model to obtain the image 24021. Image 24021 has an image resolution of 1920×1080.

As shown in FIG. 25 , the image resolution of image 24021 is greater than that of image 24012 , and the resolution of image 24021 is greater than that of image 24012 , so the image quality of image 24021 is higher than that of image 24012 .

After the smart TV 2402 obtains the image 24021, it transmits the image 24021 frame by frame to the display screen of the smart TV 2402 for on-screen display.

After receiving the image 24012, the computer 2403 obtains the application identifier corresponding to the image 24012, and finds the multiple-enhanced super-score model for the game A according to the application identifier of the game A, and selects the multiple-enhanced super-score model for the game A as the Goal overscore model.

The NPU of the computer 2403 inputs the image 24012 into the target super-resolution model frame by frame, and performs super-resolution reconstruction on the image 24012 through the target super-resolution model to obtain the image 24031. Image 24031 has an image resolution of 2560×1440.

The image resolution of image 24031 is greater than that of image 24012, and the resolution of image 24031 is greater than that of image 24012, so the image quality of image 24031 is higher than that of image 24012.

After the computer 2403 obtains the image 24031, the image 24031 is transmitted frame by frame to the display screen of the computer 2403 for display on the upper screen.

To sum up, in the image processing method provided in this embodiment, the first electronic device firstly renders the native image data of the application program to obtain the first image. Then, the first electronic device transmits the first image to the designated device that needs to be projected, and the designated device performs super-resolution reconstruction on the first image, improves the quality of the first image, and obtains the target image for display on the upper screen . That is to say, in the image processing method improved in this embodiment, the image rendering process is divided into two steps: preliminary rendering and super-resolution reconstruction. The preliminary rendering step is performed by the first electronic device on the screen projection side, and super-resolution reconstruction is performed. The steps are executed by the designated device of the screen-casting party, thereby reducing the load on the hardware resources of the first electronic device of the screen-casting party, reducing the rendering efficiency, and making full use of the hardware resources of the screen-casting party's first electronic device and the casting party's first electronic device. The hardware resources of the specified device on the screen side.

In addition, by using the image processing method provided in this embodiment, the display resolution of the target image after image quality enhancement can be adapted to the display resolution of the specified device, thereby improving the user experience.

Wherein, when the designated device performs super-resolution reconstruction, the super-resolution model selected by the designated device may be a single-enhanced super-resolution model, or may also be a multiple-enhanced super-resolution model. When the super-resolution model selected by the specified device is the multiple-enhanced super-resolution model, the hardware resources occupied by the preliminary rendering and the rendering power consumption of the preliminary rendering can be reduced.

In addition, the above-mentioned super-scoring model may be a general super-scoring model, or may also be a specific super-scoring model for a certain application or a certain type of application. The applicable range of the specific super-resolution model is smaller than that of the general super-resolution model, but usually the image quality enhancement effect of the specific super-resolution model is better than that of the general super-resolution model.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Referring to FIG. 26 , an embodiment of the present application further provides an electronic device. As shown in FIG. 26 , the electronic device 26 of this embodiment includes a processor 260 , a memory 261 , and a computer program 262 stored in the memory 261 and executable on the processor 260 . When the processor 260 executes the computer program 262, the steps in the above embodiments of the screen expansion method are implemented, for example, steps S301 to S302 shown in FIG. 1 . Alternatively, when the processor 260 executes the computer program 262, the functions of the modules/units in the foregoing device embodiments are implemented, for example, the functions of the modules 2601 to 2602 shown in FIG. 26 .

Exemplarily, the computer program 262 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 261 and executed by the processor 260 to complete the this application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 262 in the electronic device 26 . For example, the computer program 262 can be divided into a native data module, a preliminary rendering module and a first super-divided module, and the specific functions of each module are as follows:

The electronic device 26 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The electronic device may include, but is not limited to, the processor 260 and the memory 261 . Those skilled in the art can understand that FIG. 26 is only an example of the electronic device 26, and does not constitute a limitation on the electronic device 26, and may include more or less components than the one shown, or combine some components, or different components For example, the electronic device may further include an input and output device, a network access device, a bus, and the like.

The so-called processor 260 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware hardware, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 261 may be an internal storage unit of the electronic device 26 , such as a hard disk or a memory of the electronic device 26 . The memory 261 may also be an external storage device of the electronic device 26, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) equipped on the electronic device 26 card, Flash Card, etc. Further, the memory 261 may also include both an internal storage unit of the electronic device 26 and an external storage device. The memory 261 is used to store the computer program and other programs and data required by the electronic device. The memory 261 may also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus/electronic device and method may be implemented in other manners. For example, the above-described embodiments of the apparatus/electronic device are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Either the hardware may be incorporated or integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Electric carrier signals and telecommunication signals are not included.

Finally, it should be noted that: the above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this, and any changes or replacements within the technical scope disclosed in the present application should be covered by the present application. within the scope of protection of the application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

An image processing method, characterized in that, applied to a first electronic device, comprising:

The first electronic device acquires native image data, where the native image data is unrendered image data generated by an application;

The first electronic device renders the native image data through the first graphics rendering hardware to obtain a first image;

The first electronic device performs super-resolution reconstruction on the first image through second graphics rendering hardware to obtain a target image, and the first graphics rendering hardware and the second graphics rendering hardware are different graphics rendering hardware.
The image processing method according to claim 1, wherein the first electronic device performs super-resolution reconstruction on the first image through first graphics rendering hardware to obtain the target image, comprising:

The first electronic device obtains the identifier of the application;

the first electronic device searches for a target super-score model associated with the identifier;

The first electronic device performs super-resolution reconstruction on the first image through the first graphics rendering hardware and the found target super-resolution model to obtain a target image.
The image processing method according to claim 2, wherein after the first electronic device searches for the target super-score model associated with the identifier, the method further comprises:

If the target super-resolution model associated with the identifier is not found, the first electronic device performs super-resolution reconstruction on the first image by using the first graphics rendering hardware and a preset general super-resolution model, Get the target image.
The image processing method according to claim 1, wherein the first electronic device renders the native image data through first graphics rendering hardware to obtain the first image, comprising:

The first electronic device renders the native image data by using the first graphics rendering hardware and a preset first image resolution to obtain a first image.
The image processing method according to claim 4, wherein the first electronic device performs super-resolution reconstruction on the first image through second graphics rendering hardware to obtain the target image, comprising:

The first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware and the single-enhanced super-resolution model to obtain a target image, wherein the first image resolution and the The image resolutions of the target images are the same, and the single-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image and the image resolution of the output image are the same.
The image processing method according to claim 4, wherein the first electronic device performs super-resolution reconstruction on the first image through second graphics rendering hardware to obtain the target image, comprising:

The first electronic device performs super-resolution reconstruction on the first image by using the second graphics rendering hardware and the multiple-enhanced super-resolution model to obtain a target image, wherein the resolution of the first image is smaller than the resolution of the first image. The image resolution of the target image. The multiple-enhanced super-resolution model is a super-resolution model in which the image resolution of the input image is smaller than the image resolution of the output image.
The image processing method according to claim 1, wherein the first electronic device renders the native image data through first graphics rendering hardware to obtain the first image, comprising:

The first electronic device renders the native image data through a graphics processor to obtain a first image;

Correspondingly, the first electronic device performs super-resolution reconstruction on the first image through the second graphics rendering hardware to obtain the target image, including:

The first electronic device performs super-resolution reconstruction on the first image through a neural network processor to obtain a target image.
An image processing method, characterized in that, applied to a second electronic device, comprising:

The second electronic device receives a first image sent by the first electronic device, where the first image is an image obtained by the first electronic device rendering the native image data generated by the application;

The second electronic device performs super-resolution reconstruction on the first image to obtain a target image.
The image processing method according to claim 8, wherein the second electronic device performs super-resolution reconstruction on the first image to obtain a target image, comprising:

acquiring, by the second electronic device, the identifier of the application;

the second electronic device searches for a target super-score model associated with the identifier;

The second electronic device performs super-resolution reconstruction on the first image by using the found target super-resolution model to obtain a target image.
The image processing method according to claim 9, wherein after the second electronic device searches for the target super-score model associated with the identifier, further comprising:

If the target super-resolution model associated with the identifier is not found, the second electronic device performs super-resolution reconstruction on the first image by using a preset general super-resolution model to obtain a target image.
The image processing method according to claim 8, wherein the first image resolution of the first image is consistent with the image resolution of the target image;

The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

The second electronic device performs super-resolution reconstruction on the first image through a single-enhanced super-resolution model to obtain the target image, wherein the single-enhanced super-resolution model is the image resolution of the input image A superresolution model with the same image resolution as the output image.
The image processing method according to claim 8, wherein the first image resolution of the first image is lower than the image resolution of the target image;

The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

The second electronic device performs up-sampling processing on the first image to obtain a second image, and the image resolution of the second image is consistent with the image resolution of the target image;

The second electronic device performs super-resolution reconstruction on the second image through a single-enhanced super-resolution model to obtain the target image, wherein the single-enhanced super-resolution model is the image resolution of the input image A superresolution model with the same image resolution as the output image.
The image processing method according to claim 8, wherein the first image resolution of the first image is lower than the resolution of the target image;

The second electronic device performs super-resolution reconstruction on the first image to obtain a target image, including:

The second electronic device performs super-resolution reconstruction on the first image through a multiple-enhanced super-resolution model to obtain the target image, wherein the multiple-enhanced super-resolution model is the image resolution of the input image A superresolution model with an image resolution smaller than the output image.
An image processing system, characterized in that the system includes a first electronic device and a second electronic device;

The first electronic device is configured to render the native image data generated by the application to obtain a first image, and send the first image to the second electronic device;

The second electronic device is used to execute the image processing method according to any one of claims 8 to 13 .
An electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, when the processor executes the computer program, the computer program according to claim 1 to 7. The method of any one of claims 8 to 13, or implementing the method of any one of claims 8 to 13.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented, or, A method as claimed in any one of claims 8 to 13 is carried out.
A chip system, characterized in that the chip system includes a memory and a processor, and the processor executes a computer program stored in the memory to implement the method according to any one of claims 1 to 7, or , implementing the method as claimed in any one of claims 8 to 13.