KR20240143549A - Image Sensor and operating method of image sensor - Google Patents

Abstract

An image sensor including a plurality of pixels according to the technical idea of the present disclosure may include a pixel array including a plurality of pixels that convert a received optical signal into an electrical signal, a readout circuit that converts the electrical signal into image data including pixel values of each of the plurality of pixels and outputs the image data, a first bad pixel corrector that corrects pixel values of bad pixels included in the image data by using pixel values of pixels surrounding each of the bad pixels among the plurality of pixels and generates first pixel data, and a second bad pixel corrector that corrects pixel values of cluster bad pixels included in the image data by using a neural network trained to correct cluster bad pixels and generates second pixel data.

Description

Image sensor and operating method of image sensor {Image Sensor and operating method of image sensor}

The technical idea of the present disclosure relates to an image sensor, and more particularly, to an image sensor for correcting a pixel value of a bad pixel based on at least one of pixel values of surrounding pixels of the bad pixels and deep learning.

An image sensor is a device that captures a two-dimensional or three-dimensional image of an object. An image sensor creates an image of an object using a photoelectric conversion element that reacts to the intensity of light reflected from the object.

Recently, as the demand for high-quality and high-definition photos and videos has increased, pixels are becoming smaller as a large number of pixels are integrated into the pixel array to increase the resolution of the image sensor. Due to process issues, bad pixels may occur at specific locations in the pixel array. A large number of bad pixels of any shape are not used to generate photos, videos, etc., which causes problems in lowering the performance of the image sensor.

Deep learning (neural network) technology is a technology that extracts valid information from input data using trained neural networks. Deep learning technology can be used to correct bad pixels, but it is difficult to process in real time due to excessive computational load, and a lot of power is consumed to process a large amount of computational load, and the cost may increase when correcting bad pixels.

Accordingly, a technology is required to reduce the cost incurred in correcting bad pixels and improve the performance of bad pixel correction.

The problem to be solved by the technical idea of the present invention is to provide an image sensor and an operating method thereof for improving the performance of the image sensor by correcting pixel values of bad pixels including cluster bad pixels and basic bad pixels according to a first algorithm and correcting cluster bad pixels according to a second algorithm based on deep learning.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure can provide an image sensor including a pixel array including a plurality of pixels which convert a received optical signal into an electrical signal, a readout circuit which converts the electrical signal into image data including pixel values of each of the plurality of pixels and outputs the image data, a first bad pixel corrector which corrects pixel values of the bad pixels included in the image data by using pixel values of pixels surrounding each of the bad pixels among the plurality of pixels and generates first pixel data, and a second bad pixel corrector which corrects pixel values of the cluster bad pixels included in the image data by using a neural network trained to correct cluster bad pixels and generates second pixel data.

In addition, as a technical means for achieving the above-described technical task, the second aspect of the present disclosure can provide an image sensor including a pixel array including a plurality of pixels that convert a received optical signal into an electrical signal, a readout circuit that converts the electrical signals into image data including pixel values of each of the plurality of pixels and outputs the image data, and an image signal processor that corrects at least one of the pixel value of the cluster bad pixel and the pixel value of the basic bad pixel in the image data based on position information indicating the positions of the cluster bad pixel and the basic bad pixel in the pixel array, wherein the image signal processor corrects the pixel value of the cluster bad pixel and the pixel value of the basic bad pixel according to a first algorithm that uses pixel values of the surrounding pixels of each of the cluster bad pixel and the basic bad pixel, and corrects the pixel value of the cluster bad pixel according to a second algorithm that uses a neural network trained to correct the cluster bad pixel.

In addition, as a technical means for achieving the above-described technical problem, a third aspect of the present disclosure may include a method for operating an image sensor including a plurality of pixels, the step of generating image data including pixel values of each of the plurality of pixels based on a received optical signal as an electrical signal, a step of distinguishing the cluster bad pixel and the basic bad pixel in the image data based on position information indicating positions of the cluster bad pixel and the basic bad pixel among the plurality of pixels, and a step of correcting the pixel value of the basic bad pixel according to a first algorithm, and correcting the pixel value of the cluster bad pixel according to at least one of the first algorithm and a second algorithm that is different from the first algorithm and utilizes a neural network.

An image sensor according to the technical idea of the present disclosure corrects pixel values of all bad pixels according to a first algorithm, and corrects pixel values of cluster bad pixels according to a second algorithm, so that pixel values of cluster bad pixels can be corrected based on at least one of the first algorithm and the second algorithm, and pixel values of bad pixels can be corrected more efficiently. Therefore, pixel values of bad pixels can be processed in real time, and power consumption can be reduced.

In addition, the image sensor according to the technical idea of the present disclosure distinguishes cluster bad pixels that need to be corrected using a neural network among cluster bad pixels, and corrects pixel values of the cluster bad pixels that need to be corrected using the neural network according to a second algorithm, thereby reducing the amount of computation of the neural network and improving the performance of the image sensor. Accordingly, bad pixel correction can be performed on image data in real time, and the visibility of the bad pixels of the image data can be reduced.

The effects obtainable from the exemplary embodiments of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly derived and understood by a person having ordinary skill in the technical field to which the exemplary embodiments of the present disclosure belong from the following description. That is, unintended effects resulting from practicing the exemplary embodiments of the present disclosure can also be derived by a person having ordinary skill in the technical field from the exemplary embodiments of the present disclosure.

FIG. 1 is a block diagram illustrating an image sensor according to an exemplary embodiment of the present disclosure.
FIG. 2 is a drawing for explaining a bad pixel according to an exemplary embodiment of the present disclosure.
FIG. 3 is a diagram for explaining the operation of a first bad pixel corrector and a second bad pixel corrector according to an exemplary embodiment of the present disclosure.
FIG. 4 is a diagram illustrating a first bad pixel corrector according to an exemplary embodiment of the present disclosure.
FIG. 5 is a diagram illustrating a second bad pixel corrector according to an exemplary embodiment of the present disclosure.
FIG. 6 is a block diagram illustrating an image signal processor according to an exemplary embodiment of the present disclosure.
FIG. 7 is a diagram for explaining the operation of a merger according to an exemplary embodiment of the present disclosure.
FIG. 8 is a block diagram illustrating components of an image signal processor according to an exemplary embodiment of the present disclosure.
FIG. 9a is a diagram for explaining the operation of a pre-processor according to an exemplary embodiment of the present disclosure.
FIG. 9b is a diagram for explaining the operation of a pre-processor according to an exemplary embodiment of the present disclosure.
FIG. 10 is a drawing for explaining an inter gradient according to an exemplary embodiment of the present disclosure.
FIG. 11 is a drawing for explaining an inter gradient according to an exemplary embodiment of the present disclosure.
FIG. 12 is a flowchart for explaining an operation method of an image sensor according to an exemplary embodiment of the present disclosure.
FIG. 13 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. The same reference numerals are used for the same components in the drawings, and redundant descriptions thereof are omitted.

The term 'circuit' as used in this disclosure means a software or hardware component such as an FPGA or an ASIC, and the 'circuit' performs certain roles. However, the 'circuit' is not limited to software or hardware. The 'circuit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the 'circuit' may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

FIG. 1 is a block diagram illustrating an image sensor according to an exemplary embodiment of the present disclosure.

The image sensor (10) can convert an optical signal of an object incident through an optical lens (LS) into image data. The image sensor (10) can be mounted on an electronic device having an image or light sensing function. For example, it can be implemented as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device can include a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, a wearable device, etc. In addition, the image sensor (10) can be mounted on an electronic device such as a drone, an Advanced Drivers Assistance System (ADAS), or an electronic device provided as a component in a vehicle, furniture, manufacturing equipment, doors, various measuring devices, etc.

Referring to FIG. 1, the image sensor (10) may include a pixel array (100), a readout circuit (200), and an image signal processor (ISP) (300). The image sensor (10) may further include general-purpose components, such as memory, interface circuits, etc., other than the pixel array (100), readout circuit (200), and image signal processor (300) illustrated in FIG. 1.

The pixel array (100) may be implemented with a photoelectric conversion element such as, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and may also be implemented with various types of photoelectric conversion elements. The pixel array (100) includes a plurality of pixels (PX) that convert a received optical signal (light) into an electrical signal, and the plurality of pixels (PX) may be arranged in rows and columns. Each of the plurality of pixels (PX) may include a photosensitive element. For example, the photosensitive element may be a photosensitive element made of an organic material or an inorganic material, such as an inorganic photodiode, an organic photodiode, a perovskite photodiode, a phototransistor, a photogate, or a pinned photodiode. In an embodiment, each of the plurality of pixels (PX) may include a plurality of photosensitive elements.

Meanwhile, a micro lens for focusing light may be arranged above each of the plurality of pixels (PX), or above each of the pixel groups composed of adjacent pixels (PX). Each of the plurality of pixels (PX) may detect light of a specific spectrum region from light received through the micro lens. For example, the pixel array (100) may include a red pixel for converting light of a red spectrum region into an electric signal, a green pixel for converting light of a green spectrum region into an electric signal, and a blue pixel for converting light of a blue spectrum region into an electric signal. However, the present invention is not limited thereto, and the pixel array (100) may include pixels for converting light of other spectrum regions into electric signals in addition to red, green, and blue. For example, the pixel array (100) may include pixels for converting light of cyan, yellow, and magenta spectrum regions into electric signals.

The pixel array (100) may include a plurality of row lines, a plurality of column lines, and a plurality of pixels (PX) arranged in a matrix form and each connected to the row lines and the column lines, and a color filter array arranged to correspond to each of the plurality of pixels (PX). A color filter array for transmitting light of a specific spectrum region may be arranged on each of the plurality of pixels (PX). For example, the color filter array may have a configuration in which 2 Υ 2 sized cells including a red color pixel, a blue color pixel, and two green color pixels are repeatedly arranged. Such a pattern may be referred to as a BAYER pattern.

In another example, the color filter array may be configured in which color pixel groups corresponding to each of the reference colors are repeatedly arranged. For example, the color filter array may be configured in which a red color pixel group including red color pixels arranged in a 2 Υ 2 pattern, a first green color pixel group including first green color pixels arranged in a 2 Υ 2 pattern, a blue color pixel group including blue color pixels arranged in a 2 Υ 2 pattern, and a second green color pixel group including second green color pixels arranged in a 2 Υ 2 pattern are repeatedly arranged. Such a pattern may be referred to as a TETRA pattern.

In another example, the color filter array may be configured such that a red color pixel group including red color pixels arranged in a 3 Υ 3 pattern, a first green color pixel group including first green color pixels arranged in a 3 Υ 3 pattern, a blue color pixel group including blue color pixels arranged in a 3 Υ 3 pattern, and a second green color pixel group including second green color pixels arranged in a 3 Υ 3 pattern are repeatedly arranged. Such a pattern may be referred to as a NONA pattern.

Meanwhile, the present disclosure is not limited thereto, and the color filter array may include a red color pixel group, a color blue pixel group, a first green color pixel group, and a second color pixel group, which include color pixels arranged in a 2n X 2n or 3n X 3n (n is a positive integer), and may be repeatedly arranged.

In some embodiments, the plurality of pixels (PX) may have a multi-layer structure. The pixel (PX) of the multi-layer structure includes a plurality of stacked photo-sensing elements that convert light of different spectral regions into electrical signals, and electrical signals corresponding to different colors may be generated from the plurality of photo-sensing elements. In other words, electrical signals corresponding to multiple colors may be output from one pixel (PX).

The readout circuit (200) can receive a pixel signal, which is an electrical signal, from the pixel array (100), and convert the pixel signal into image data (IDT) including pixel values of each of a plurality of pixels (PX). The image data (IDT) can be a signal in the form of a stream including a pixel value, which is a digital value corresponding to each pixel (PX) of the pixel array (100). The pixel signal can include a reset signal or an image signal (or a sensing signal). The readout circuit (200) can generate and output pixel values corresponding to a plurality of pixels (PX) by converting reset signals and image signals received from the pixel array (100) into digital signals. The readout circuit (200) can transfer the image data (IDT) to the image signal processor (300).

The image signal processor (300) can receive image data (IDT). The image signal processor (300) can perform image processing on the image data (IDT). For example, the image signal processor (300) can perform image processing such as noise reduction processing, gain adjustment, waveform shaping processing, interpolation processing, white balance processing, gamma processing, edge enhancement processing, and binning. The image signal processor (300) can perform bad pixel correction on the image data (IDT).

In one embodiment, the image signal processor (300) can detect a bad pixel. The image data (IDT) may include a pixel value due to the bad pixel. Here, the bad pixel may refer to a pixel in which a difference in pixel value from surrounding pixels is greater than or equal to a reference value, or which does not have a pixel value corresponding to a specific grayscale when the image data (IDT) represents an image of the specific grayscale. The bad pixel may include a static bad pixel that is continuously turned on or off among a plurality of pixels (PX) and a dynamic bad pixel that is randomly turned on or off.

A bad pixel may include a cluster bad pixel and a basic bad pixel. A cluster bad pixel may refer to a pixel generated by a plurality of adjacent bad pixels forming a cluster. Among the bad pixels, a pixel that is not a cluster bad pixel may be referred to as a basic bad pixel. A basic bad pixel may refer to a single bad pixel or at least one bad pixel that does not form a cluster.

The image signal processor (300) can distinguish bad pixels from image data (IDT). The image signal processor (300) can distinguish cluster bad pixels and basic bad pixels based on position information. The position information can indicate positions of cluster bad pixels and basic bad pixels among a plurality of pixels (PX) in the pixel array (100) and can be stored in the image signal processor (300). For example, the image signal processor (300) can distinguish pixel values of cluster bad pixels and pixel values of basic bad pixels based on the position information. However, it is not necessarily limited thereto, and the image signal processor (300) can also distinguish position information of cluster bad pixels and position information of basic bad pixels.

In one embodiment, the image signal processor (300) can perform bad pixel correction on the image data (IDT). The image signal processor (300) can correct pixel values of bad pixels among pixel values of a plurality of pixels (PX) included in the image data (IDT). The image signal processor (300) can correct at least one of the pixel value of a cluster bad pixel and the pixel value of a basic bad pixel included in the image data (IDT).

The image signal processor (300) can correct the pixel value of each of all bad pixels according to the first algorithm. The image signal processor (300) can correct the pixel value of the cluster bad pixel and the pixel value of the basic bad pixel according to the first algorithm. The first algorithm is an algorithm that corrects the pixel value of the bad pixels by using the pixel values of the surrounding pixels of each of the bad pixels in the pixel array (100), and can use a hand-crafted feature. The image signal processor (300) can correct the pixel value of the cluster bad pixel according to the first algorithm that uses the pixel values of the surrounding pixels of the cluster bad pixel. The image signal processor (300) can correct the pixel value of the basic bad pixel according to the first algorithm that uses the pixel values of the surrounding pixels of the basic bad pixel.

The image signal processor (300) can correct the pixel value of the cluster bad pixel according to the second algorithm. The second algorithm may be an algorithm that uses a neural network trained to correct the cluster bad pixel. The image signal processor (300) can correct the pixel value of the cluster bad pixel using the neural network.

Referring to FIG. 1, the image signal processor (300) may include a first bad pixel corrector (310) and a second bad pixel corrector (320). The first bad pixel corrector (310) may correct pixel values of bad pixels according to a first algorithm. The first bad pixel corrector (310) may correct pixel values of a cluster bad pixel and pixel values of a basic bad pixel. The first bad pixel corrector (310) may correct pixel values of a cluster bad pixel by using pixel values of pixels surrounding the cluster bad pixel. The first bad pixel corrector (310) may correct pixel values of a basic bad pixel by using pixel values of pixels surrounding the basic bad pixel. The first bad pixel corrector (310) may correct pixel values of a cluster bad pixel and pixel values of a basic bad pixel to generate first pixel data.

The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel according to the second algorithm. The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel using a neural network trained to correct the cluster bad pixel. The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel to generate second pixel data. The second bad pixel corrector (320) may be referred to as a deep learning based bad pixel corrector (DLBPC). The second bad pixel corrector (320) may correct the pixel values of all cluster bad pixels, or may correct the pixel values of some of the cluster bad pixels among the cluster bad pixels based on the time for performing one cluster bad pixel correction according to the second algorithm. For example, while the second bad pixel corrector (320) performs bad pixel correction for the first cluster bad pixel, it may not perform bad pixel correction for other cluster bad pixels that are received. However, even if the second bad pixel corrector (320) does not perform bad pixel correction for other cluster bad pixels, the first bad pixel corrector (310) may perform bad pixel correction for other cluster bad pixels.

In one embodiment, the image signal processor (300) can generate image-processed corrected image data (IDTa) based on at least one of the first pixel data and the second pixel data. The corrected image data (IDTa) can be image data in which a pixel value of a bad pixel in the image data (IDT) is corrected. The corrected image data (IDTa) can be provided to an external processor (20) (e.g., a main processor of an electronic device in which the image sensor (100) is mounted, a graphic processor, etc.). The external processor (20) can perform image processing on the corrected image data (IDTa) to improve image quality or reduce the resolution of the corrected image data (IDTa), and provide the corrected image data (IDTa) to a configuration that stores the image-processed image data, displays it on a display, or performs an operation based on the image data.

In the embodiment, each of the first bad pixel corrector (310) and the second bad pixel corrector (320) may be implemented as hardware. However, it is not limited thereto, and each of the first bad pixel corrector (310) and the second bad pixel corrector (320) may be implemented as software or a combination of hardware and software. In the embodiment, the pixel array (100), the readout circuit (200), and the image signal processor (300) may be implemented as one semiconductor chip or semiconductor module. In the embodiment, the pixel array (100) and the readout circuit (200) may be implemented as one semiconductor chip, and the image signal processor (300) may be implemented as another semiconductor chip.

An image sensor according to an embodiment of the present disclosure can correct pixel values of bad pixels according to a first algorithm, and can correct pixel values of cluster bad pixels according to a second algorithm. Since the pixel values of basic bad pixels are corrected according to the first algorithm, the amount of computation can be reduced and power consumption can be reduced. When correcting bad pixels according to the second algorithm, by increasing the time for performing bad pixel correction, even if the pixel values of cluster bad pixels are not corrected in real time according to the second algorithm, the pixel values of cluster bad pixels can be corrected in real time according to the first algorithm, so the amount of computation for bad pixel correction can be reduced and performance can be improved. Accordingly, an image sensor can be provided in which power consumption is reduced during bad pixel correction and performance for bad pixel correction is improved.

FIG. 2 is a diagram for explaining a bad pixel according to an exemplary embodiment of the present disclosure. FIG. 2 shows image data (IDT) in which each of a plurality of pixels (PX) of a pixel array (for example, the pixel array (100) of FIG. 1) is associated with the image data (IDT). The image data (IDT) may include a pixel value corresponding to each of the plurality of pixels (PX). In this specification, a pixel value corresponding to a pixel (PX) may be referred to as a pixel value of the pixel (PX). Any content overlapping with that described in FIG. 1 is omitted.

Referring to FIG. 2, the image data (IDT) may include pixel values of a plurality of pixels (PX). The image data (IDT) may include pixel values corresponding to each of the first green pixels (Gr), the red pixels (R), the blue pixels (B), and the second green pixels (Gb). FIG. 2 illustrates a TETRA pattern in which a red pixel group including red pixels (R) arranged in 2 Υ 2, a first green pixel group including first green pixels (Gr) arranged in 2 Υ 2, a blue pixel group including blue pixels (B) arranged in 2 Υ 2, and a second green pixel group including second green pixels (Gb) arranged in 2 Υ 2 are repeatedly arranged, but this is only an example and is not necessarily limited thereto. The image data (IDT) may include pixel values of a plurality of pixels (PX) forming various patterns.

The image data (IDT) may include a pixel value of at least one bad pixel (BP). The image data (IDT) may include a pixel value of a cluster bad pixel (CBP) and a pixel value of a basic bad pixel (BBP). The cluster bad pixel (CBP) may be a pixel in which a plurality of adjacent bad pixels (BP) form a cluster. For example, the cluster bad pixel (CBP) may include eight adjacent bad pixels (BP). The cluster bad pixel (CBP) may include a 2x4 matrix of bad pixels (BP). However, the present invention is not necessarily limited thereto, and a variety of bad pixels (BP) may form a cluster.

A basic bad pixel (BBP) is a bad pixel (BP) that is not a cluster of adjacent bad pixels (BP), and may be a bad pixel (BP) that is not a cluster bad pixel (CBP). The basic bad pixel (BBP) may be a single bad pixel (BP) that is not a cluster bad pixel (CBP). However, it is not necessarily limited thereto, and the basic bad pixels (BBP) may be two adjacent bad pixels (BP). The cluster bad pixel (CBP) may be arranged to be spaced apart from the basic bad pixel (BBP) by at least one pixel unit, or may be arranged adjacent to the basic bad pixel (BBP).

Although FIG. 2 illustrates one cluster bad pixel (CBP) and one basic bad pixel (BBP), it is not necessarily limited thereto, and the pixel array may include at least one of a cluster bad pixel (CBP) and a basic bad pixel (BBP), or may include one or more cluster bad pixels (CBPs) and one or more basic bad pixels (BBPs).

FIG. 3 is a diagram for explaining the operation of a first bad pixel corrector and a second bad pixel corrector according to an exemplary embodiment of the present disclosure. Any content that overlaps with the above-described content is omitted.

An image signal processor (for example, the image signal processor (300) of FIG. 1) can distinguish a bad pixel from the image data (IDT). The image signal processor can distinguish a cluster bad pixel (CBP) and a basic bad pixel (BBP) based on the location information. For example, the image signal processor can distinguish a pixel value of a cluster bad pixel (CBP) and a pixel value of a basic bad pixel (BBP) and transmit the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP) to a first bad pixel corrector (310). The image signal processor can transmit the pixel value of the cluster bad pixel (CBP) to a second bad pixel corrector (320). However, the present invention is not necessarily limited thereto, and the image signal processor can distinguish location information of a cluster bad pixel (CBP) and location information of a basic bad pixel (BBP). For example, image data (IDT) and position information of cluster bad pixels (CBP) and basic bad pixels (BBP) may be transmitted to a first bad pixel corrector (310). Image data (IDT) and position information of cluster bad pixels (CBP) may be transmitted to a second bad pixel corrector (320).

The first bad pixel corrector (310) can correct the pixel values of bad pixels (BP) according to the first algorithm. The first bad pixel corrector (310) can correct the pixel values of the cluster bad pixel (CBP) and the pixel values of the basic bad pixel (BBP). The first bad pixel corrector (310) can correct the pixel value of the cluster bad pixel (CBP) using the pixel values of the pixels surrounding the cluster bad pixel (CBP). The first bad pixel corrector (310) can correct the pixel value of the basic bad pixel (BBP) using the pixel values of the pixels surrounding the basic bad pixel (BBP). The first bad pixel corrector (310) can correct the pixel values of the cluster bad pixel (CBP) and the basic bad pixel (BBP) to generate the first pixel data (pdt1). The first bad pixel corrector (310) will be described in detail below with reference to FIG. 4.

The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel (CBP) according to the second algorithm. The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel (CBP) using a neural network trained to correct the cluster bad pixel (CBP). The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel (CBP) to generate the second pixel data (pdt2). The second bad pixel corrector (320) will be described in detail later with reference to FIG. 5.

FIG. 4 is a diagram illustrating a first bad pixel corrector according to an exemplary embodiment of the present disclosure.

Referring to FIG. 4, the first bad pixel corrector (310) can correct the pixel value of the bad pixel (BP) for the image data (IDT). The first bad pixel corrector (310) can correct the pixel values of the cluster bad pixel (CBP) and the basic bad pixel (BBP) based on the image data (IDT) and the position information (di). The first bad pixel corrector (310) can receive the image data (IDT) and the position information (di) for the bad pixel (BP). The first bad pixel corrector (310) can perform bad pixel correction for the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP) based on the image data (IDT) and the position information (di). Referring to FIG. 4, the bad pixel (BP) may include a cluster bad pixel (CBP) and a basic bad pixel (BBP), and the first bad pixel corrector (310) may receive location information (di) for the cluster bad pixel (CBP) and the basic bad pixel (BBP). However, it is not necessarily limited thereto, and the first bad pixel corrector (310) may receive only image data corresponding to the bad pixel (BP). That is, the first bad pixel corrector (310) may receive the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP). The first bad pixel corrector (310) may perform bad pixel correction for the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP).

The location information (di) may indicate the location of the bad pixel (BP). For example, the location information (di) may be a bad pixel map indicating a bad pixel (BP) among a plurality of pixels (PX). For example, the location information (di) may be a flag indicating the bad pixel (BP). The flag may include a plurality of bits, and a bit corresponding to the bad pixel (BP) may include a flag value (e.g., '1') indicating the bad pixel (BP), and other bits may include flag values (e.g., '0') indicating general pixels that are not the bad pixel (BP). The location information (di) may also include an identification factor for identifying whether the bad pixel (BP) is a cluster bad pixel (CBP) or a basic bad pixel (BBP).

The first bad pixel corrector (310) can correct the pixel values of bad pixels (BP) according to the first algorithm. The first bad pixel corrector (310) can correct the pixel values of the cluster bad pixels (CBP) by using the pixel values of the pixels surrounding the cluster bad pixels (CBP). The surrounding pixels are pixels surrounding a specific pixel, and may be pixels adjacent to the specific pixel or pixels positioned within a certain distance from the specific pixel.

For example, the surrounding pixels of the cluster bad pixel (CBP) may include a blue pixel (B4), a second green pixel (Gb3), and a first green pixel (Gr14). In the cluster bad pixel (CBP), the pixel value of the first bad pixel (BP1) adjacent to the blue pixel (B4), the second green pixel (Gb3), and the first green pixel (Gr14) may be corrected based on the pixel values of the blue pixel (B4), the second green pixel (Gb3), and the first green pixel (Gr14), respectively. For example, the surrounding pixels of the cluster bad pixel (CBP) may include a blue pixel (B7), a second green pixel (Gb4), and a first green pixel (Gr17). The pixel value of the second bad pixel (BP2) included in the cluster bad pixel (CBP) may be corrected based on the pixel values of the blue pixel (B7), the second green pixel (Gb4), and the first green pixel (Gr17), respectively.

The first bad pixel corrector (310) can correct the pixel value of the basic bad pixel (BBP) by using the pixel values of the surrounding pixels of the basic bad pixel (BBP). For example, the surrounding pixels of the basic bad pixel (BBP) can include a first green pixel (Gr19, Gr20), a red pixel (R19), blue pixels (B17, B19, B20), and a second green pixel (Gb17, Gb19). The pixel value of the basic bad pixel (BBP) can be corrected based on the pixel values of each of the surrounding pixels of the basic bad pixel (BBP).

In one embodiment, the first bad pixel corrector (310) can correct pixel values of the bad pixels based on at least one of a mean filter, a median filter, and a weighted mean filter.

The average filter is a method of taking an average within a certain mask. The first bad pixel corrector (310) can correct the pixel value of a bad pixel (BP) to an average value of pixel values of pixels surrounding the bad pixel (BP) by using the average filter. For example, the first bad pixel corrector (310) can correct the pixel value of a basic bad pixel (BBP) to an average of pixel values of first green pixels (Gr19, Gr20), red pixels (R19), blue pixels (B17, B19, B20), and second green pixels (Gb17, Gb19), which are pixels surrounding the basic bad pixel (BBP).

A median filter is a method of selecting a center value by arranging values in a certain mask in order. The first bad pixel corrector (310) can correct the pixel value of a bad pixel (BP) to the center value of the pixel values of pixels surrounding the bad pixel (BP) by using a median filter.

The weighted average filter is a method that uses the average and variance of values within a certain mask. The first bad pixel corrector (310) can correct the pixel value of a bad pixel (BP) with a weighted average value of the pixel values of pixels surrounding the bad pixel (BP) by using the weighted average filter.

The first bad pixel corrector (310) can correct the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP) to generate first pixel data (pdt1_c, pdt1_b). The first pixel data (pdt1_c) can be a corrected pixel value corresponding to the cluster bad pixel (CBP), and the first pixel data (pdt1_b) can be a corrected pixel value corresponding to the basic bad pixel (BBP). The first pixel data (pdt1_c) can include corrected pixel values corresponding to each of the bad pixels (BP) included in the cluster bad pixel (CBP).

FIG. 5 is a drawing for explaining a second bad pixel corrector according to an exemplary embodiment of the present disclosure. Any details overlapping with the above description are omitted.

Referring to FIG. 5, the second bad pixel corrector (320) can correct the pixel value of a bad pixel (BP) with respect to image data (IDT). The second bad pixel corrector (320) can correct the pixel value of a cluster bad pixel (CBP) based on the image data (IDT) and position information (dic).

The second bad pixel corrector (320) can receive image data (IDT) and position information (dic) for the cluster bad pixel (CBP). Referring to FIG. 5, the bad pixel (BP) may include the cluster bad pixel (CBP) and the basic bad pixel (BBP), but the second bad pixel corrector (320) can receive position information (dic) for the cluster bad pixel (CBP). However, it is not necessarily limited thereto, and the second bad pixel corrector (320) may receive only image data corresponding to the cluster bad pixel (CBP). That is, the second bad pixel corrector (320) may receive the pixel value of the cluster bad pixel (CBP), and may not receive position information (dic).

The location information (dic) may indicate the location of the cluster bad pixel (CBP). The location information (dic) may be location information corresponding to the cluster bad pixel (CBP) among the location information of the bad pixel (BP). For example, the location information (dic) may be a bad pixel map (map) indicating the cluster bad pixel (CBP) among a plurality of pixels (PX). For example, the location information (dic) may be a flag indicating the cluster bad pixel (CBP).

The second bad pixel corrector (320) can correct the pixel values of cluster bad pixels (CBP) according to the second algorithm. The second bad pixel corrector (320) can include a neural network (NN). The neural network (NN) can perform bad pixel processing on the pixel values of cluster bad pixels (CBP) to generate second pixel data (pdt2).

The neural network (NN) may include a deep learning model trained to correct cluster bad pixels (CBPs). The deep learning model may include a plurality of layers, for example, a first to n-th layer. In an embodiment, the neural network (NN) may include a deep learning model trained to correct cluster bad pixels (CBPs) based on training data of various cases including cluster bad pixels (CBPs). For example, the neural network (NN) may include a deep learning model trained to correct cluster bad pixels (CBPs) based on training data including pixel values of bad pixels (BPs) arranged in a 2x4 matrix.

According to an embodiment, the neural network (NN) may include at least one convolutional layer and at least one fully connected layer, and image data (IDT) and location information (dic) may be provided to the neural network (NN).

The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel (CBP) to generate the second pixel data (pdt2). The second pixel data (pdt2) can be a corrected pixel value corresponding to the cluster bad pixel (CBP). The second pixel data (pdt2) can include corrected pixel values corresponding to each of the bad pixels (BP) included in the cluster bad pixel (CBP).

FIG. 6 is a block diagram for explaining an image signal processor according to an exemplary embodiment of the present disclosure. The image signal processor (300), the first bad pixel corrector (310), and the second bad pixel corrector (320) of FIG. 6 correspond to the image signal processor (300), the first bad pixel corrector (310), and the second bad pixel corrector (320) of FIG. 1, respectively, and therefore, overlapping content is omitted.

Referring to FIG. 6, the image signal processor (300) may include a bad pixel controller (330), a bad pixel corrector (310, 320), and a merger (340). The bad pixel controller (330) may distinguish a bad pixel from image data (IDT). The bad pixel controller (330) may distinguish a cluster bad pixel (CBP) and a basic bad pixel (BBP) based on position information. The position information may be stored in the image signal processor (300). However, the present invention is not limited thereto, and the position information may be stored outside the image signal processor (300).

For example, the bad pixel controller (330) may distinguish between the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP) and transmit the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP) to the first bad pixel corrector (310). The bad pixel controller (330) may distinguish between the pixel value of the cluster bad pixel (CBP) and transmit the pixel value of the cluster bad pixel (CBP) to the second bad pixel corrector (320).

For example, the bad pixel controller (330) can distinguish between the location information of the cluster bad pixel (CBP) and the location information of the basic bad pixel (BBP). The bad pixel controller (330) can transfer the image data (IDT) and the location information of the cluster bad pixel (CBP) and the basic bad pixel (BBP) to the first bad pixel corrector (310). The bad pixel controller (330) can transfer the image data (IDT) and the location information of the cluster bad pixel (CBP) to the second bad pixel corrector (320).

The first bad pixel corrector (310) can correct the pixel value of the cluster bad pixel (CBP) and the pixel value of the basic bad pixel (BBP) according to the first algorithm. The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel (CBP). The first bad pixel corrector (310) can generate first pixel data (pdt1_c, pdt1_d) and transmit it to the merger (340). The second bad pixel corrector (320) can generate second pixel data (pdt2) and transmit it to the merger (340). The first bad pixel corrector (310) and the second bad pixel corrector (320) may be implemented as one semiconductor chip or may be implemented as separate semiconductor chips.

The merger (340) can generate corrected image data (IDTa) based on pixel data received from at least one of the first bad pixel corrector (310) and the second bad pixel corrector (320). The corrected image data (IDTa) may be image data (IDT) in which pixel values of bad pixels are corrected based on the pixel data. The corrected image data (IDTa) may be image data (IDT) in which pixel values of basic bad pixels (BBP) and cluster bad pixels (CBP) are corrected based on the pixel data.

The merger (340) can correct the pixel value of the basic bad pixel (BBP) based on the first pixel data (pdt1_b). For example, the merger (340) can replace the pixel value of the basic bad pixel (BBP) in the image data (IDT) with the first pixel data (pdt1_b). The merger (340) can generate corrected image data (IDTa) in which the pixel value of the basic bad pixel (BBP) is replaced with the first pixel data (pdt1_b).

The merger (340) can correct the pixel value of the cluster bad pixel (CBP) based on at least one of the first pixel data (pdt1_c) and the second pixel data (pdt2). For example, the merger (340) can replace the pixel value of the cluster bad pixel (CBP) in the image data (IDT) with the first pixel data (pdt1_c), or can replace the pixel value of the cluster bad pixel (CBP) with the second pixel data (pdt2). However, the present invention is not necessarily limited thereto, and the merger (340) can replace the pixel value of the cluster bad pixel (CBP) with a value obtained by merging the first pixel data (pdt1_c) and the second pixel data (pdt2).

In one embodiment, when the first bad pixel corrector (310) generates first pixel data (pdt1_c) corresponding to the cluster bad pixel (CBP) and the second bad pixel corrector (320) generates second pixel data (pdt2), the merger (340) can generate corrected image data (IDTa) based on the second pixel data (pdt2) among the first pixel data (pdt1_c) and the second pixel data (pdt2) corresponding to the cluster bad pixel (CBP).

The merger (340) can receive first pixel data (pdt1_c) and second pixel data (pdt2) corresponding to the cluster bad pixel (CBP). When each of the first bad pixel corrector (310) and the second bad pixel corrector (320) completes bad pixel correction for the pixel value of the cluster bad pixel (CBP), the pixel data can be transmitted to the merger (340). When the merger (340) receives the first pixel data (pdt1_c) and the second pixel data (pdt2), the merger can select the second pixel data (pdt2) among the first pixel data (pdt1_c) and the second pixel data (pdt2), and generate corrected image data (IDTa) by replacing the pixel value of the cluster bad pixel (CBP) with the second pixel data (pdt2).

In one embodiment, when the first bad pixel corrector (310) generates the first pixel data (pdt1_c) corresponding to the cluster bad pixel (CBP) and the second bad pixel corrector (320) does not generate the second pixel data (pdt2), the merger (340) may generate the corrected image data (IDTa) based on the first pixel data (pdt1_c) corresponding to the cluster bad pixel (CBP). For example, the merger (340) may generate the corrected image data (IDTa) by replacing the pixel value of the cluster bad pixel (CBP) with the first pixel data (pdt1_c).

The merger (340) may not receive the second pixel data (pdt2). Since the second bad pixel corrector (320) performs bad pixel correction using a neural network, the amount of computation may be relatively large. For example, while the second bad pixel corrector (320) operates to correct the pixel value of the first cluster bad pixel, the pixel value of the second cluster bad pixel received by the second bad pixel corrector (320) may be difficult to correct. Therefore, the pixel value of the second cluster bad pixel may be corrected by the first bad pixel corrector (310), and the merger (340) may receive the first pixel data (pdt1_c) corresponding to the second cluster bad pixel. The merger (340) may not receive the second pixel data (pdt2) corresponding to the second cluster bad pixel.

The merger (340) receives the first pixel data (pdt1_c) corresponding to the second cluster bad pixel, and since the second pixel data (pdt2) is not received, the merger can generate the correction image data (IDTa) based on the first pixel data (pdt1_c). The merger (340) can replace the pixel value of the cluster bad pixel (CBP) of the image data (IDT) with the first pixel data (pdt1_c). The merger (340) can generate the correction image data (IDTa) including the first pixel data (pdt1_c) and the first pixel data (pdt1_b).

FIG. 7 is a diagram for explaining the operation of a merger according to an exemplary embodiment of the present disclosure. Specifically, FIG. 7 explains a case where a merger (340) receives first pixel data (pdt1_c, pdt1_b) and second pixel data (pdt2). Any content that overlaps with the above description is omitted.

The merger (340) can receive first pixel data (pdt1_c, pdt1_b) and second pixel data (pdt2). The merger (340) can replace the pixel value of the basic bad pixel (BBP) in the image data (IDT) with the first pixel data (pdt1_b). The merger (340) can generate corrected image data (IDTa) in which the pixel value of the basic bad pixel (BBP) is replaced with the first pixel data (pdt1_b).

When the merger (340) receives the first pixel data (pdt1_c) and the second pixel data (pdt2), the merger may select the second pixel data (pdt2) from the first pixel data (pdt1_c) and the second pixel data (pdt2), and may generate the correction image data (IDTa) by replacing the pixel value of the cluster bad pixel (CBP) with the second pixel data (pdt2). The correction image data (IDTa) may include the first pixel data (pdt1_b) in which the pixel value of the basic bad pixel is replaced, and the second pixel data (pdt2) in which the pixel value of the cluster bad pixel is replaced.

FIG. 8 is a block diagram illustrating components of an image signal processor according to an exemplary embodiment of the present disclosure. Any content that overlaps with the above description is omitted.

Referring to FIG. 8, the image signal processor (300) may include a bad pixel controller (330), a pre-processor (350), a first bad pixel corrector (310), and a second bad pixel corrector (320). The pre-processor (350) may determine whether a cluster bad pixel corresponds to a first cluster type. The first cluster type may mean that a pixel value of the cluster bad pixel needs to be corrected according to a second algorithm. That is, the pre-processor (350) may determine whether the cluster bad pixel needs to be corrected according to the second algorithm.

In one embodiment, the pre-processor (350) may determine whether the cluster bad pixel corresponds to the first cluster type and may inform the second bad pixel corrector (320) of the determination result. For example, the pre-processor (350) may determine whether the cluster bad pixel corresponds to the first cluster type and may generate a determination signal based on the determination. The determination signal may be a signal that informs the second bad pixel corrector (320) of whether the cluster bad pixel corresponds to the first cluster type.

For example, if the pre-processor (350) determines that the cluster bad pixel corresponds to the first cluster type, the pre-processor (350) may be implemented to transfer the pixel value of the cluster bad pixel to the second bad pixel corrector (320). If the pre-processor (350) determines that the cluster bad pixel does not correspond to the first cluster type, the pre-processor (350) may not transfer the pixel value of the cluster bad pixel to the second bad pixel corrector (320).

The pre-processor (350) can determine whether the cluster bad pixel corresponds to the first cluster type based on the pixel values of the pixels included in the judgment area. The judgment area may be an area of a preset size that includes the cluster bad pixel in the pixel array. For example, the judgment area may be an area of a 10x10 pixel size that includes the cluster bad pixel, or an area of a 12x12 pixel size that includes the cluster bad pixel. However, the present invention is not necessarily limited thereto.

In one embodiment, the pre-processor (350) may determine whether the cluster bad pixel corresponds to the first cluster type based on whether the judgment region includes at least one of an edge region and a high-frequency region. The edge region may refer to a region where the pixel value of the pixel changes rapidly, and the high-frequency region may refer to a region where the pixel value of the pixel changes repeatedly. The edge region and the high-frequency region may be determined based on the pixel values of the pixels included in the judgment region.

The pre-processor (350) may determine that the pixel value of a cluster bad pixel included in the judgment area needs to be corrected according to the second algorithm when the judgment area includes at least one of an edge area and a high-frequency area. That is, the pre-processor (350) may determine that a cluster bad pixel corresponds to the first cluster type when the judgment area includes at least one of an edge area and a high-frequency area.

In one embodiment, the pre-processor (350) may determine whether the cluster bad pixel corresponds to the first cluster type based on the gradient of each of the pixel values of the pixels included in the judgment area. The edge area and the high frequency area may be determined based on the gradient of the pixel values of the pixels included in the judgment area. For example, whether the cluster bad pixel corresponds to the first cluster type may be determined based on at least one of an intra gradient and an inter gradient. The intra gradient and the inter gradient are described in detail later in FIGS. 10 and 11. In FIG. 8, the pre-processor (350) is illustrated as a separate logic from the bad pixel controller (330), but is not necessarily limited thereto, and the pre-processor (350) may also be included in the bad pixel controller (330).

The first bad pixel corrector (310) can correct the pixel values of the basic bad pixel and the cluster bad pixel. The first bad pixel corrector (310) can correct the pixel value of the cluster bad pixel according to the first algorithm regardless of whether the cluster bad pixel corresponds to the first cluster type. However, it is not necessarily limited thereto, and according to an embodiment, the first bad pixel corrector (310) may be implemented to correct the pixel value of the cluster bad pixel when the cluster bad pixel does not correspond to the first cluster type.

The second bad pixel corrector (320) can correct the pixel value of the cluster bad pixel depending on whether the cluster bad pixel corresponds to the first cluster type. The second bad pixel corrector (320) can receive a judgment signal from the pre-processor (350) and correct the pixel value of the cluster bad pixel based on the judgment signal.

The second bad pixel corrector (320) may correct the pixel value of the cluster bad pixel according to the second algorithm using a neural network if the cluster bad pixel corresponds to the first cluster type. If the second bad pixel corrector (320) determines that the cluster bad pixel does not correspond to the first cluster type, the pixel value of the cluster bad pixel may not be corrected.

A pixel array may include a plurality of cluster bad pixels, and pixel values of the cluster bad pixels may be continuously transmitted to the image signal processor (300). If the pixel values of all cluster bad pixels are corrected according to the second algorithm, the amount of computation may be large, and it may be difficult to perform bad pixel correction in real time. The image sensor according to the present disclosure can identify the type of a cluster bad pixel, and if it corresponds to the first cluster type, correct the pixel value of the cluster bad pixel using a neural network, so that the number of cluster bad pixels corrected using the neural network can be reduced. Accordingly, the amount of computation and power consumption required to perform bad pixel correction can be reduced.

Referring to FIG. 8, the image signal processor (300) may further include a post-processor (360). The post-processor (360) may determine the reliability of the second bad pixel corrector (320). That is, the post-processor (360) may determine the reliability of the second algorithm. The second bad pixel corrector (320) may include a deep learning model trained to correct cluster bad pixels, and may correct pixel values of the cluster bad pixels using the deep learning model. Since the deep learning model is trained based on training data of various cases including cluster bad pixels, the reliability may decrease in the case of untrained cluster bad pixels. Therefore, the reliability of the second bad pixel corrector (320) needs to be determined.

The post-processor (360) can determine the reliability of the second bad pixel corrector (320) based on the second pixel data generated by correcting the pixel value of the cluster bad pixel. In one embodiment, the post-processor (360) can determine the reliability of the second bad pixel corrector (320) based on the first pixel data and the second pixel data from the first bad pixel corrector (310).

The post-processor (360) can compare the first pixel data and the second pixel data corresponding to the cluster bad pixel to determine the reliability of the second bad pixel corrector (320). For example, if the difference between the first pixel data and the second pixel data corresponding to the cluster bad pixel is greater than or equal to a preset value, the post-processor (360) can determine that the reliability of the second bad pixel corrector (320) is low or non-existent. If the difference between the first pixel data and the second pixel data corresponding to the cluster bad pixel is less than a preset value, the post-processor (360) can determine that the reliability of the second bad pixel corrector (320) is high or non-existent.

In one embodiment, the post-processor (360) can determine the reliability of the second bad pixel corrector (320) based on the pixel values of the surrounding pixels of the cluster bad pixel and the second pixel data corresponding to the cluster bad pixel. For example, if the difference between the pixel values of the surrounding pixels of the cluster bad pixel and the second pixel data is greater than or equal to a preset value, the post-processor (360) can determine that the reliability of the second bad pixel corrector (320) is low or non-existent. If the difference between the pixel values of the surrounding pixels of the cluster bad pixel and the second pixel data is less than a preset value, the post-processor (360) can determine that the reliability of the second bad pixel corrector (320) is high or non-existent.

The merger (340) can generate corrected image data based on the reliability determination of the second bad pixel corrector (320). If the post-processor (360) determines that the second bad pixel corrector (320) is reliable, the merger (340) can generate corrected image data based on at least one of the second pixel data and the first pixel data corresponding to the cluster bad pixel. For example, if the second bad pixel corrector (320) is reliable, the merger (340) can generate corrected image data by replacing the pixel value of the cluster bad pixel with the second pixel data.

If the post-processor (360) determines that the second bad pixel corrector (320) is unreliable, the merger (340) can generate corrected image data based on the first pixel data corresponding to the cluster bad pixel. If the second bad pixel corrector (320) is unreliable, the merger (340) can generate corrected image data by replacing the pixel value of the cluster bad pixel with the first pixel data corresponding to the cluster bad pixel.

An image sensor according to an exemplary embodiment of the present disclosure determines the reliability of a second bad pixel corrector (320) and generates corrected image data based on the reliability of the second bad pixel corrector (320), so that pixel values of bad pixels can be corrected more accurately. Accordingly, the performance of the image sensor for correcting pixel values of bad pixels can be improved.

FIG. 9A is a diagram for explaining the operation of a pre-processor according to an exemplary embodiment of the present disclosure. In FIG. 9A, each of a plurality of pixels of a pixel array is represented as corresponding to image data (IDT). The pre-processor (350) and the second bad pixel corrector (320) of FIG. 9A correspond to the pre-processor (350) and the second bad pixel corrector (320) of FIG. 8, respectively, and therefore, overlapping content is omitted.

The pre-processor (350) can determine whether the cluster bad pixel corresponds to the first cluster type. The pre-processor (350) can determine whether the cluster bad pixel needs to be corrected according to the second algorithm. In one embodiment, the pre-processor (350) can determine whether the cluster bad pixel corresponds to the first cluster type and can notify the second bad pixel corrector (320) of the determination result. For example, the pre-processor (350) can determine whether the cluster bad pixel corresponds to the first cluster type and generate a determination signal based on the determination, and the pre-processor (350) can be implemented to transfer the pixel value of the cluster bad pixel to the second bad pixel corrector (320). In FIG. 9A, it is assumed and described that the pixel value of the cluster bad pixel is transferred to the second bad pixel corrector (320) depending on whether the cluster bad pixel corresponds to the first cluster type.

The pre-processor (350) can determine whether the cluster bad pixel corresponds to the first cluster type based on the pixel values of the pixels included in the first judgment area (DA1). The first judgment area (DA1) can be an area of a preset size that includes the first cluster bad pixel (CBP1) in the pixel array. For example, the first judgment area (DA1) can be an area of 10x10 pixels in size, but is not necessarily limited thereto.

In one embodiment, the pre-processor (350) can determine whether the first cluster bad pixel (CBP1) corresponds to the first cluster type based on whether the first judgment area (DA1) includes at least one of an edge area (ea) and a high-frequency area. The edge area (ea) and the high-frequency area can be determined based on pixel values of pixels included in the judgment area.

The edge area (ea) and the high-frequency area can be determined based on the gradient of each of the pixel values of the pixels included in the first judgment area (DA1). For example, the pre-processor (350) can determine the first green pixel (Gr1) as the edge area (ea) because the difference between the pixel value of the first green pixel (Gr1) and the pixel values of the first green pixel (Gr2), the first green pixel (Gr3), and the first green pixel (Gr4) among the pixel values of the pixels included in the first judgment area (DA1) is greater than or equal to a preset threshold value.

The pre-processor (350) can determine that the pixel value of the first cluster bad pixel (CBP1) corresponds to the first cluster type because the edge area (ea) is included in the first judgment area (DA1). The pre-processor (350) can transfer the pixel value of the first cluster bad pixel (CBP1) to the second bad pixel corrector (320). The second bad pixel corrector (320) can correct the pixel value of the first cluster bad pixel (CBP1) using a neural network.

FIG. 9b is a diagram for explaining the operation of a pre-processor according to an exemplary embodiment of the present disclosure. FIG. 9b shows a case where the pixel value of a cluster bad pixel is not transmitted to the second bad pixel corrector (320). The overlapping content with FIG. 9a is omitted.

The pre-processor (350) can determine whether the cluster bad pixel corresponds to the first cluster type based on the pixel values of the pixels included in the second judgment area (DA2). The second judgment area (DA2) can be an area of a preset size that includes the second cluster bad pixel (CBP2) in the pixel array. For example, the second judgment area (DA2) can be an area of 8x8 pixels in size, but is not necessarily limited thereto.

In one embodiment, the pre-processor (350) may determine whether the second cluster bad pixel (CBP2) corresponds to the first cluster type based on whether the second judgment area (DA2) includes at least one of an edge area and a high-frequency area. If the second judgment area (DA2) includes only a flat area, not an edge area and a high-frequency area, the pre-processor (350) may determine that the second cluster bad pixel (CBP2) does not correspond to the first cluster type.

For example, the pre-processor (350) may determine that the second cluster bad pixel (CBP2) does not correspond to the first cluster type when the second judgment area (DA2) includes a flare area. The flare area may refer to a portion where the image appears bright in a different portion from the object when photographing an object with strong light. The flare area may be determined based on the gradient of each pixel value of the pixels included in the second judgment area (DA2).

For example, the edge region and the high frequency region may not be included in the second judgment area (DA2). The pre-processor (350) may determine that the pixel value of the second cluster bad pixel (CBP2) does not correspond to the first cluster type. That is, the pixel value of the second cluster bad pixel (CBP2) may not need to be corrected according to the second algorithm. The pre-processor (350) may not transfer the pixel value of the second cluster bad pixel (CBP2) to the second bad pixel corrector (320). The second bad pixel corrector (320) may not correct the pixel value of the second cluster bad pixel (CBP2).

FIG. 10 is a diagram for explaining an inter gradient according to an exemplary embodiment of the present disclosure. In FIG. 10, each of a plurality of pixels of a pixel array is represented as corresponding to image data (IDT).

A pre-processor (e.g., pre-processor (350) of FIG. 8) can determine whether a cluster bad pixel (CBP) corresponds to a first cluster type based on a gradient of each pixel value of pixels included in a judgment area (DA). Referring to FIG. 10, the judgment area (DA) may have a size of 10x10 pixels and may include a cluster bad pixel (CBP).

The pre-processor can calculate the gradient of the target pixel. The target pixel can mean a pixel that is a target for calculating the gradient of pixel values within the judgment area (DA). The pre-processor can calculate the inter gradient of the target pixel. The inter gradient of the target pixel can be calculated based on the pixel value of each pixel of the same color channel as the target pixel among the pixels included in the judgment area (DA).

For example, assume that the target pixel is a first green pixel (Gr1). The pre-processor can calculate the inter gradient of the first green pixel (Gr1). The pre-processor can calculate the inter gradient of the first green pixel (Gr1) based on the pixel value of each pixel in the same color channel as the first green pixel (Gr1). The pixel in the same color channel as the first green pixel (Gr1) may be a pixel in a position corresponding to the first green pixel (Gr1) among the first green pixels. The first green pixel (Gr5), the first green pixel (Gr9), the first green pixel (Gr13), the first green pixel (Gr17), the first green pixel (Gr21), the first green pixel (Gr25), and the first green pixel (Gr9) may be pixels in the same color channel as the first green pixel (Gr1). The pre-processor can calculate an inter gradient of a pixel value of the first green pixel (Gr1) based on the pixel values of each of the first green pixel (Gr5), the first green pixel (Gr9), the first green pixel (Gr13), the first green pixel (Gr17), the first green pixel (Gr21), the first green pixel (Gr25), and the first green pixel (Gr9).

The pre-processor can determine whether the cluster bad pixel (CBP) corresponds to the first cluster type based on the inter gradient of the target pixel. The pre-processor can calculate the inter gradient of each pixel included in the judgment area (DA) and determine whether the judgment area (DA) includes at least one of an edge area and a high-frequency area.

In one embodiment, the pre-processor may determine that the cluster bad pixel (CBP) corresponds to the first cluster type if there is a pixel among the pixels included in the judgment area (DA) whose inter gradient is greater than or equal to a threshold value. The pixel among the pixels included in the judgment area (DA) whose inter gradient is greater than or equal to the threshold value may be at least one of an edge area and a high-frequency area. For example, assume that the threshold value is 0.12. If the inter gradient of the first green pixel (Gr1) is greater than or equal to 0.12, the pre-processor may determine that the cluster bad pixel (CBP) corresponds to the first cluster type because the first green pixel (Gr1) corresponds to the edge area.

The pre-processor can determine that the judgment area (DA) does not include an edge area and a high-frequency area if the inter-gradient of each of all pixels included in the judgment area (DA) is less than a threshold value. The pre-processor can determine that the cluster bad pixel (CBP) does not correspond to the first cluster type.

Fig. 11 is a diagram for explaining an intergradient according to an exemplary embodiment of the present disclosure. In Fig. 11, each of a plurality of pixels of a pixel array is represented as corresponding to image data (IDT). The same content as in Fig. 10 is omitted.

Referring to FIG. 10, the judgment area (DA) may have a size of 10x10 pixels and may include a cluster bad pixel (CBP). The judgment area (DA) may include a plurality of pixel groups in which pixels of the same color are grouped together. For example, the judgment area (DA) may include a first group (G1), a second group (G2), and a third group (G3) in which first green pixels are grouped together. The first group (G1) may include a first green pixel (Gr1), a first green pixel (Gr2), a first green pixel (Gr3), and a first green pixel (Gr4).

The pre-processor can calculate the intra gradient of the target pixel. The intra gradient of the target pixel can be calculated based on the pixel value of each pixel included in the target pixel group including the target pixel among the plurality of pixel groups. For example, if the target pixel is the first green pixel (Gr1), the target pixel group can be the first group (G1).

Assume that the target pixel is a first green pixel (Gr1). The pre-processor can calculate the intra gradient of the first green pixel (Gr1). The pre-processor can calculate the intra gradient of the first green pixel (Gr1) based on the pixel value of each pixel in the same pixel group as the first green pixel (Gr1). The pixels in the same pixel group as the first green pixel (Gr1) may be the first green pixel (Gr2), the first green pixel (Gr3), and the first green pixel (Gr4). The pre-processor can calculate the intra gradient of the pixel value of the first green pixel (Gr1) based on the pixel values of each of the first green pixel (Gr2), the first green pixel (Gr3), and the first green pixel (Gr4).

The pre-processor can determine whether the cluster bad pixel (CBP) corresponds to the first cluster type based on the intra gradient of the target pixel. The pre-processor can calculate the intra gradient of each pixel included in the judgment area (DA) and determine whether the judgment area (DA) includes at least one of an edge area and a high-frequency area.

In one embodiment, the pre-processor may determine that the cluster bad pixel (CBP) corresponds to the first cluster type if there is a pixel among the pixels included in the judgment area (DA) whose intra gradient is greater than or equal to a threshold value. The pixel among the pixels included in the judgment area (DA) whose intra gradient is greater than or equal to the threshold value may be at least one of an edge area and a high-frequency area. For example, assume that the threshold value is 0.12. If the intra gradient of the first green pixel (Gr1) is 0.12 or greater, the pre-processor may determine that the cluster bad pixel (CBP) corresponds to the first cluster type because the first green pixel (Gr1) corresponds to the edge area.

The pre-processor can determine that the judgment area (DA) does not include an edge area and a high-frequency area if the intra gradient of each of all pixels included in the judgment area (DA) is less than a threshold value. The pre-processor can determine that the cluster bad pixel (CBP) does not correspond to the first cluster type.

In one embodiment, the pre-processor may determine that the cluster bad pixel (CBP) corresponds to the first cluster type if there is a pixel in which at least one of the inter gradient and the intra gradient of each of the pixels included in the judgment area (DA) is greater than or equal to a threshold value. The pre-processor may determine that the cluster bad pixel (CBP) does not correspond to the first cluster type if the inter gradient of each of the pixels included in the judgment area (DA) is less than the threshold value and the intra gradient is less than the threshold value.

The pre-processor can determine that the cluster bad pixel (CBP) does not correspond to the first cluster type because the judgment area (DA) only includes a flat region if the inter-gradient and intra-gradient of each pixel included in the judgment area (DA) are less than a threshold value. If only a flat region exists in the judgment area (DA), the pixel value of the cluster bad pixel (BP) can be corrected according to the first algorithm and may not be corrected according to the second algorithm.

In one embodiment, the pre-processor can determine whether the judgment area (DA) includes a flare area. If the judgment area (DA) includes a flare area, the pre-processor can determine that the cluster bad pixel (CBP) does not correspond to the first cluster type. The pre-processor can determine the flare area based on the inter gradient and intra gradient of each of the pixels included in the judgment area (DA).

The pre-processor may determine that a pixel in the judgment area (DA) whose inter gradient is less than a first threshold value and whose intra gradient is greater than or equal to a second threshold value corresponds to the play area. For example, assume that the first threshold value is 0.12 and the second threshold value is 0.85. The pre-processor may determine that the first green pixel (Gr1) corresponds to the flare area if the inter gradient of the first green pixel (Gr1) is less than 0.12 and the intra gradient is greater than or equal to 0.85. The pre-processor may determine that the cluster bad pixel (CBP) does not correspond to the first cluster type.

FIG. 12 is a flowchart for explaining an operation method of an image sensor according to an exemplary embodiment of the present disclosure.

In step S1210, the image sensor can generate image data. The image sensor can include a plurality of pixels, and the plurality of pixels can be arranged in a matrix. The plurality of pixels can convert a received optical signal into an electrical signal. The image sensor can convert the electrical signal generated from the plurality of pixels into image data including pixel values of each of the plurality of pixels. The pixel values can be digital values.

In step S1220, the image sensor can distinguish cluster bad pixels and basic bad pixels from the image data based on the location information. The image data may include pixel values due to bad pixels. A bad pixel may refer to a pixel whose pixel value difference from surrounding pixels is greater than or equal to a reference value, or which does not have a pixel value corresponding to a specific gradation when the image data represents an image of the specific gradation.

The bad pixel may include a cluster bad pixel and a basic bad pixel. The cluster bad pixel may refer to a pixel generated by a plurality of adjacent bad pixels forming a cluster. may. A pixel that is not a cluster bad pixel among the bad pixels may be referred to as a basic bad pixel. The image sensor may distinguish the cluster bad pixel and the basic bad pixel based on position information. The position information may indicate positions of the cluster bad pixel and the basic bad pixel among the plurality of pixels (PX) and may be stored in the image sensor. For example, the image sensor may distinguish the pixel value of the cluster bad pixel and the pixel value of the basic bad pixel based on the position information. For example, the image sensor may also distinguish the position information of the cluster bad pixel and the position information of the basic bad pixel.

In step S1230, the image sensor can perform bad pixel correction on the image data. The image sensor can correct pixel values of bad pixels among pixel values of a plurality of pixels included in the image data. The image sensor can correct the pixel value of the basic bad pixel according to the first algorithm, and correct the pixel value of the cluster bad pixel according to at least one of the first algorithm and the second algorithm.

The image sensor can correct the pixel value of each of all bad pixels according to a first algorithm. The image sensor can correct the pixel value of the cluster bad pixel and the pixel value of the basic bad pixel according to the first algorithm. The first algorithm is an algorithm that corrects the pixel value of the bad pixels by using the pixel values of the surrounding pixels of each of the bad pixels, and can utilize a hand-crafted feature. The image sensor can correct the pixel value of the cluster bad pixel according to the first algorithm that uses the pixel values of the surrounding pixels of the cluster bad pixel. The image sensor can correct the pixel value of the basic bad pixel according to the first algorithm that uses the pixel values of the surrounding pixels of the basic bad pixel.

The image sensor can correct the pixel value of the cluster bad pixel according to the second algorithm. The second algorithm is an algorithm that uses a neural network trained to correct the cluster bad pixel, and may be different from the first algorithm. The image sensor can correct the pixel value of the cluster bad pixel using the neural network.

In one embodiment, the image sensor can determine whether the pixel value of a cluster bad pixel corresponds to a first cluster type that needs to be corrected using a neural network. If the image sensor determines that the cluster bad pixel corresponds to the first cluster type, the image sensor can correct the pixel value of the cluster bad pixel according to a second algorithm.

The image sensor can determine whether the cluster bad pixel corresponds to the first cluster type based on pixel values of pixels included in the judgment region. The judgment region can be a region of a preset size that includes the cluster bad pixel in the pixel array. The image sensor can determine whether the cluster bad pixel corresponds to the first cluster type based on whether the judgment region includes at least one of an edge region and a high-frequency region. For example, when the judgment region includes at least one of an edge region and a high-frequency region, the image sensor can determine that the pixel value of the cluster bad pixel included in the judgment region needs to be corrected according to the second algorithm. The image sensor can determine whether the cluster bad pixel corresponds to the first cluster type based on a gradient of each of the pixel values of the pixels included in the judgment region.

FIG. 13 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. For example, the electronic device (1000) may be a portable terminal.

Referring to FIG. 13, an electronic device (1000) according to an exemplary embodiment of the present disclosure may include an application processor (1100), an image sensor (1200), a display device (1300), a working memory (1400), a storage (1500), a user interface (1600), and a wireless transceiver (1700). The description of the image sensor and the operating method of the image sensor according to the exemplary embodiments of the present disclosure described in FIGS. 1 to 12 may be applied to the image sensor (1200).

The application processor (1100) controls the overall operation of the electronic device (1000) and may be provided as a system on chip (SoC) that runs application programs, an operating system, etc.

The application processor (1100) can receive output data from the image sensor (1200).

The image sensor (1200) can generate image data, for example, image data, based on a received optical signal and provide the image data to the application processor (1100). The image sensor (1200) can perform bad pixel correction. The image sensor (1200) can include a first bad pixel corrector (1210) and a second bad pixel corrector (1220).

The first bad pixel corrector (1210) can correct a bad pixel value by using pixel values of surrounding pixels of each of a plurality of pixels of the image sensor (1200). The first bad pixel corrector (1210) can correct the pixel values of a basic bad pixel and a cluster bad pixel. The second bad pixel corrector (1220) can correct the pixel values of a cluster bad pixel included in the image data by using a neural network trained to correct the cluster bad pixel.

The working memory (1400) may be implemented as a volatile memory such as DRAM, SRMA, or a nonvolatile resistive memory such as FeRAM, RRAM, or PRAM. The working memory (1400) may store programs and/or data processed or executed by the application processor (1100).

The storage (1500) may be implemented as a non-volatile memory device such as NADN flash, resistive memory, etc., and for example, the storage (1500) may be provided as a memory card (MMC, eMMC, SD, micro SD), etc. The storage (1500) may store data and/or a program for an execution algorithm that controls an image processing operation of the image sensor (1200), and when the image processing operation is performed, the data and/or the program may be loaded into the working memory (1400). In an embodiment, the storage (1500) may store output image data generated by the image sensor (1200), such as corrected image data or post-processed image data.

The user interface (1600) may be implemented with various devices capable of receiving user input, such as a keyboard, curtain key panel, touch panel, fingerprint sensor, microphone, etc. The user interface (1600) may receive user input and provide a signal corresponding to the received user input to the application processor (1100).

The wireless transceiver (1700) may include a transceiver (1720), a modem (1710), and an antenna (1730).

As described above, exemplary embodiments have been disclosed in the drawings and the specification. Although specific terms have been used in the specification to describe the embodiments, these have been used only for the purpose of explaining the technical idea of the present disclosure and have not been used to limit the meaning or the scope of the present disclosure set forth in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present disclosure should be determined by the technical idea of the appended claims.

Claims (10)

A pixel array comprising a plurality of pixels that convert a received optical signal into an electrical signal;
A readout circuit that converts the electrical signal into image data including pixel values of each of the plurality of pixels and outputs the image data;
A first bad pixel corrector that corrects pixel values of the bad pixels included in the image data by using pixel values of pixels surrounding each of the bad pixels among the plurality of pixels and generates first pixel data; and
An image sensor comprising: a second bad pixel corrector for correcting pixel values of cluster bad pixels included in the image data using a neural network trained to correct cluster bad pixels, and generating second pixel data. In the first paragraph,
The above image sensor,
A pre-processor for determining whether the above cluster bad pixel corresponds to the first cluster type,
The above pre-processor,
An image sensor characterized in that it is determined whether the cluster bad pixel corresponds to the first cluster type based on pixel values of pixels included in a judgment area of a preset size and including the cluster bad pixel in the pixel array. In the second paragraph,
The above pre-processor,
An image sensor characterized in that it is determined whether the cluster bad pixel corresponds to the first cluster type based on the gradient of each pixel value of the pixels included in the judgment area. In the third paragraph,
The above pre-processor,
An image sensor characterized in that the inter gradient of the target pixel is calculated based on the pixel value of each pixel of the same color channel as the target pixel among the pixels included in the judgment area. In the third paragraph,
The above judgment area is,
Contains multiple groups of pixels grouped together with pixels of the same color,
The above pre-processor,
An image sensor characterized in that the intra gradient of the target pixel is calculated based on the pixel value of each pixel included in the target pixel group including the target pixel among the plurality of pixel groups. In the second paragraph,
If the above cluster bad pixel corresponds to the first cluster type,
The second bad pixel corrector is,
An image sensor characterized in that the pixel value of the cluster bad pixel is corrected using the above neural network. In the first paragraph,
The above image sensor,
Further comprising a post-processor for judging the reliability of the second bad pixel corrector,
The above post-processor,
An image sensor characterized in that the reliability of the second bad pixel corrector is determined based on a comparison of the first pixel data and the second pixel data corresponding to the cluster bad pixel. In the first paragraph,
The above image sensor,
Further comprising a post-processor for judging the reliability of the second bad pixel corrector,
The above post-processor,
An image sensor characterized in that the reliability of the second bad pixel corrector is determined based on pixel values of pixels surrounding the cluster bad pixel and the second pixel data corresponding to the cluster bad pixel. A pixel array comprising a plurality of pixels that convert a received optical signal into an electrical signal;
A readout circuit that converts the electrical signal into image data including pixel values of each of the plurality of pixels and outputs the image data; and
An image signal processor for correcting at least one of a pixel value of a cluster bad pixel and a pixel value of a basic bad pixel in the image data based on position information indicating positions of a cluster bad pixel and a basic bad pixel in the pixel array;
The above image signal processor,
An image sensor characterized in that the pixel value of the cluster bad pixel and the pixel value of the basic bad pixel are corrected according to a first algorithm that uses pixel values of surrounding pixels of each of the cluster bad pixel and the basic bad pixel, and the pixel value of the cluster bad pixel is corrected according to a second algorithm that uses a neural network trained to correct the cluster bad pixel. In a method of operating an image sensor including a plurality of pixels,
A step of generating image data including pixel values of each of the plurality of pixels based on an electrical signal from a received optical signal;
A step of distinguishing the cluster bad pixel and the basic bad pixel from the image data based on position information indicating the positions of the cluster bad pixel and the basic bad pixel among the plurality of pixels; and
An operating method of an image sensor, comprising: a step of correcting a pixel value of the basic bad pixel according to a first algorithm; and a step of correcting a pixel value of the cluster bad pixel according to at least one of the first algorithm and a second algorithm that is different from the first algorithm and uses a neural network.

Images