KR20240145854A - Electronic apparatus comprising image sensor and operating method thereof - Google Patents

Abstract

An electronic device according to various embodiments includes at least one camera including an image sensor, a memory storing instructions, and at least one processor, wherein the processor may be configured to transmit, to the image sensor or at least one camera, information on an output area to be included in an image frame output by the changed operation mode or the changed camera when an operation mode of the image sensor is changed or a camera used to capture an image is switched to another camera.

Description

{ELECTRONIC APPARATUS COMPRISING IMAGE SENSOR AND OPERATING METHOD THEREOF}

The present disclosure relates to an electronic device including an image sensor and a method of operating the electronic device.

Electronic devices can acquire digital images using digital cameras. Digital cameras can acquire data for acquiring digital images using image sensors that detect light and output it as electrical signals. For example, data for acquiring images of scenes can be acquired using charge coupled device (CCD) image sensors or complementary metal-oxide semiconductor (CMOS) image sensors (CIS).

With the advancement of technology, pixels that detect light at a high density can be arranged on an image sensor. Accordingly, the image sensor can support high-resolution image capture. The image sensor can operate in various modes. For example, the image sensor can perform a binning operation to combine multiple pixel information and output image data. The binning operation can be performed analogically through a circuit configuration or through digital operation.

Additionally, a camera including an image sensor can capture images based on a field of view (FOV) that the camera can support. The field of view may refer to a range that the camera can capture through a lens included in the camera. For example, the lens may be referred to as an ultra-wide angle lens, a wide angle lens, a standard lens, or a telephoto lens in that order of increasing angle of view. A camera equipped with each lens may be referred to as an ultra-wide angle camera, a wide angle camera, a standard camera, or a telephoto camera.

The electronic device may include a plurality of cameras to capture images based on different angles of view. The plurality of cameras may output image frames containing data for images captured at different angles of view. The electronic device may provide images according to various conditions by using the image frames captured at different angles of view.

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

An electronic device according to one embodiment may include a camera including an image sensor outputting image data, a memory storing instructions, and at least one processor. The at least one processor may be configured to, when executing the instructions, obtain a first image frame through the image sensor based on a first magnification within a reference range. The at least one processor may be configured to transmit a signal including output area information indicating a range of data to be output by the image sensor to the image sensor. The image sensor may be configured to output a second image frame including an image corresponding to the second magnification to the at least one processor based on the output area information, based on a magnification related to capturing the image being changed to a second magnification exceeding the reference range.

According to one embodiment, a method of operating an electronic device including a camera including an image sensor may include an operation in which at least one processor of the electronic device acquires a first image frame through the image sensor based on a first magnification within a reference range. The method may include an operation in which the at least one processor transmits a signal including output area information indicating a range of data to be output by the image sensor to the image sensor. The method may include an operation in which the image sensor outputs a second image frame including an image corresponding to the second magnification based on the output area information to the at least one processor based on a change in a magnification related to capturing the image to a second magnification exceeding the reference range.

An electronic device according to one embodiment may include a first camera supporting a first angle of view, a second camera supporting a second angle of view different from the first angle of view, a memory storing instructions, and at least one processor. The at least one processor may be configured to, when executing the instructions, obtain a first image frame through the first camera based on a first magnification within a reference range related to capturing an image. The at least one processor may be configured to transmit a signal including output area information indicating a range of data to be output by an image sensor included in the second camera to the second camera based on a change in the magnification related to capturing the image to a second magnification out of the reference range. The at least one processor may be configured to obtain a second image frame including an image corresponding to the second magnification from the second camera based on the signal including the output area information.

A computer-readable recording medium according to one embodiment may store instructions that, when executed, cause an electronic device to perform the method or operation of the electronic device described above.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
FIG. 2 is a block diagram illustrating a camera module according to various embodiments.
FIG. 3 is a drawing for explaining an example of operation based on multiple operation modes of an image sensor included in an electronic device according to one embodiment.
FIG. 4 is a drawing illustrating an example of an electronic device including a plurality of cameras according to one embodiment.
FIG. 5 is a flowchart illustrating a process by which an electronic device acquires an image frame according to one embodiment.
FIG. 6 is a flowchart illustrating a process by which an electronic device changes a magnification related to capturing an image based on a single camera according to one embodiment.
FIG. 7 is a flowchart illustrating a process by which an electronic device according to one embodiment changes a magnification related to capturing an image based on a first camera and a second camera.
FIG. 8 is a diagram conceptually explaining an operation of an electronic device according to one embodiment of the present invention to output an image frame based on shooting area information according to a changing magnification.
FIG. 9 is a drawing for explaining an operation of an electronic device according to one embodiment of the present invention to change a camera based on a changing magnification.
FIG. 10 is a drawing illustrating an example of a screen displayed by an electronic device in an operation of changing a magnification according to one embodiment.
FIG. 11 is a diagram illustrating an example of an operation of an electronic device according to one embodiment of the present invention to enlarge a preview image based on a user input.
FIG. 12 is a diagram illustrating information output by a processor and an image sensor in an operation in which the operation mode of an image sensor of an electronic device is changed according to one embodiment.
FIG. 13 is a diagram illustrating an example in which an electronic device according to one embodiment transmits output area information based on a magnification usage condition.
FIG. 14 is a diagram illustrating information output by a processor and an image sensor in an operation of an electronic device switching a camera for capturing an image according to one embodiment.
FIG. 15 is a diagram illustrating an example of an electronic device according to one embodiment of the present invention switching at least one of a magnification, a size, or a position of an image being captured.

FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of an electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a nonvolatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display module (160), the sensor module (176), or the communication module (190)), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), terminal power minimization and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in an electronic device (101), an external electronic device (e.g., an electronic device (104)), or a network system (e.g., a second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components can be connected to each other and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

In one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). In one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result, as is or additionally processed, as at least a part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an IoT (Internet of Things) device. The server (108) may be an intelligent server using machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

The electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. The electronic devices according to embodiments of this document are not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but rather to encompass various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component that is configured integrally or a minimum unit of the component that performs one or more functions, or a part thereof. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, one or more of the components or operations of the above-described components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of an image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as, for example, an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of an image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to a shutter, or when a plurality of images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130), or as a separate memory that operates independently therefrom.

The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, or read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., the image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or may be provided to an external component of the camera module (180) (e.g., the memory (130), the display module (160), the electronic device (102), the electronic device (104), or the server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a part of the image signal processor (260) or as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

FIG. 2 illustrates an example of a configuration of a camera module (180) included in another electronic device (101) according to one embodiment, and the camera module (180) may include only some of the components illustrated in FIG. 2 or may further include other components. For example, the flash (220) of FIG. 2 may not be included in the camera module (180) but may be separately arranged around a flicker sensor.

The image sensor (230) included in the electronic device (101) may operate based on different operation modes depending on the operating conditions. For example, when the electronic device (101) captures an image at a first magnification through the camera (180), the electronic device (101) may operate in a binning mode that bins a plurality of pixels to output an image frame. When the electronic device (101) captures an image at a second magnification higher than the first magnification through the camera (180), the image sensor (230) may operate in an operation mode for outputting an image frame generated based on a higher resolution. Here, when the image sensor (230) operates to output an image frame at a higher resolution, the pattern of color channels represented by the pixels included in the image frame may be changed. The image signal processor (260) may perform a demosaic operation that converts an image frame output from the image sensor into an image (e.g., an RGB image) having color values expressed similarly to those recognized by the human eye. The image sensor (230) can perform a remosaic operation that converts an image frame in which a pattern of color channels has been changed into a pattern (e.g., a Bayer pattern) on which an image signal processor (260) can perform a demosaic operation. However, the remosaic operation is not limited to being performed in the image signal processor (260). For example, the remosaic operation can be performed in the sensor or the processor (120) (e.g., an application processor (AP)). An operation mode in which the image sensor (230) performs remosaic and outputs an image frame can be referred to as a remosaic mode.

When the operation mode of the image sensor (230) is changed from the binning mode to the re-mosaic mode, the size of the image frame may increase. For example, in the re-mosaic mode, image data captured at 17.3 Mp may be acquired by considering a margin so that the area within the field of view becomes 12 Mp (megapixel). However, the value of 17.3 Mp presented in this document is presented as an example of a higher resolution than the resolution of an image actually provided by considering a margin of about 20%, and the resolution value may be changed according to the margin value. However, the image signal processor (260) may not be able to process an image frame whose size has been changed. Therefore, the image sensor (230) may down-scale the size of the image frame to a size that the image signal processor (260) can process. For example, the image sensor (230) may down-scale the size of the image frame from 17.3 Mp to 12 Mp. Here, the area within the field of view can be configured with a resolution of 8 Mp. The image signal processor (260) can perform image processing on an image frame whose size is down-scaled, and then perform up-scaling to restore the image to the original size of the area within the field of view. For example, the area within the field of view configured with a resolution of 8 Mp can be up-scaled to 12 Mp.

As described above, in the mode change operation of the image sensor, when down-scaling and up-scaling are performed, image deterioration may occur. According to one embodiment, an image sensor capable of preventing deterioration occurring when the operation mode of the image sensor is changed, an electronic device including the image sensor, and an operating method thereof may be provided.

The electronic device (101) may include a plurality of cameras (e.g., a first camera and a second camera). The electronic device (101) may switch the camera used to capture an image to a camera with a different angle of view depending on the operating conditions. For example, when the electronic device (101) captures an image at a first magnification, the image may be captured through the first camera, and when the electronic device (101) captures an image at a second magnification, the image may be captured through the second camera. The first camera and the second camera cannot be physically positioned at the same location. Therefore, when the electronic device (101) switches the camera for capturing an image, a sense of incongruity may occur between an image captured through the camera used before the change and an image captured through the camera used after the change. For example, when the electronic device (101) is displaying a live preview image captured through the first camera and then switches to the second camera to provide a live preview image, a phenomenon such as the live preview image suddenly moving may occur due to parallax between the two cameras.

In one embodiment, an electronic device may include multiple (e.g., three to six) multi-cameras for high-quality image and zoom system. Synchronization and switching of each camera in various operation modes of the camera are important techniques for natural scene display and improved camera operation. An electronic device and an operation method thereof according to one embodiment may acquire and utilize information (e.g., calibration, address information, or optical image stabilization information) about each of the multi-cameras in real time or periodically, thereby switching to another camera or switching the operation mode of the camera during zoom operation of the camera. An electronic device or an operation method thereof according to one embodiment may transmit a signal including output area information from a processor (e.g., an application processor) to an image sensor using an I2C (inter-integrated circuit) (or I3C (improved inter-integrated circuit)) communication method, and at the same time, the image sensor may transmit data to the processor through a MIPI (mobile industry processor interface), thereby providing a multi-camera system and a smooth zoom scenario. Image signal processing may be performed, for example, by at least one of an image sensor (e.g., an image sensor (230) of FIG. 2) or an image signal processor (e.g., an image signal processor (260) of FIG. 2).

According to one embodiment, an image sensor capable of simply compensating for parallax that occurs when switching a camera for capturing an image, an electronic device including the image sensor, and an operating method thereof can be provided.

The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description of this disclosure.

FIG. 3 is a drawing for explaining an example of an operation based on multiple operation modes of an image sensor (300) included in an electronic device (101) according to one embodiment.

An electronic device (101) according to one embodiment may include at least one processor (120), a memory (130), and a camera module (180). The camera module (180) may include an image sensor (300) that receives light. The processor (120) according to one embodiment may control the image sensor (300) to output an image frame based on one or more operation modes. The camera module (180) may transmit an image frame output through the image sensor (300) to the processor (120). For example, the processor (120) may transmit a control signal to the camera module (180) through I2C (or I3C, SPI (serial peripheral interface)) communication, and the camera module (180) may transmit an image frame to the processor (120) through an interface (for example, MIPI (mobile industry processor interface)).

In one embodiment, the processor (120) can control the operation mode of the image sensor (300) based on a magnification set for capturing an image. For example, the processor (120) can control the image sensor (300) to output an image frame to the processor (120) based on a first pattern (311) while capturing an image based on a first magnification within a reference range (e.g., x1 to x1.9). For example, the first pattern (311) can be a pattern in which the image sensor (300) bins and outputs values of a plurality of pixels. The processor (230) can control the image sensor (300) to output an image frame based on a second pattern (313) while capturing an image based on a second magnification exceeding the reference range (e.g., x2 to x3.9). For example, the second pattern (313) may be configured through a remosaic operation on raw data read-out from the image sensor (300). An image frame according to the second pattern (313) may include an image having a second resolution higher than a first resolution of an image output according to the first pattern (311). The image sensor (300) may stream the image frame to the processor (120) based on the pattern according to the operation mode. The image sensor (300) may output an image frame based on the third pattern (315) to output an image frame including an image having a third resolution higher than the second resolution, but is not limited thereto.

The first pattern (311), the second pattern (313), and the third pattern (315) may be examples of a case where the image sensor (300) has a structure in which four micro lenses are arranged to correspond to one color channel, and four light-receiving elements (e.g., photo diodes (PDs)) are arranged to correspond to each micro lens. However, the patterns illustrated in FIG. 3 are merely for explaining an example in which the image sensor (300) outputs an image frame according to an operation mode, and is not limited thereto. The image sensor (300) according to one embodiment may have another structure that can operate based on a plurality of operation modes. For example, the image sensor (300) may have a structure in which nine micro lenses are arranged to correspond to one color channel, and one light-receiving element is arranged to correspond to each micro lens. In this case, the image sensor (300) may output an image frame based on the fourth pattern (321) while capturing an image based on a first magnification within a reference range. The image sensor (300) can output an image frame based on the fifth pattern (325) while capturing an image based on a second magnification exceeding the reference range.

The processor (120) according to one embodiment may transmit a signal including output area information to the camera module (180) (or, the image sensor (300)). The processor (120) may monitor and transmit the output area information periodically, continuously, or repeatedly. For example, the processor (120) may transmit the signal including the output area information through communication (e.g., I2C communication) for transmitting a signal between integrated circuits (ICs). The output area information may include information indicating a range of data to be output by the image sensor (300) when the operation mode of the image sensor (300) is changed or a camera used to acquire an image frame is changed to a camera including the image sensor (300). The range of data to be output by the image sensor (300) may mean pixels to be included in the image frame among pixels from which the image sensor (300) can read pixel values. For example, the output area information may include a coordinate value (hereinafter, “start coordinate”) for a start point of a pixel to be included in an image frame, among an area from which an image sensor (300) whose operation mode has been changed or an image sensor of a changed camera can read out a pixel value. The output area information may further include a coordinate value (hereinafter, “end coordinate”) for an end point of a pixel to be included in an image frame, among an area from which the image sensor (300) can read out a pixel value. The processor (120) may also determine the output area information based on an offset that occurs due to a change in magnification.

In one embodiment, the output region information may further include additional information for determining a region to be included in the image frame. For example, the output region information may include at least one of control information of an image stabilizer (e.g., the image stabilizer (240) of FIG. 2), a process compensation value, or a zoom operation position. The control information of the image stabilizer may include information associated with a parameter value related to an image stabilization operation for correcting shaking of the image. The process compensation value may include a preset offset value related to the image sensor (300). The zoom operation position may include information determined based on whether a user inputs a zoom operation for a certain location or region within the image.

An image sensor (300) according to one embodiment may output data (e.g., an image frame) including pixel values corresponding to output area information. The image frame output from the image sensor (300) may be transmitted to the processor (120) through an interface (e.g., a mobile industry processor interface (MIPI)) connected to the processor (120). In one embodiment, the image sensor (300) may be configured to select data based on output area information among pixel values read from pixels included in the image sensor (300) and output an image frame. In one embodiment, the image sensor (300) may be configured to read only pixel values corresponding to output area information among pixels included in the image sensor (300) and output an image frame. The image sensor (300) may be configured to output an image frame including an image corresponding to an area determined through output area information based on a change in an operation mode of the image sensor (300). For example, the image sensor (300) may be configured to output an image frame including an image corresponding to the changed magnification to the processor (120) based on output area information received from the processor (120) based on a change in magnification to a magnification exceeding a reference range.

An electronic device (101) according to one embodiment may further include a touchscreen display (e.g., a display module (160) of FIG. 1). The processor (230) may display a screen including a preview image generated based on an image frame output from an image sensor (300) through the touchscreen display. The processor (230) may change a magnification associated with capturing an image based on a touch input received through the touchscreen display. The processor (230) may control the camera module (180) so that an operation mode of the image sensor (300) is changed based on the changed magnification.

FIG. 4 is a drawing for explaining an example of an electronic device (101) including a plurality of cameras (401, 402) according to one embodiment.

In one embodiment, the electronic device (101) may include a processor (120), a memory (130), and a camera module (180). The camera module (180) may include a plurality of cameras (401, 402). The block diagram illustrated in FIG. 4 illustrates a first camera (401) and a second camera (402), but the camera module (180) may include a greater number of cameras. The plurality of cameras (401, 402) may be arranged to photograph in substantially the same or parallel directions. Referring to FIG. 4, the first camera (401-1) and the second camera (402-2) included in the camera module (180-1) of the electronic device (101) may be arranged to photograph an image through one side (410) of the housing. However, the first camera (401-1) and the second camera (402-1) illustrated in FIG. 4 illustrate an example of arranging the first camera (401) and the second camera (402), and the shape in which the first camera (401) and the second camera (402) are arranged is not limited to that illustrated in FIG. 4. In one embodiment, the first camera (401) may be configured to support a first angle of view. The second camera (402) may be configured to support a second angle of view different from the first angle of view. For example, the first camera (401) may be configured as a wide-angle camera, and the second camera (402) may be configured as a telephoto camera configured to support a narrower angle of view than the first camera (401). The processor (120) according to one embodiment may obtain an image frame from at least one of the first camera (401) or the second camera (402). The processor (120) can transmit a control signal to at least one of the first camera (401) or the second camera (402), and the camera module (180) can transmit an image frame through an interface connected to the processor (120).

In one embodiment, the processor (120) can acquire image frames from at least one of the first camera (401) or the second camera (402) based on a magnification set for capturing an image. For example, the processor (120) can receive image frames from the first camera (401) when capturing an image based on the second magnification, and can receive image frames from the second camera (402) when capturing an image based on a third magnification exceeding a reference range. If the second camera (402) is deactivated while receiving image frames from the first camera (401), the processor (120) can activate the second camera (402) based on a change from the second magnification to the third magnification. The first camera (401) or the second camera (402) can stream image frames to the processor (120). At least one of the first camera (401) or the second camera (402) may be configured to support multiple operation modes as illustrated in FIG. 3. However, the present invention is not limited thereto. The first camera (401) or the second camera (402) may be configured to support only one operation mode. In one embodiment, at least one of the first camera (401) or the second camera (402) may perform an operation corresponding to a magnification based on the flowchart (600) of FIG. 6, respectively.

The processor (120) according to one embodiment may transmit a signal including output area information to the camera module (180) (or, the first camera (401) or the second camera (402)). The processor (120) may monitor and transmit the output area information periodically, continuously, or repeatedly. The output area information may include information indicating a range of data to be output by a changed camera when a camera that acquires an image is changed. The range of data to be output by a camera (e.g., the first camera (401) or the second camera (402)) may mean pixels to be included in an image frame among pixels from which pixel values can be read out. For example, the output area information may include a start coordinate value of a pixel to be included in an image frame among areas from which the changed camera can read out pixel values. The output area information may further include an end coordinate value of a pixel to be included in an image frame among areas from which the image sensor (300) can read out pixel values. The processor may also determine the output area information based on an offset that occurs due to a change in magnification.

In one embodiment, the output region information may further include additional information for determining a region to be included in the image frame. For example, the output region information may include at least one of control information of the image stabilizer, a process compensation value, or a zoom operation position. The control information of the image stabilizer may include information associated with a parameter value related to an image stabilization operation for correcting shaking of the image. The process compensation value may include a preset offset value related to the first camera (401) and the second camera (402). The zoom operation position may include information determined based on whether a user inputs a zoom operation for a certain location or area within the image.

According to one embodiment, at least one of the first camera (401) or the second camera (402) may output an image frame including pixel values corresponding to output area information. The image frame output through at least one of the first camera (401) or the second camera (402) may be transmitted to the processor (120) through an interface connected to the processor (120). In one embodiment, at least one of the first camera (401) or the second camera (402) may be configured to select data based on the output area information among the read-out pixel values and output the image frame. In one embodiment, at least one of the first camera (401) or the second camera (402) may be configured to output the image frame by reading only the pixel values corresponding to the output area information among the pixels of the image sensor included in each camera. Based on a change in the camera used to capture an image, the changed camera may be configured to output an image frame including an image corresponding to an area determined through the output area information. For example, in a state where a preview image is displayed on a touchscreen display (e.g., display module (160) of FIG. 1) based on an image captured through a first camera (401), in response to a user input for changing the magnification, the second camera (402) can output an image frame corresponding to the changed magnification to the processor (120) based on output area information received from the processor (120).

In one embodiment, the output region information may further include additional information for determining regions to be included in the image frame. For example, the output region information may further include information related to control information of the image stabilizer. The control information of the image stabilizer may include a value for correcting a position of an image being captured to compensate for shaking of the device. For example, the control information of the image stabilizer may include a control value for moving a position of an optical image stabilizer, and/or a position value of the image stabilizer detected via a Hall sensor included in the image stabilizer.

In one embodiment, the first camera (401) may be configured to operate based on any one of a plurality of operating modes as illustrated in FIG. 3. However, this is not limited thereto. The first camera (401) may also be configured to operate based on a single operating mode.

FIG. 5 is a flowchart (500) illustrating a process by which an electronic device (101) acquires an image frame according to one embodiment. In this document, the operation of the electronic device (101) may be understood as being performed by a processor (e.g., processor (120) of FIG. 1, FIG. 3, or FIG. 4) executing instructions stored in a memory (e.g., memory (130) of FIG. 1, FIG. 3, or FIG. 4) to perform a calculation or controlling a component of the electronic device (101).

According to one embodiment, in operation 510, the electronic device (101) may obtain a first image frame based on a first magnification. For example, in operation 501, the electronic device (101) may obtain an image frame including data output by an image sensor (e.g., an image sensor (300) of FIG. 3) included in a first camera (e.g., a first camera (401) of FIG. 4) based on a first pattern (e.g., a first pattern (311) and a fourth pattern (321) of FIG. 3)) to obtain an image corresponding to a x1.5 magnification. For example, in operation 501, the electronic device (101) may acquire an image frame including data output by an image sensor (e.g., an image sensor (300) of FIG. 3) included in a first camera (e.g., a first camera (401) of FIG. 4) based on a second pattern (e.g., a second pattern (313) of FIG. 3) to acquire an image corresponding to a magnification of x2.5.

According to one embodiment, in operation 520, the processor (120) of the electronic device (101) may transmit a signal including output area information to a camera module (e.g., the camera module (180) of FIG. 1, 3 or 4). For example, the processor (120) may transmit the signal including the output area information based on a communication method between integrated circuits (e.g., I2C communication). Although the flowchart (500) of FIG. 5 illustrates that operation 520 is performed after operation 510 for convenience of explanation, the present invention is not limited to the operation order of the flowchart included in the present disclosure. For example, while a first image frame acquired based on a first magnification is streamed, the processor (120) may monitor the output area information by repeatedly performing operation 520.

According to one embodiment, in operation 530, the electronic device (101) may obtain a second image frame corresponding to a second magnification based on the output area information. Here, the second magnification may be a magnification exceeding a reference range including the first magnification of operation 510. For example, the electronic device (101) may change a pattern in which an image sensor (e.g., the image sensor (300) of FIG. 3) outputs data from a first pattern (e.g., the first pattern (311) of FIG. 3) to a second pattern (e.g., the second pattern (313) of FIG. 3) based on the magnification for capturing an image being changed from a state in which the first image frame is obtained based on a x1.5 magnification based on operation 510 to a x2.0 magnification exceeding a x1.9 magnification. The electronic device (101) can obtain a second image frame including pixel values corresponding to output area information from data output by an image sensor (e.g., the image sensor (300) of FIG. 3) to a second pattern (e.g., the second pattern (313) of FIG. 3).

As another example, the electronic device (101) may switch the camera used to acquire an image from a first camera (e.g., the first camera (401) of FIG. 4) to a second camera (e.g., the second camera (402) of FIG. 4) based on the fact that the magnification for capturing an image is changed from x2.9 to x3.0. The electronic device (101) may acquire a second image frame including pixel values corresponding to output area information from data output by an image sensor of the second camera (e.g., the second camera (402) of FIG. 4). However, in one embodiment, these are only examples for explaining a process in which the electronic device (101) acquires an image frame according to a change in magnification, and are not limited thereto.

FIG. 6 is a flowchart (600) illustrating a process in which an electronic device (101) according to one embodiment changes a magnification related to capturing an image based on a single camera (e.g., camera module (180) of FIG. 3).

According to one embodiment, in operation 610, the electronic device (101) may acquire a first image frame based on a first magnification through a camera (e.g., a camera module (180) of FIG. 3). The first magnification may fall within a reference range in which an image sensor (e.g., an image sensor (300) of FIG. 3) included in the camera (e.g., a camera module (180) of FIG. 3) operates to output pixel values for an image captured based on a first pattern (e.g., a first pattern (311) of FIG. 3). The reference range may mean a range in which a magnification is changed while maintaining a pattern of pixels output by the image sensor (e.g., an image sensor (300) of FIG. 3). For example, in operation 610, a camera (e.g., a camera module (180) of FIGS. 1 to 4) of an electronic device (101) may be configured to output pixel values based on a first pattern (e.g., a first pattern (311) of FIG. 3) when a magnification set for capturing an image is within a range of x1 to x1.9. For example, in operation 610, the image sensor (e.g., the image sensor (300) of FIG. 3) may output pixel values by binning them. The electronic device (101) may display a screen including a preview image based on the first image frame acquired in operation 610 through a touchscreen display (e.g., a display module (160) of FIG. 1). The electronic device (101) may store an image or video in a memory (e.g., memory (130, 250) of FIGS. 1 to 4) based on the first image frame. The electronic device (101) may also transmit the image to an external device based on the first image frame.

According to one embodiment, in operation 620, the processor (120) of the electronic device (101) may transmit a signal including output area information to a camera (e.g., the camera module (180) of FIGS. 1 to 4) (or an image sensor of the camera (e.g., the image sensor (300) of FIG. 3)). For example, the electronic device (101) may check a setting value for a zoom operation for an application while an application using a camera (e.g., a camera application, a video call application) is running. When a pattern of pixel values output by an image sensor (e.g., the image sensor (300) of FIG. 3) changes as a set magnification goes out of a reference range, the electronic device (101) may determine output area information for determining pixel values to be included in an image frame output to the processor (120) among pixel values according to the changed pattern. For example, the output area information may include coordinate values. The coordinate values may include a location of an area in an image to which a zoom operation is to be enlarged, a control value of an image stabilizer, or a preset offset value. can be calculated based on at least one of the following.

In operation 630, the electronic device (101) can determine whether a set magnification for capturing an image exceeds a reference range. For example, the electronic device (101) can receive a user input for commanding a zoom operation through a touchscreen display (e.g., the display module (160) of FIG. 1). If the set magnification is within the reference range, the electronic device (101) can re-perform operation 610 of obtaining a first image frame based on a pattern corresponding to the first magnification. The electronic device (101) can update a preview image output through the touchscreen display (e.g., the display module (160) of FIG. 1) based on the re-performed first image frame. The electronic device (101) can also add the updated first image frame to a stored video or transmit it to an external device.

According to one embodiment, based on determining that a magnification set for capturing an image exceeds a reference range, in operation 640, the electronic device (101) may obtain a second image frame including an image corresponding to the second magnification based on the output area information transmitted in operation 620. For example, the image sensor (e.g., the image sensor (300) of FIG. 3) of the electronic device (101) may output an image frame based on the second pattern (e.g., the second pattern (313) of FIG. 3) based on the magnification being changed to x2.0. The image that the image sensor (e.g., the image sensor (300) of FIG. 3) outputs based on the second pattern (e.g., the second pattern (313) of FIG. 3) may have a higher resolution than the image that it outputs based on the first pattern (e.g., the first pattern (311) of FIG. 3). In operation 630, the image sensor (e.g., the image sensor (300) of FIG. 3) may crop and output an area corresponding to the output area information, or read out only pixel values corresponding to the output area information. Although not shown in FIG. 6, the processor (120) may also monitor the output area information while the second image frame generated based on the second pattern (e.g., the second pattern (313) of FIG. 3) is being streamed.

According to one embodiment, at operation 650, the electronic device (101) may determine whether to terminate the camera operation. For example, it may determine whether an event indicating that execution of a camera application has been terminated or a video call has been terminated has occurred. Based on the identification of termination of the camera operation, the electronic device (101) may terminate the camera operation and deactivate or power off the camera module (e.g., the camera module (180) of FIGS. 1 to 4).

According to one embodiment, if the camera operation is not terminated, in operation 660, the electronic device (101) may determine whether a magnification set for capturing an image is within a reference range. Based on the magnification exceeding the reference range, the electronic device (101) may perform operation 640 so that the image sensor (e.g., the image sensor (300) of FIG. 3) continues to output image frames based on the second pattern (e.g., the second pattern (313) of FIG. 3). If the magnification is changed to within the reference range, the electronic device (101) may perform at least one of operation 610 or operation 620.

FIG. 7 is a flowchart (700) illustrating a process in which an electronic device (101) according to one embodiment changes a magnification related to capturing an image based on a first camera (e.g., the first camera (401) of FIG. 4) and a second camera (e.g., the second camera (402) of FIG. 4).

According to one embodiment, the electronic device (101) may determine output area information in operation 710. The output area information may include information indicating an area of pixels whose pixel values will include an image frame among pixels included in an image sensor according to each operation mode for each camera. The processor (120) of the electronic device (101) may determine the output area information and transmit the determined output area information to the camera module (180). The camera module (180) may control each camera (e.g., the first camera (401) or the second camera (402) of FIG. 4) or the image sensor of the camera (e.g., the image sensor (300) of FIG. 3) based on the output area information. Alternatively, the output area information may be transmitted to each camera (e.g., the first camera (401) or the second camera (402) of FIG. 4) or the image sensor of the camera (e.g., the image sensor (300) of FIG. 3) so that the camera operates based on the output area information.

According to one embodiment, in operation 720, the electronic device (101) may determine whether the magnification set for capturing an image is less than or equal to a first threshold. For example, the electronic device (101) may determine whether the magnification set for capturing an image is less than x2.0. If it is determined that the magnification set for capturing an image is less than the first threshold (for example, when the magnification is x1.9), the electronic device (101) may obtain a first image frame through a first camera (e.g., the first camera (401) of FIG. 4) based on the first magnification in operation 730. In operation 730, an image sensor (e.g., the image sensor (300) of FIG. 3) of the first camera (e.g., the first camera (401) of FIG. 4) may output an image frame based on a first pattern (e.g., the first pattern (311) of FIG. 3). For example, an image sensor (e.g., an image sensor (300) of FIG. 3) of a first camera (e.g., a first camera (401) of FIG. 4) may operate based on an operation mode (hereinafter, referred to as a 'binning mode') that bins a plurality of pixels and outputs them. The camera module (180) may transmit an image frame configured based on information indicating the positions of pixels to be included in an image frame among pixels that the first camera (e.g., a first camera (401) of FIG. 4) can output in a binning mode among output area information to the processor (120).

According to one embodiment, if it is determined that the magnification set for capturing an image in operation 720 is greater than or equal to the first threshold, the electronic device (101) may determine whether the magnification set for capturing an image is less than a second threshold in operation 740. If it is determined that the magnification set for capturing an image is less than the second threshold (e.g., when the magnification is x2.9), the electronic device (101) may obtain a second image frame through the first camera (e.g., the first camera (401) of FIG. 4) based on the second magnification in operation 750. In operation 750, the image sensor (e.g., the image sensor (300) of FIG. 3) of the first camera (e.g., the first camera (401) of FIG. 4) may output the image frame based on a second pattern (e.g., the second pattern (313) of FIG. 3). For example, an image sensor (e.g., the image sensor (300) of FIG. 3) of a first camera (e.g., the first camera (401) of FIG. 4) may operate based on an operation mode (hereinafter, referred to as a 're-mosaic mode') that re-mosaics a pattern of pixels and outputs it. The camera module (180) may transmit an image frame configured based on information indicating positions of pixels to be included in an image frame among pixels that the first camera (e.g., the first camera (401) of FIG. 4) can output in the re-mosaic mode) to the processor (120) among the output area information. For example, the image frame may be configured by cropping an area corresponding to the output area information among pixel values read out from the image sensor (e.g., the image sensor (300) of FIG. 3). For example, the image frame may also be configured by the image sensor (e.g., the image sensor (300) of FIG. 3) reading only pixel values of an area corresponding to the output area information.

According to one embodiment, if it is determined that the magnification set for capturing the image in operation 740 is greater than or equal to the second threshold (for example, when the magnification is x3.0), the electronic device may acquire a third image frame through the second camera (for example, the second camera (402) of FIG. 4) based on the third magnification in operation 760. The second camera (for example, the second camera (401) of FIG. 4) may be a camera configured to support a different angle of view from that of the first camera (for example, the first camera (402) of FIG. 4). For example, the second camera (for example, the second camera (402) of FIG. 4) may be a camera (with a longer focal length) that supports a narrower angle of view than that of the first camera (for example, the first camera (401) of FIG. 4). The camera module (180) may transmit an image frame configured based on information indicating the positions of pixels to be included in the image frame among pixels that can be output by the second camera (e.g., the second camera (402) of FIG. 4) among the output area information to the processor (120). For example, the image frame may be configured by cropping an area corresponding to the output area information among pixel values to be read from the image sensor of the second camera (e.g., the second camera (402) of FIG. 4). For example, the image frame may also be configured by the image sensor of the second camera (e.g., the second camera (402) of FIG. 4) reading only pixel values of an area corresponding to the output area information.

In one embodiment, the electronic device (101) may determine whether to terminate the camera operation at operation 770. For example, it may determine whether an event indicating that execution of a camera application has been terminated or a video call has been terminated has occurred. Based on the identification of termination of the camera operation, the electronic device (101) may terminate the camera operation and deactivate or turn off the camera module (e.g., the camera module (180) of FIGS. 1 to 4). If the camera operation is not terminated, the electronic device (101) may perform operation 710 again to update the output area information. Although operation 710 is illustrated in the flowchart (700) for convenience of explanation, the order in which the electronic device (101) performs operation 710 is not limited to the order illustrated in FIG. 7. For example, the processor (120) of the electronic device (101) can continuously, repeatedly, or periodically monitor output area information as a separate routine from the operation of the camera or image sensor, and transmit the determined output area information to the camera module (180) for updating.

FIG. 8 is a diagram conceptually explaining an operation of an electronic device (101) according to one embodiment of the present invention to output an image frame based on shooting area information according to a changing magnification.

In one embodiment, when the magnification for capturing an image is changed, the electronic device (101) can obtain an image corresponding to the changed magnification based on the output area information. Referring to FIG. 8, when the magnification for capturing an image is set to x1.0 times, the electronic device (101) can obtain an image frame including a first image (810) based on the first pattern (811). When the magnification for capturing an image is set to x1.0 times, the electronic device (101) can obtain the first image (810) based on a binning mode. For example, data having the first pattern (811) can be read from each pixel of an image sensor (e.g., the image sensor (300) of FIG. 3) of the electronic device (101). An image sensor (e.g., an image sensor (300) of FIG. 3) can output an image frame including data having a pattern (813) formed by merging multiple pixels through a binning operation (812) on data having a first pattern (811) to a processor (e.g., a processor (120) of FIG. 3).

In one embodiment, the electronic device (101) can change the magnification for capturing an image from x1.0 times to the first magnification. For example, the electronic device (101) can change the set magnification for capturing an image in response to a user input for setting the magnification. In one embodiment, the processor (120) of the electronic device (101) can transmit output area information indicating an area corresponding to an image according to the changed first magnification to a camera or an image sensor (e.g., the image sensor (300) of FIG. 3). Referring to FIG. 8, the processor (120) can transmit output area information including start coordinates (X ₁ , Y ₁ ). The output area information may further include end coordinates (X ₂ , Y ₂ ).

In one embodiment, the electronic device (101) can obtain a second image (820) corresponding to the first magnification based on the output area information. The electronic device (101) can obtain the second image (820) composed of pixel values from the start coordinate (X ₁ , Y ₁ ) to the end coordinate (X ₂ , Y ₂ ) within the image (810). For example, the image sensor (e.g., the image sensor (300) of FIG. 3 ) of the electronic device (101) can read data having a first pattern (821) from each pixel of the image sensor (e.g., the image sensor (300) of FIG. 3 ). An image sensor (e.g., an image sensor (300) of FIG. 3) can output an image frame including data (825) that is cropped at least a portion (824) of data having a pattern (823) in which a plurality of pixels are merged through a binning operation (822) on data having a first pattern (821) to a processor (e.g., a processor (120) of FIG. 3). The image sensor (e.g., an image sensor (300) of FIG. 3) can determine an area to crop data based on at least one of a start coordinate (X ₁ , Y ₁ ) or an end coordinate (X ₂ , Y ₂ ).

In one embodiment, the electronic device (101) may obtain an image frame having a higher resolution than the data (825) based on the data having the first pattern (821). Referring to FIG. 8, the electronic device (101) may obtain data having a configured pattern (827) by performing a re-mosaic operation (826) on the data having the first pattern (821). Here, the re-mosaic operation (826) may include, for example, a software re-mosaic operation that performs re-mosaic on data output from an image sensor (e.g., the image sensor (300) of FIG. 3). The electronic device (101) may output an image frame including data (829) that is a cropped portion (828) of the data having the pattern (827) to a processor (e.g., the processor (120) of FIG. 3). An image sensor (e.g., image sensor (300) of FIG. 3) can determine an area to crop data from based on at least one of a start coordinate (X ₁ , Y ₁ ) or an end coordinate (X ₂ , Y ₂ ). The data (829) can include information about an image having a higher resolution than the data (825).

In one embodiment, the electronic device (101) can change the magnification for capturing an image from a first magnification to a second magnification that is out of a reference range. The electronic device (101) can obtain a second image (830) corresponding to the second magnification based on the output area information. Based on the magnification for capturing an image being changed to the second magnification, the image sensor (e.g., the image sensor (300) of FIG. 3) can switch to an operation mode for outputting data about the image based on the second pattern (831) to obtain an image with a higher resolution. For example, the image sensor (e.g., the image sensor (300) of FIG. 3) of the electronic device (101) can read data having the second pattern (831) from each pixel of the image sensor (e.g., the image sensor (300) of FIG. 3). Data having the second pattern (831) may be obtained by performing a re-mosaic operation (832) on data read from an image sensor (e.g., the image sensor (300) of FIG. 3). For example, the re-mosaic operation (832) may include a hardware re-mosaic operation that performs re-mosaic on the image sensor (832) side and outputs it. The image sensor (e.g., the image sensor (300) of FIG. 3) may perform a binning operation (834) on data obtained by performing the re-mosaic operation on data having the second pattern (831) and output an image frame including data having the Bayer pattern (833) to a processor (e.g., the processor (120) of FIG. 3). An image sensor (e.g., image sensor (300) of FIG. 3) can obtain data having a Bayer pattern (833) through a remosaic operation based on at least one of the start coordinates (X ₃ , Y ₃ ) or the end coordinates (X ₄ , Y ₄ ).

In one embodiment, when an image sensor (e.g., the image sensor (300) of FIG. 3) outputs data composed of pixel values read based on the second pattern (831), the size of the data may increase compared to data output based on the first pattern (821). However, if the size of the data output from the image sensor (e.g., the image sensor (300) of FIG. 3) changes, the image signal processor (e.g., the image signal processor (260) of FIG. 2) may not be able to perform image processing or the complexity for configuring the image signal processor (e.g., the image signal processor (260) of FIG. 2) may increase. Therefore, the image sensor (e.g., the image sensor (300) of FIG. 3) may output an image frame including pixel values corresponding to an area within the start coordinate (X ₃ , Y ₃ ) to the end coordinate (X ₄ , Y ₄ ) included in the output area information. Since the size of the output image frame can be maintained, the operation mode of the image sensor (e.g., the image sensor (300) of FIG. 3) can be seamlessly switched without down-scaling and up-scaling operations for the image.

FIG. 9 is a drawing for explaining an operation of changing a camera based on a changing magnification of an electronic device (101) according to one embodiment.

In one embodiment, the electronic device (101) may include a plurality of cameras (401, 402). For example, the electronic device (101) may include a first camera (401) including a wide-angle camera and a second camera (402) including a telephoto camera supporting a narrower angle of view than the first camera (401). The electronic device (101) may select a camera to be used for capturing an image from among the plurality of cameras (401, 402) based on a magnification set for capturing an image.

In one embodiment, the electronic device (101) can obtain a second image (910) through the first camera (401) when the magnification set for capturing an image is a second magnification included in a first range for capturing an image based on the first camera (401). For example, the first range can indicate a magnification of x2.9 or less. When the magnification set for capturing an image is x2.9, the electronic device (101) can obtain a second image (910) at a magnification of x2.9 through the first camera (401).

In one embodiment, the electronic device (101) can obtain a third image (920) through the second camera (402) when the magnification set for capturing an image is a third magnification included in the second range for capturing an image based on the second camera (402). For example, the second range can indicate a magnification of x3.0 or greater. When the magnification set for capturing an image is changed from the first range to the second range, the electronic device (101) can obtain the third image (920) through the second camera (402). When the second camera (402) is turned off or deactivated, the electronic device (101) can turn on or activate the second camera (402) based on the change in the magnification. For example, based on the magnification set for capturing an image being changed to x3.0, the electronic device (101) can acquire a third image (920) through the second camera (402). The first camera (101) that is not being used can be turned off or deactivated.

FIG. 10 is a drawing illustrating an example of a screen displayed by an electronic device (101) in an operation of changing a magnification according to one embodiment.

In one embodiment, the electronic device (101) can display a screen including an image (1010) acquired based on an image frame output based on a first pattern (1015) by binning pixels included in the image sensor when a magnification set for capturing an image falls within a first reference range (e.g., x1 to x1.9).

In one embodiment, when a magnification set for capturing an image is changed and falls within a second reference range (e.g., x2 to x2.9), data output from an image sensor may be remosaiced and output. The processor (120) of the electronic device (101) may transmit output area information to the image sensor so that an image displayed in an operation in which an operation mode of the image sensor is changed may not move or may move to a reduced degree. The image sensor may output an image frame including pixel values corresponding to the output area information among pixels output by remosaicing. For example, the electronic device (101) may display a screen including an image (1020) acquired based on an image frame output based on a second pattern (1025) configured by remosaicing data output from the image sensor. Alternatively, for example, the electronic device (101) may acquire the image (1020) based on image data having a second pattern (1025) configured by the image sensor performing remosaicing.

In one embodiment, when a magnification set for capturing an image is changed and falls within a third reference range (e.g., x3.0 or more), the electronic device (101) may change the camera used for capturing the image (e.g., change to a camera supporting a narrower angle of view or a camera with a longer focal length). The processor (120) of the electronic device (101) may transmit output area information to the changed camera so that the displayed image does not move or moves to a reduced degree in the operation of switching the camera. The changed camera may output an image frame including pixel values corresponding to the output area information among pixels that can be read. The electronic device (101) may display a screen including an image (1030) acquired based on the image frame output from the image sensor.

FIG. 11 is a diagram illustrating an example of an operation of an electronic device (101) according to one embodiment of the present invention to enlarge a preview image (1110) based on a user input.

In one embodiment, the electronic device (101) may display a screen including a preview image (1110) based on image frames streamed from a camera module (180) through a touchscreen display (e.g., the display module (160) of FIG. 1). The electronic device (101) may receive a magnification change input for changing a magnification of an image captured with respect to the preview image (1110). For example, the electronic device (101) may receive a pinch-out input through the touchscreen display (e.g., the display module (160) of FIG. 1). The pinch-out input may be referred to as an input including a first input (1101) in which two or more touch points are identified, followed by a first input (1102) in which the positions of the two or more touch points move apart. A pinch-in input may be referred to as an input (not shown) in which, conversely, a pinch-out input refers to an input in which the positions of two or more touch points move closer following a first input (1101) in which two or more touch points are identified. However, the magnification change input is not limited to a pinch-out input or a pinch-in input. For example, an input for changing the magnification of an image may consist of a touch input for selecting an icon indicating a specified magnification or an input for dragging a bar indicating a magnification.

In one embodiment, the electronic device (101) can determine output area information based on a location where a magnification change input is input. For example, referring to FIG. 11, when a pinch-out input starting from a first point (1111) and a second point (1112) is received, the electronic device (101) can obtain a first magnified image (1131) that is zoomed in to include a first area (1121). When the operation mode of the image sensor or the camera for capturing the image is switched according to the magnification changed as a result of performing the zoom-in operation, the processor of the electronic device (101) can transmit output area information indicating an area including the first area (1121) to the image sensor. The image sensor can output an image frame including the first magnified image (1131) based on the output area information. The electronic device (101) can display a screen including the first magnified image (1131) through a touchscreen display.

In one embodiment, when the electronic device (101) receives a pinch-out input starting from the third point (1113) and the fourth point (1114), the electronic device (101) can obtain a second enlarged image (1132) zoomed in to include the second area (1122). When the operation mode of the image sensor or the camera for capturing the image is switched according to the changed magnification as a result of performing the zoom-in operation, the processor of the electronic device (101) can transmit output area information indicating an area including the second area (1122) to the image sensor. The image sensor can output an image frame including the second enlarged image (1132) based on the output area information. The electronic device (101) can display a screen including the second enlarged image (1132) through a touchscreen display.

Since the processor (120) transmits output area information to the image sensor, and the image sensor provides an image with a changed magnification based on the output area information, when the magnification of the image is changed, the streaming image is not enlarged or reduced only based on a fixed point (e.g., the center of the image), but an image enlarged or reduced based on a point desired by the user can be provided.

FIG. 12 is a diagram illustrating information output by a processor (120) and an image sensor (e.g., the image sensor (300) of FIG. 3) in an operation in which the operation mode of the image sensor (e.g., the image sensor (300) of FIG. 3) of an electronic device (101) is changed according to one embodiment.

In FIG. 12, a signal (1201) represents a signal transmitted from a processor (120) to an image sensor (e.g., the image sensor (300) of FIG. 3). An interface (1202) represents an interface for outputting an image frame from the image sensor (e.g., the image sensor (300) of FIG. 3). For example, the electronic device (101) may display a preview image that is periodically updated based on a stream of image frames output from the image sensor (e.g., the image sensor (300) of FIG. 3) through a display (e.g., the display module (160) of FIG. 1). FIG. 12 illustrates an example of changing from 2.0x to 2.1x. The data output through the interface (1202) may include data obtained as a result of performing binning or remosaic on information readout from photodetectors of the image sensor (e.g., the image sensor (300) of FIG. 3).

In one embodiment, the processor (120) may determine at least one of the start coordinates or the end coordinates of pixels to be included in the image frame from among pixels that can be output by an image sensor operating in a changed mode of an image sensor (e.g., the image sensor (300) of FIG. 3) that constitutes an image frame. For example, referring to FIG. 12, the processor (120) may determine at least one of the start coordinates or the end coordinates of pixels to be included in the image frame at a magnification of 2.1x. The processor (120) may transmit a signal including output area information (1200) including at least one of the start coordinates or the end coordinates of the pixels to the image sensor (e.g., the image sensor (300) of FIG. 3). The output area information (1200) may further include additional information for determining an area to be included in the image frame. For example, the additional information may include at least one of control information of an image stabilizer included in a camera, a preset offset value, or a reference position for a magnification change. Alternatively, the processor (120) may further consider additional information to determine at least one of the start coordinate or the end coordinate.

Referring to FIG. 12, in an operation where the magnification is changed from 2.0x to 2.1x, if the processor (120) transmits the output area information (1200) at the time when the Nth frame (1210) is output, the frame output with the changed magnification based on the output area information (1200) may be the N+2th frame (1212). However, this is not limited thereto. For example, it may be configured to be output with the changed magnification in the N+1th frame (1211).

FIG. 13 is a diagram illustrating an example in which an electronic device (101) according to one embodiment transmits output area information based on a magnification usage condition.

In FIG. 13, a signal (1301) represents a signal transmitted from a processor (120) to an image sensor (e.g., the image sensor (300) of FIG. 3) or each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4). An interface (1302) represents an interface that outputs an image frame from an image sensor (e.g., the image sensor (300) of FIG. 3) or each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4). For example, the electronic device (101) may display a preview image that is periodically updated based on a stream of image frames output from an image sensor (e.g., the image sensor (300) of FIG. 3) through a display (e.g., the display module (160) of FIG. 1). FIG. 13 illustrates an example in which the magnification is changed from 2.0x to 2.1x. The data output through the interface (1302) may include data obtained as a result of performing binning or remosaic on information read from light-receiving elements of the image sensor (e.g., the image sensor (300) of FIG. 3).

In one embodiment, instead of transmitting output area information every time the magnification is changed, the electronic device (101) may transmit a signal including magnification usage condition setting information (1311) that sets usage conditions for each magnification as output area information at the beginning of the camera operation to an image sensor (e.g., the image sensor (300) of FIG. 3) or each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4). The image sensor (e.g., the image sensor (300) of FIG. 3) or each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4) may set operation conditions for each magnification based on the magnification usage condition setting information (1311).

In one embodiment, when a magnification set for capturing an image is changed, the electronic device (101) may transmit a signal including information (1312) about the changed magnification setting. An image sensor (e.g., image sensor (300) of FIG. 3) or each camera (e.g., first camera (401) and second camera (402) of FIG. 4) may determine an area to be included in an image frame based on the changed magnification and an operating condition corresponding to the changed magnification. Referring to FIG. 13, when a signal including information (1312) on magnification setting is transmitted at the time when the Nth frame (1320) is output, an image sensor (e.g., the image sensor (300) of FIG. 3) or each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4) can output an (N+2)th frame (1322) including an image corresponding to a changed magnification (e.g., 2.1x magnification) based on an operating condition (e.g., at least one of the start coordinate or the end coordinate) according to the changed magnification.

In one embodiment, the electronic device (101) may further transmit a signal including additional information about the information (1312) regarding the magnification setting. For example, the additional information may include at least one of control information of an image stabilizer included in the camera, a preset offset value, or a reference position for a magnification change. An image sensor (e.g., an image sensor (300) of FIG. 3) or each camera (e.g., a first camera (401) and a second camera (402) of FIG. 4) may modify an operating condition according to a changed magnification based on the additional information, and output an N+2-th frame (1322) corresponding to the changed magnification based on the modified operating condition.

FIG. 14 is a diagram illustrating information output by a processor and an image sensor in an operation of switching a camera for capturing an image by an electronic device (101) according to one embodiment.

In FIG. 14, a signal (1401) represents a signal transmitted from a processor (120) to each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4). An interface (1402) represents an interface for outputting image frames from each camera (e.g., the first camera (401) and the second camera (402) of FIG. 4). For example, the electronic device (101) may display a preview image that is periodically updated based on a stream of image frames output from an image sensor (e.g., the image sensor (300) of FIG. 3) through a display (e.g., the display module (160) of FIG. 1). FIG. 14 shows an example of switching from a first camera (e.g., the first camera (401) of FIG. 4) supporting a resolution of 12 M pixels to a second camera (e.g., the second camera (402) of FIG. 4) supporting a resolution of 17.3 M pixels.

In one embodiment, the processor (120) may determine start coordinates and end coordinates for pixels to be included in an image frame to be output, among pixels that an image sensor of a second camera (e.g., the second camera (402) of FIG. 4) can output. For example, referring to FIG. 14, the processor (120) may determine at least one of the start coordinates or the end coordinates for the second camera (e.g., the second camera (402) of FIG. 4) that supports 17.3 Mpixels in a re-mosaic mode. The processor (120) may transmit a signal including output area information (1400) including at least one of the start coordinates or the end coordinates of the pixels to the image sensor (e.g., the image sensor (300) of FIG. 3). The output area information (1400) may further include additional information for determining an area to be included in the image frame. For example, the additional information may include at least one of control information of an image stabilizer included in the camera, a preset offset value, or a reference position for a magnification change. Alternatively, the processor (120) may further consider additional information to determine at least one of the start coordinate or the end coordinate.

Referring to FIG. 14, when a signal including processor (120) output area information (1400) is transmitted at the time when the Nth frame (1411) is output, a frame including an image captured through a camera that has been changed based on the output area information (1400) may be output as the N+2th frame (1413). However, this is not limited thereto. For example, the N+1th frame (1412) may also be output through the changed camera.

In one embodiment, the pattern of image frames output before the mode change (e.g., the Nth frame (1411), the N+1th frame (1412)) and the pattern of image frames output after the mode change (e.g., the N+2th frame (1413)) may be different from each other. For example, the image frames output before the mode change may have a pattern in which four adjacent pixels have the same color, which may be referred to as a tetra pattern. The image frames output after the mode change may have a Bayer pattern based on the result of performing remosaicing. However, this is not limited thereto and is an example described for understanding one embodiment.

FIG. 15 is a drawing illustrating an example of an electronic device (101) switching at least one of a magnification, a size, or a position of an image being captured according to one embodiment.

In one embodiment, the electronic device (101) can provide a high-pixel image of an area desired by a user without interruption even when the operation mode of the image sensor (e.g., the image sensor (300) of FIG. 3) changes by causing the image sensor (e.g., the image sensor (300) of FIG. 3) to output an image frame based on output area information provided from the processor (120).

In one embodiment, the electronic device (101) can obtain a first image (1512) of low magnification (or wide field of view) based on a first pattern (1511) that bins pixels in a binning mode (1510) and outputs them. Based on an event (e.g., change in magnification) that causes a switch from the binning mode (1510) to the re-mosaic mode (1520, 1530), the electronic device (101) can obtain a second image (1522, 1532) that is continuous with the first image (1512) provided in the binning mode (1510) from pixels included in an output area (1523, 1533) within the second pattern (1521, 1531) that is output based on the re-mosaic.

In one embodiment, the electronic device (101) can obtain an image for an arbitrary position within pixels output from an image sensor by obtaining an image based on output area information in the re-mosaic mode (1520, 1530). Referring to FIG. 15, the electronic device (101) can obtain an image frame based on the output area (1524) of the first position in the re-mosaic mode (1520), thereby obtaining an image (1525) corresponding to the output area (1524) of the first position. When the electronic device (101) obtains an image frame based on the output area (1534) of the second position in the re-mosaic mode (1530), it can obtain an image (1535) corresponding to the output area (1534) of the second position.

In one embodiment, the electronic device (101) can obtain an image for an area having an arbitrary size within pixels output from an image sensor by obtaining an image based on output area information in the re-mosaic mode (1520, 1530). Referring to FIG. 15, the electronic device (101) can obtain an image frame based on an area (1526, 1536) having a smaller size than the output area (1523, 1533) in the re-mosaic mode (1520, 1530), thereby obtaining an image (1527, 1537) corresponding to the area (1526, 1536) having a smaller size.

In this document, the embodiment of acquiring an image based on output area information in the case where the operation mode of the image sensor or the camera for capturing the image is changed based on the magnification for capturing the image has been mainly described, but the method and device for acquiring an image based on output area information are applicable in relation to the operation in which the operation mode of the image sensor or the camera for capturing the image is changed. The method and device for acquiring an image based on output area information are not limited to the case in which the magnification is changed. For example, the operation of acquiring an image based on the above-described output area information can be applied even in the case in which the operation mode of the image sensor or the camera for capturing the image is changed depending on other shooting conditions, such as the distance to the subject.

[An electronic device according to one embodiment may include a camera including an image sensor outputting image data, a memory storing instructions, and at least one processor. The at least one processor may be configured to, when executing the instructions, obtain a first image frame through the image sensor based on a first magnification within a reference range. The at least one processor may be configured to transmit a signal including output area information indicating a range of data to be output by the image sensor to the image sensor. The image sensor may be configured to output a second image frame including an image corresponding to the second magnification to the at least one processor based on the output area information, based on a magnification related to capturing the image being changed to a second magnification exceeding the reference range.

In one embodiment, the image sensor may be configured to output the first image frame based on first data read-out based on a first resolution when capturing an image based on a magnification within the reference range. The image sensor may be configured to output the second image frame based on an area determined based on start coordinates included in the output area information within second data read-out based on a second resolution higher than the first resolution when capturing an image based on a magnification exceeding the reference range.

In one embodiment, the camera may further include an image stabilizer for compensating for shaking of an output image. The at least one processor may be configured to determine the output area information based on control information of the image stabilizer.

In one embodiment, the at least one processor may be configured to periodically transmit the output area information to the image sensor.

In one embodiment, the at least one processor may be configured to determine the output area information based on an offset resulting from a change in the operating mode of the image sensor to output image frames having different resolutions.

An electronic device according to one embodiment may further include a touchscreen display that displays a preview image based on an image captured by the camera. The at least one processor may be configured to receive a magnification change input associated with an input area within the preview image displayed on the touchscreen display. The at least one processor may be configured to determine the output area information based on a position at which the magnification change input is input.

In one embodiment, the at least one processor may be configured to control the touchscreen display to display the preview image in an enlarged manner based on the input area, based on the second image frame output from the image sensor based on the output area information.

In one embodiment, the at least one processor may be configured to determine the output area information based on a preset offset value and zoom arrangement for the image sensor.

According to one embodiment, a method of operating an electronic device including a camera including an image sensor may include an operation in which at least one processor of the electronic device acquires a first image frame through the image sensor based on a first magnification within a reference range. The method may include an operation in which the at least one processor transmits a signal including output area information indicating a range of data to be output by the image sensor to the image sensor. The method may include an operation in which the image sensor outputs a second image frame including an image corresponding to the second magnification based on the output area information to the at least one processor based on a change in a magnification related to capturing the image to a second magnification exceeding the reference range.

In one embodiment, the operation of obtaining the first image frame may include an operation of outputting the first image frame based on first data read out based on a first resolution when operating based on a magnification within the reference range. The operation of outputting the second image frame may include an operation of outputting the second image frame based on an area determined based on start coordinates included in the output area information within second data read out based on a second resolution higher than the first resolution when capturing an image based on a magnification exceeding the reference range.

In one embodiment, the act of transmitting a signal including the output area information may include an act of determining the output area information based on control information for an image stabilizer of the camera.

In one embodiment, the operation of transmitting a signal including the output area information may include an operation of periodically transmitting the output area information.

In one embodiment, the act of transmitting a signal including the output area information may include an act of determining the output area information based on an offset resulting from a change from the first magnification to the second magnification.

In one embodiment, the method may further include the operation of displaying a preview image based on the first image frame and the operation of receiving a scale change input associated with an input area within the preview image. The operation of transmitting a signal including the output area information may include the operation of determining the output area information based on a location where the scale change is input.

In one embodiment, the method may further include an operation of displaying the preview image in an enlarged manner relative to the input area based on the second image frame.

An electronic device according to one embodiment may include a first camera supporting a first angle of view, a second camera supporting a second angle of view different from the first angle of view, a memory storing instructions, and at least one processor. The at least one processor may be configured to, when executing the instructions, obtain a first image frame through the first camera based on a first magnification within a reference range related to capturing an image. The at least one processor may be configured to transmit a signal including output area information indicating a range of data to be output by an image sensor included in the second camera to the second camera based on a change in the magnification related to capturing the image to a second magnification out of the reference range. The at least one processor may be configured to obtain a second image frame including an image corresponding to the second magnification from the second camera based on the signal including the output area information.

In one embodiment, the first camera may include an image stabilizer for compensating for shaking of an output image. The at least one processor may be configured to determine the output area information based on control information of the image stabilizer.

In one embodiment, the at least one processor may be configured to determine the output area information based on an offset resulting from a change from the first magnification to the second magnification.

In one embodiment, the electronic device may further include a touchscreen display that displays a preview image based on an image captured through the first camera or the second camera. The at least one processor may be configured to control the touchscreen display to display the preview image based on the first image frame acquired through the first camera. The at least one processor may be configured to receive a magnification change input associated with an input area within the preview image displayed on the touchscreen display. The output area information may be determined based on a position at which the magnification change input is input.

In one embodiment, the at least one processor may be configured to control the touchscreen display to display the preview image enlarged based on the input area based on the second image frame output from the second camera based on the output area information.

A computer-readable recording medium according to one embodiment may store instructions that, when executed, cause an electronic device to perform the method described above or the operation of the electronic device.

An electronic device and an operating method thereof according to various embodiments can prevent deterioration that occurs when the operating mode of an image sensor is changed.

An electronic device and an operating method thereof according to various embodiments can compensate for parallax between cameras without a complex structure when switching cameras.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

In the case of software implementation, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors in an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

These programs (software modules, software) may be stored in a random access memory, a non-volatile memory including a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, they may be stored in a memory composed of a combination of some or all of these. In addition, each configuration memory may be included in multiple numbers.

Additionally, the program may be stored in an attachable storage device that is accessible via a communication network, such as the Internet, an Intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. The storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in the singular or plural form according to the specific embodiments presented. However, the singular or plural expressions are selected to suit the presented situation for the convenience of explanation, and the present disclosure is not limited to the singular or plural components, and even if a component is expressed in the plural form, it may be composed of the singular form, or even if a component is expressed in the singular form, it may be composed of the plural form.

Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or a circuit, and/or a software component executed by a hardware component such as a processor.

A “component”, a “module” may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a “component”, a “module” may be implemented by components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The specific implementations described in this disclosure are only examples and are not intended to limit the scope of the disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, or comprising only b, or comprising only c, or comprising both a and b, or comprising both b and c, or comprising all of a, b, and c.”

Meanwhile, although the detailed description of the present disclosure has described specific embodiments, it is obvious that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be determined not only by the scope of the claims described below, but also by equivalents of the scope of the claims.

Claims (20)

In electronic devices,
A camera including an image sensor that outputs image data;
Memory for storing instructions; and
comprising at least one processor,
At least one of the above processors, when executing the instructions,
Acquire a first image frame through the image sensor based on a first magnification within a reference range,
The image sensor is configured to transmit a signal including output area information indicating a range of data to be output by the image sensor to the image sensor,
An electronic device, wherein the image sensor is configured to output a second image frame including an image corresponding to the second magnification to the at least one processor based on the output area information, based on a change in the magnification related to capturing the image to a second magnification exceeding the reference range. In claim 1,
The above image sensor,
When capturing an image based on a magnification within the above reference range, the first image frame is output based on the first data read-out based on the first resolution,
An electronic device configured to output the second image frame based on an area determined based on start coordinates included in the output area information within second data read based on a second resolution higher than the first resolution when capturing an image based on a magnification exceeding the above reference range. In claim 2,
The above camera further includes an image stabilizer that compensates for shaking of the output image,
An electronic device wherein said at least one processor is configured to determine said output area information based on control information of said image stabilizer. In claim 1,
An electronic device wherein said at least one processor is configured to periodically transmit said output area information to said image sensor. In claim 1,
An electronic device wherein said at least one processor is configured to determine the output area information based on an offset resulting from a change in the operating mode of the image sensor to output an image frame having a different resolution. In claim 1,
Further comprising a touchscreen display that displays a preview image based on an image captured through the camera;
At least one processor of the above,
Receive a scale change input associated with an input area within the preview image displayed on the touch screen display,
An electronic device configured to determine the output area information based on a position at which the above scale change input is input. In claim 6,
At least one processor of the above,
An electronic device configured to control the touchscreen display to display the preview image in an enlarged manner based on the input area, based on the second image frame output from the image sensor based on the output area information. In claim 1,
An electronic device wherein said at least one processor is configured to determine said output area information based on a preset offset value and zoom arrangement for said image sensor. A method of operating an electronic device including a camera including an image sensor,
An operation of at least one processor of the electronic device to acquire a first image frame through the image sensor based on a first magnification within a reference range;
An operation in which at least one processor transmits a signal including output area information indicating a range of data to be output by the image sensor to the image sensor; and
A method comprising: an operation of the image sensor outputting a second image frame including an image corresponding to the second magnification based on the output area information to the at least one processor, based on a change in the magnification related to capturing the image to a second magnification exceeding the reference range. In claim 9,
The operation of obtaining the first image frame includes an operation of outputting the first image frame based on first data read out based on the first resolution when operating based on a magnification within the reference range,
A method wherein the operation of outputting the second image frame includes an operation of outputting the second image frame based on an area determined based on start coordinates included in the output area information within second data read based on a second resolution higher than the first resolution when capturing an image based on a magnification exceeding the reference range. In claim 10,
A method wherein the operation of transmitting a signal including the output area information includes an operation of determining the output area information based on control information for an image stabilizer of the camera. In claim 9,
A method for transmitting a signal including the above output area information, wherein the operation comprises periodically transmitting the above output area information. In claim 9,
A method wherein the operation of transmitting a signal including the output area information includes an operation of determining the output area information based on an offset that occurs due to a change from the first magnification to the second magnification. In claim 11,
An operation of displaying a preview image based on the first image frame; and
Further comprising an action of receiving a scale change input associated with an input area within the above preview image,
The operation of transmitting a signal including the above output area information is:
A method comprising an operation of determining the output area information based on a position where the change in scale is input. In claim 14,
A method further comprising an action of displaying the preview image in an enlarged manner based on the input area based on the second image frame. In electronic devices,
First camera supporting the first angle of view;
A second camera supporting a second angle of view different from the first angle of view;
Memory for storing instructions; and
comprising at least one processor,
At least one of the above processors, when executing the instructions,
Acquire a first image frame through the first camera based on a first magnification within a reference range related to capturing the image,
Based on the fact that the magnification related to the shooting of the above image is changed to a second magnification that is outside the above reference range, a signal including output area information indicating the range of data to be output by the image sensor included in the second camera is transmitted to the second camera,
An electronic device configured to acquire a second image frame including an image corresponding to the second magnification from the second camera based on a signal including the output area information. In claim 16,
The above first camera includes an image stabilizer that compensates for shaking of the output image,
An electronic device wherein said at least one processor is configured to determine said output area information based on control information of said image stabilizer. In claim 16,
An electronic device wherein said at least one processor is configured to determine said output area information based on an offset resulting from a change from said first magnification to said second magnification. In claim 16,
Further comprising a touchscreen display that displays a preview image based on an image captured through the first camera or the second camera,
At least one processor of the above,
Controlling the touchscreen display to display the preview image based on the first image frame acquired through the first camera;
Receive a scale change input associated with an input area within the preview image displayed on the touch screen display,
An electronic device configured to determine the output area information based on a position at which the above scale change input is input. In claim 19,
At least one processor of the above,
An electronic device configured to control the touchscreen display to display the preview image in an enlarged form with respect to the input area based on the second image frame output from the second camera based on the output area information.

Images