KR20220035820A - Electronic apparatus and control method thereof - Google Patents

Abstract

An electronic device and a method for controlling the same are disclosed. The electronic device includes a display, a memory that stores at least one instruction, and a processor that executes the at least one instruction stored in the memory to control the electronic device, where the processor receives a user input for entering a first mode, and , In response to the user input, a search is performed based on a search word corresponding to the user history through a first application related to the first mode, and the first image provided by the first application is displayed in the first area of the display based on the search results. and controls the display to display the second image acquired through a camera connected to the electronic device in the second area of the display.

Description

Electronic device and control method thereof {ELECTRONIC APPARATUS AND CONTROL METHOD THEREOF}

This disclosure relates to an electronic device and a control method thereof, and more specifically, to an electronic device and a control method for displaying an image provided through a camera together with other images.

Recently, with the development of electronic technology, a function has been provided that allows the display screen of an electronic device to be divided into several and various contents to be viewed simultaneously.

Additionally, with the development of electronic technology, it has become possible for users to easily find desired images using applications within electronic devices.

Accordingly, there is a need for a multi-view function that provides images provided through a camera connected to an electronic device in one area of the display and provides content desired by the user in another area.

However, in the past, in order to execute the multi-view function, there was the inconvenience of having to directly search for and display the content the user wanted to watch within the video application after connecting to the camera.

Accordingly, the need for technology that automatically plays content suitable for the user when executing the multi-view function that displays the camera screen and content screen is increasing.

The present disclosure was created in response to the above-mentioned need, and relates to an electronic device capable of simultaneously displaying content corresponding to user history along with an image acquired through a camera, which is the object of the present disclosure, and a method of controlling the same.

An electronic device according to an embodiment of the present disclosure for achieving the above object includes a display; a memory storing at least one instruction; and a processor configured to control the electronic device by executing at least one instruction stored in the memory, wherein the processor receives a user input for entering a first mode and operates in the first mode in response to the user input. Performs a search based on a search word corresponding to the user history through a first application related to the first application, displays the first image provided by the first application in the first area of the display based on the search result, and displays the electronic device The display is controlled to display a second image acquired through a camera connected to the second area of the display.

Additionally, the user history includes the user's search history and viewing history in the first mode, and the processor may obtain the search term based on the user history in the first mode.

In addition, the processor may obtain the search word based on at least one keyword included in the search history, the time when the video in the viewing history was played, and at least one keyword corresponding to the video.

Additionally, the processor may identify a first application among a plurality of applications installed on the electronic device based on the viewing history in the first mode.

Then, the processor performs a search using the search term in the first application, obtains a playlist including a plurality of videos corresponding to the search term, and selects one video from the plurality of videos included in the playlist. can be identified as the first image.

And, the processor controls the display to display a third image provided through another camera connected to the electronic device in a third area different from the area of the display where the first image and the second image are displayed. You can.

And, while operating in the first mode, when a user input for executing a plurality of applications is received, the processor displays a screen provided by a second application different from the first application with the first image and the second application. The display can be controlled to display the image in a fourth area that is different from the area of the display where the image is displayed.

And, when the screen provided by the second application is displayed in the fourth area, the processor stops playing the second video and displays the fourth video provided by the second application based on the search result. Can be played in area 4.

And, when a user input for entering the second mode is received, the processor performs a search using a search word corresponding to the user history in the second mode through a third application related to the second mode, Based on the search results, the display can be controlled to display the image provided by the third application in the first area and the second image in the second area.

Meanwhile, a method of controlling an electronic device according to an embodiment of the present disclosure for achieving the above object includes: receiving a user input for entering a first mode; performing a search based on a search term corresponding to user history through a first application related to the first mode in response to the user input; and displaying a first image provided by the first application in a first area of the display based on the search results, and displaying a second image obtained through a camera connected to the electronic device in a second area of the display. Step; includes.

And, the user history includes the user's search history and viewing history in the first mode, and performing the search includes obtaining the search term based on the user history in the first mode. It may further include.

In addition, the acquiring step may be a step of acquiring the search word based on at least one keyword included in the search history, the point in time when the video in the viewing history was played, and at least one keyword corresponding to the video. .

Additionally, performing the search may further include identifying a first application among a plurality of applications installed on the electronic device based on the viewing history in the first mode.

And, the displaying step includes performing a search using the search term in the first application and obtaining a play list including a plurality of images corresponding to the search term; and identifying one video among a plurality of videos included in the play list as the first video.

The method may further include displaying a third image provided through another camera connected to the electronic device in a third area different from the area of the display where the first image and the second image are displayed.

And, while operating in the first mode, when a user input for executing a plurality of applications is received, the first image and the second image are displayed on a screen provided by a second application that is different from the first application. The method may further include displaying a display in a fourth area that is different from the area of the display.

Then, when the screen provided by the second application is displayed in the fourth area, playback of the second video is stopped and the fourth image provided by the second application is displayed in the fourth area based on the search result. It may further include a step of reproducing.

And, when a user input for entering the second mode is received, performing a search using a search word corresponding to the user history in the second mode through a third application related to the second mode; and displaying an image provided by the third application in the first area and displaying the second image in the second area based on the search results.

According to the various embodiments described above, when the electronic device enters the multi-view mode, it can provide images appropriate for the user's history along with images acquired through a camera.

1 is a diagram for explaining a multi-view mode according to the present disclosure.
FIG. 2 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure.
FIG. 3 is a diagram illustrating a plurality of modules for performing a multi-view mode according to the present disclosure.
FIG. 4A is a diagram illustrating providing a first image and a second image in a PBP method according to the present disclosure.
FIG. 4B is a diagram illustrating providing a first image and a second image in a PIP method according to the present disclosure.
FIG. 4C is a diagram illustrating providing a first image and a second image in a PIP method according to the present disclosure.
FIG. 4D is a diagram illustrating a plurality of application screens being displayed along with an image acquired through a camera according to the present disclosure.
FIG. 5 is a diagram illustrating a method of performing a search through an application according to the present disclosure.
FIG. 6 is a diagram illustrating a UI for receiving a user input for entering the first mode according to the present disclosure.
FIG. 7 is a diagram illustrating a UI for receiving a user input for selecting a multi-view mode according to the present disclosure.
Figure 8 is a flowchart for explaining the operation of an electronic device according to the present disclosure.
FIG. 9 is a block diagram for explaining a specific configuration of an electronic device according to the present disclosure.
FIG. 10A is a diagram illustrating a broadcast image being displayed on a display of an electronic device according to the present disclosure.
FIG. 10B is a diagram illustrating a search screen according to the present disclosure being displayed together with a broadcast video according to the present disclosure.
FIG. 10C is a diagram showing that an image corresponding to the search result of FIG. 10B is displayed together with a broadcast image.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

1 is a diagram for explaining a multi-view mode according to the present disclosure.

Referring to FIG. 1, when the electronic device 100 receives a user input to enter the first mode, it can perform a search based on a search word corresponding to the first mode through the first application related to the first mode. there is.

According to the present disclosure, the first mode may be a multi-view mode for simultaneously displaying a screen provided through the camera 200 and a screen provided through an application in the electronic device 100. At this time, the first mode may be referred to as home training mode. And, the user input for entering the first mode is a user input for selecting an icon corresponding to the first mode displayed on the display of the electronic device 100, a user input through a remote control connected to the electronic device 100, and the user's voice. It may be a variety of user inputs, such as user input through commands and user input for connecting the camera 200 to the electronic device 100.

According to the present disclosure, the first application is an application that performs a search and provides content corresponding to the search results. For example, the first application may be various applications for providing video, such as an application providing streaming video, a video sharing application, a one-person media platform application, or a video application for playing video in the electronic device 100. The electronic device 100 may perform a search through an external server using the first application and receive an image identified as a search result from the external server in a streaming manner.

Specifically, upon entering the first mode, the electronic device 100 may perform a search through the first application based on a search term corresponding to the first mode.

In one embodiment, the electronic device 100 may perform a search through the first application based on a basic search term corresponding to the first mode. As an example, the default search word corresponding to the first mode, which is the home training mode, may be set to “home training.” These basic search terms may be preset by the manufacturer of the electronic device 100, etc.

Accordingly, when a user input for entering the first mode is received, the electronic device 100 may perform a search by identifying “home training” as a basic search word corresponding to the first mode through the first application.

In one embodiment, when the user history corresponding to the first mode is stored in the memory 110, the electronic device 100 obtains a search term corresponding to the user history from the first application and performs a search through the obtained search term. It can be done. The user history may include the user's search history, viewing history, purchase history, etc. in the first mode. That is, the electronic device 100 may acquire a keyword suitable for the first mode based on the past search history and past viewing history of the user of the electronic device 100 while operating in the first mode. Additionally, the electronic device 100 may perform a search using the obtained search term and display the first image 20 provided by the first application in the first area.

That is, when there is no user history in the first mode, the electronic device 100 may perform a search in the first application using a basic search term corresponding to the first mode. And, if there is a user history in the first mode, the electronic device 100 may obtain a search word based on the user history in the first mode and perform a search in the first application through the obtained search word. .

Then, the electronic device 100 displays the first image 20 provided by the first application in the first area of the display based on the search result, and displays the first image 20 provided through the camera 200 connected to the electronic device 100. The second image 10 may be displayed in the second area of the display.

In Figure 1, the first image and the second image are shown as being displayed in a PBP (Picture By Picture) method, but this is not limited to this, and the first and second images are displayed in a PIP (Picture In Picture) method as shown in Figures 4b and 4b. This may be displayed.

Additionally, in Figure 1, two images provided by a camera and a first application are shown simultaneously, but the present invention is not limited to this, and multiple images provided by a camera and multiple applications may be displayed simultaneously.

In FIG. 1 , the camera 200 is shown as an external component of the electronic device 100, but the present invention is not limited thereto, and the electronic device 100 may also include a camera. According to the above-described embodiment, when the electronic device 100 enters the first mode, it can identify an image corresponding to the user history and provide it together with the image provided from the camera.

FIG. 2 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the electronic device 100 may be implemented as various display devices such as a TV, smart TV, monitor, electronic picture frame, electronic blackboard, electronic table, laptop, etc.

Referring to FIG. 2, according to an embodiment of the present disclosure, the electronic device 100 may include a memory 110, a display 120, a camera 130, a communication interface 140, and a processor 150. there is.

At least one command related to the electronic device 100 may be stored in the memory 110 . Additionally, an operating system (O/S) for driving the electronic device 100 may be stored in the memory 110 . Additionally, the memory 110 may store various software programs or applications for operating the electronic device 100 according to various embodiments of the present disclosure. Additionally, the memory 110 may include a semiconductor memory such as flash memory or a magnetic storage medium such as a hard disk.

Specifically, the memory 110 may store various software modules for operating the electronic device 100 according to various embodiments of the present disclosure, and the processor 150 executes various software modules stored in the memory 110. Thus, the operation of the electronic device 100 can be controlled. That is, the memory 110 is accessed by the processor 150, and data read/write/modify/delete/update, etc. can be performed by the processor 150.

Meanwhile, in the present disclosure, the term memory 110 refers to memory 110, ROM (not shown), RAM (not shown) in the processor 150, or a memory card (not shown) mounted on the electronic device 100 (e.g. For example, it can be used to mean including micro SD card, memory stick).

In particular, the memory 110 according to the present disclosure can store user history in the first mode. Specifically, the memory 110 may store information about the user's search history and viewing history while operating in the first mode.

The display 120 can display various images. Here, video is a concept that includes images and videos, and the display 120 can display various videos such as broadcast content and multimedia content. Additionally, the display 120 can display a screen provided through the camera 130. Additionally, the display 120 may display various user interfaces (UIs) and icons.

In addition, the display 120 is equipped with a variety of displays such as LCD (Liquid Crystal Display Panel), LED (light emitting diode), OLED (Organic Light Emitting Diodes), LCoS (Liquid Crystal on Silicon), DLP (Digital Light Processing), etc. It can be implemented in the form of a display. Additionally, the display 120 may also include a driving circuit and a backlight unit that may be implemented in the form of a-si TFT, low temperature poly silicon (LTPS) TFT, or organic TFT (OTFT).

Additionally, the display 120 may be implemented as a touch screen by combining it with a touch sensor.

The camera 130 is configured to photograph the environment in which the electronic device 100 is located. The electronic device 100 may obtain an image of a user in an environment where the electronic device 100 is located through a camera. In FIG. 2, the camera 130 is shown as a component of the electronic device 100, but it is not limited to this, and the electronic device 100 is connected to an external camera and displays the image acquired from the external camera on the display 120. It can be displayed.

The communication interface 140 is a component that communicates with various types of external devices according to various types of communication methods. The communication interface 140 may include a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, and an NFC chip. The processor 150 may communicate with various external devices or external servers using the communication interface 140.

In particular, the communication interface 140, the Wi-Fi chip and the Bluetooth chip perform communication using the WiFi method and the Bluetooth method, respectively. When using a Wi-Fi chip or a Bluetooth chip, various connection information such as SSID and session key are first transmitted and received, and various information can be transmitted and received after establishing a communication connection using this. A wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), and LTE (Long Term Evolution). NFC chip refers to a chip that operates in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc.

The communication interface 140 can communicate with an external server and receive search results corresponding to a search word corresponding to the user history from the external server through the first application. Additionally, the communication interface 140 may receive images provided by the first application in a streaming manner according to the search results.

The processor 150 is electrically connected to the memory 110 and can control the overall operation and functions of the electronic device 100. The processor 150 controls the overall operation of the electronic device 100. To this end, the processor 150 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). . Processor 150 may be implemented in various ways. For example, the processor 150 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor. Processor, DSP). Meanwhile, in the present disclosure, the term processor 150 may be used to include a central processing unit (CPU), a graphics processing unit (GPU), and a main processing unit (MPU).

The processor 150 can control hardware or software components connected to the processor 150 by running an operating system or application program, and can perform various data processing and calculations. Additionally, the processor 150 may load and process commands or data received from at least one of the other components into volatile memory and store various data in non-volatile memory.

In particular, the processor 150 may receive a user input for entering the first mode. The user input for entering the first mode is a user input for selecting an icon corresponding to the first mode displayed on the display 120 of the electronic device 100, a user input through an external remote control connected to the electronic device 100, and a user input for entering the first mode. It may be a variety of user inputs, such as a user input through a voice command and a user input for connecting the camera 130 to the electronic device 100.

Additionally, the first mode may be a mode for simultaneously displaying a screen provided through the camera 130 and a screen provided through an application within the electronic device 100. For example, if the first mode is a home training mode, the processor (150) displays a content image provided through an application that can provide home training images in the first area of the display 120, and displays the image acquired through the camera 130 in the second area of the display 120. It can be displayed in .

Specifically, the processor 150 may perform a search based on a search term corresponding to the first mode through a first application related to the first mode in response to a user input for entering the first mode.

In one embodiment, the search word corresponding to the first mode may be a preset basic search word corresponding to the first mode.

In one embodiment, a search word corresponding to the first mode may be a search word obtained based on the user's history in the first mode. The user history may include the user's search history and viewing history in the first mode, and the processor 150 may obtain a search term based on the user's past search history and past viewing history while operating in the first mode.

In one embodiment, the processor 150 may identify a first application among a plurality of applications installed on the electronic device 100 based on the viewing history in the first mode. For example, among a plurality of applications installed in the electronic device 100, the application most used in the first mode may be identified as the first application. For example, when the first mode is the home training mode, the processor 150 may perform a search through the first application most used in the home training mode.

Then, when the processor 150 enters the first mode, it can obtain a search word corresponding to the user history from the first application. For example, the processor 150 based on at least one keyword included in the user's search history in the first mode, the time when the videos in the viewing history in the first mode were played, and at least one keyword corresponding to the video. You can obtain search terms. A specific method for the processor 150 to obtain a search word will be described later with reference to FIG. 5.

Then, the processor 150 performs a search using a basic search term corresponding to the first mode or a search term obtained based on user history and displays the first image provided by the first application in the first area of the display 120. The display 120 can be controlled to do so. That is, if there is no user history in the first mode, the processor 150 may perform a search in the first application using a basic search term corresponding to the first mode. Also, if there is a user history in the first mode, the processor 150 may obtain a search term based on the user history in the first mode and perform a search in the first application using the obtained search term.

For example, the processor 150 receives search results corresponding to a search term through the communication interface 140, receives the first image provided based on the search results in a streaming manner, and displays the first image 120. The display 120 can be controlled to display in the first area of .

In one embodiment, the processor 150 may perform a search using a search term in the first application and obtain a playlist including a plurality of images corresponding to the search term. Additionally, the processor 150 may identify one image among a plurality of images included in the playlist as the first image.

Then, the processor 150 displays the identified first image in the first area of the display 120 and displays the second image acquired through the camera 130 in the second area of the display 120. 120) can be controlled.

For example, the processor 150 can display the first image and the second image together through various display methods, such as a Picture In Picture (PIP) method or a Picture By Picture (PBP) method. For further details, see FIG. 4A. This will be described later with reference to FIGS. 4C to 4C.

In the above-described embodiment, it has been described that the first image and the second image are displayed in the first area and the second area, respectively, in the first mode, but the present disclosure is not limited to this.

In one embodiment, while displaying the first and second images, the processor 150 displays a third image acquired through another camera different from the camera 130 on the display 120 on which the first and second images are displayed. The display 120 can be controlled to display more information in a third area that is different from the area.

In one embodiment, the processor 150 displays the first image and the second image on the display 120, and displays a screen provided by a second application that is different from the first application on the display where the first image and the second image are displayed. The display 120 can be controlled to display more information in a fourth area that is different from the area in 120. Then, when the screen provided by the second application is displayed in the fourth area, the processor 150 stops playing the second image and displays the fourth image provided by the second application in the fourth area based on the search result. The display 120 can be controlled to play.

Additionally, in one embodiment, the processor 150 performs a search in a second mode that is different from the first mode based on a search term corresponding to the second mode, and displays an image provided based on the search result on the second mode of the display 120. The display 120 can be controlled to display the image in area 1 and display the second image acquired through the camera 130 in the second area of the display 120. That is, when a user input to enter the second mode is received, a search is performed using a search term corresponding to the user history in the second mode through a third application related to the second mode, and the second mode is searched based on the search results. The display 120 can be controlled to display the image provided to the application in the first area and display the second image acquired through the camera 130 in the second area.

FIG. 3 is a diagram illustrating a plurality of modules for performing a multi-view mode according to the present disclosure.

In one embodiment according to the present disclosure, as shown in FIG. 3, a multi-view mode may be performed through a plurality of modules 1000 to 4000 included in the processor 150. However, a plurality of modules for performing the multi-view mode may be included in the electronic device 100, but this is only an example, and at least some of the plurality of modules may be included in an external server.

As described above, a plurality of modules 1000 to 4000 may be located in the processor 150, but the present invention is not limited thereto, and a plurality of modules 1000 to 4000 may be located in the memory 110. When a plurality of modules 1000 to 4000 are located in the memory 110, the processor 150 loads the plurality of modules 1000 to 4000 from the non-volatile memory to the volatile memory to store the plurality of modules 1000 to 4000. 4000) can execute each function. Loading refers to an operation of loading and storing data stored in non-volatile memory in volatile memory so that the processor 150 can access it.

The user input processing module 1000 is configured to receive user input for entering the multi-view mode. User input for entering the multi-view mode includes user input received from an external remote control through the communication interface 140, user motion input obtained through the camera 130, and user input input through the UI displayed on the display 120. , may include a variety of user inputs, such as a user's voice command input and a user input of selecting an icon corresponding to a multi-view mode.

And, the multi-view mode according to the present disclosure may include a plurality of modes. For example, the multi-view mode may include a first mode that is a home training mode and a second mode that is a dance mode. In this case, the user input processing module 1000 may identify a multi-view mode corresponding to the user input among a plurality of multi-view modes based on the user input.

When a user input for entering the multi-view mode is received, the user input processing module 1000 may activate the multi-view module 2000, the video processing module 3000, and the camera image processing module 4000.

The multi-view module 2000 is a module for providing a first image provided through the video processing module 3000 and a second image provided through the camera image processing module 4000.

Upon entering the multi-view mode, the multi-view module 2000 can identify a search word corresponding to the multi-view mode.

In one embodiment, the multi-view module 2000 may identify a basic search word corresponding to the multi-view mode. The basic search term corresponding to the multi-view mode may be preset by the manufacturer of the electronic device 100 or the developer of the multi-view mode.

In one embodiment, a search word related to the multi-view mode may be identified based on a search word DB previously stored in the memory 110.

The search word DB is a DB in which user search history and viewing history information is stored in multi-view mode. When the multi-view mode includes multiple modes, user search history and viewing history information for each mode may be stored.

As an example, while operating in the first mode, which is a home training mode, when a user searches for “morning yoga” and watches a “morning yoga” video, the search term DB contains the search history for “morning yoga” and “morning yoga” Video viewing history may be stored as information about the first mode. Also, while operating in the second mode, which is the dancing mode, when the user searches for “basic dance” and watches the video “easy to learn dance,” the search term DB contains the search history for “basic dance” and “easy to learn dance” in the second mode. The video viewing history of “Learning Dance” may be saved as information for the second mode.

Additionally, the multi-view module 2000 can identify a search word corresponding to an image suitable for the multi-view mode based on the search history and viewing history stored in the search term DB. For example, if the user has a history of watching videos related to “morning yoga” in the first mode in the morning, when a user input to enter the first mode in the morning is received, the multi-view module 2000 ” can be identified as a search term corresponding to the first mode.

That is, if a user history corresponding to the multi-view mode is stored in the search term DB, the multi-view module 2000 selects at least one keyword (e.g., morning, yoga) obtained based on the search history included in the user history, and Search terms can be obtained based on the time when the video in the viewing history included in the user history was played (e.g., morning) and the keyword corresponding to the video included in the user history (e.g., “morning yoga”).

When a search word corresponding to the multi-view mode is identified, the multi-view module 2000 may transmit the obtained search word to the video processing module 3000. On the other hand, if the user history corresponding to the multi-view mode is not stored in the search term DB, the multi-view module 2000 may transmit the basic search term corresponding to the multi-view mode to the video processing module 3000.

The video processing module 3000 is a module for providing an image corresponding to a search term through an application corresponding to the multi-view module 2000.

When the video processing module 3000 receives a search term from the multi-view module 2000, the video processing module 3000 may identify an application to perform a search among a plurality of applications installed in the electronic device 100. For example, when in the first mode, the video processing module 3000 may identify the application most used by the user as the first application corresponding to the first mode.

When an application to perform a search is identified, the video processing module 3000 may obtain a playlist including a plurality of videos corresponding to the search word from an external server through the communication interface 140 using the identified application. Alternatively, the video processing module 3000 may obtain a playlist including a plurality of videos corresponding to the search word through an internal video DB in the memory. The internal video DB may be a DB that stores a plurality of images previously stored in the electronic device 100. That is, the first application may be an application that can stream video from an external server, but is not limited to this and may be an application that can search for and play internal video.

Additionally, the video processing module 3000 may provide one image among a plurality of images included in the playlist as the first image to the multi-view module 2000. For example, the video processing module 3000 may provide the video located at the top of the playlist among a plurality of videos included in the playlist as the first video to the multi-view module 2000. However, the method is not limited to this, and the algorithm for identifying the first video among the plurality of videos included in the playlist may be preset by the user. For example, it may be preset to identify the video with the most views or recommendations among a plurality of videos included in the playlist as the first video. Additionally, in one embodiment, the video processing module 3000 does not automatically identify the first video, but provides the entire playlist to the multi-view module 2000, so the first video can be selected according to the user's selection. .

The camera image processing module 4000 is a module for providing images acquired through the camera 130. When the camera image processing module 4000 is activated by the user input processing module 1000, the camera image processing module 4000 provides the second image captured by the user through the camera 130 to the multi-view module 2000. can do.

When the multi-view module 2000 receives the first image from the video processing module 3000 and the second image from the camera image processing module 4000, the multi-view module 2000 displays the first image 120. The display 120 can be controlled to display the first image in the first area and display the second image in the second area of the display 120. For example, the multi-view module 2000 displays the first image in the first area in a streaming manner when received by the video processing module 3000, and displays the second image captured by the user received by the camera image processing module 4000. The display 120 can be controlled to display in real time in the second area.

The multi-view module 2000 may control the display 120 to display the first and second images in a PIP or PBP manner. And, when images are provided through two cameras, the multi-view module 2000 receives images acquired through the first camera and the second camera from the camera image processing module 4000, and sends the first images to the first camera. The image acquired through the first camera is displayed in the second area of the display 120, and the image acquired through the second camera is displayed in the area of the display 120 where the first image and the second image are displayed. The display 120 can be controlled to display in a third area that is different from the display area. In this case as well, the multi-view module 2000 can control the display 120 to display the first image, second image, and third image in the PIP method or PBP method.

In addition, in the above-described embodiment, it has been described that an image provided through one application is displayed on the display 120, but the present invention is not limited to this. That is, while operating in multi-view mode in which the first image and the second image are provided, when a user input for executing a plurality of applications is received through the user input processing module 1000, the video processing module Along with the first image provided, a screen provided through a second application that is different from the first application may be transmitted to the multi-view module 2000. In addition, the multi-view module 2000 divides the first and second areas of the existing display 120 into three areas, and each area contains a first image, a screen provided through a second application, and a second image. The display 120 can be controlled to display .

While the first image and the second image are displayed together through the multi-view module 2000, when a user input for performing a search is received through the first application, the multi-view module 2000 displays the image received from the first application. Searches can be performed based on user input. The user input for performing a search may be a variety of user inputs, such as a user input selecting a search icon in the first application or a user inputting a voice command. And, when a search is performed according to a user input, the display 120 can be controlled to display an image corresponding to the search result in the first area of the display 120. Additionally, the multi-view module 2000 may store information about the search word for which the search was performed and the image corresponding to the search result in the search term DB.

FIGS. 4A and 4D are diagrams for explaining a method of providing a plurality of images through a multi-view mode according to the present disclosure.

FIG. 4A is a diagram illustrating providing a first image and a second image in a PBP method according to the present disclosure. The electronic device 100 divides the display screen into two areas and displays the first image 20 provided by the first application and the second image 10 obtained through the camera in different areas, as shown in FIG. 4A. can do.

4B and 4C are diagrams illustrating providing a first image and a second image in a PIP method according to the present disclosure. As shown in FIG. 4B, the electronic device 100 may display the second image 20 on the entire display screen and display the first image 10 overlapping a portion of the second image 20. Alternatively, the electronic device 100 may display the first image 10 on the entire display screen and display the second image 20 overlapping on a portion of the first image 10, as shown in FIG. 4C. .

That is, in the multi-view mode according to the present disclosure, the display screen can be divided by various display methods and two images can be provided, as shown in FIGS. 4A to 4C. This display method may be preset by the user.

FIG. 4D is a diagram illustrating a plurality of application screens being displayed along with an image acquired through a camera according to the present disclosure.

In the multi-view mode according to the present disclosure, multiple application screens can be displayed simultaneously. That is, referring to FIG. 4D, the electronic device 100 displays the first image 20 provided by the first application and the second image 10 obtained through the camera, as well as the first image 20 provided by an application different from the first application. 3 Videos 30 can be displayed together in PBP format.

FIG. 5 is a diagram illustrating a method of performing a search through an application according to the present disclosure.

When entering the first mode according to the present disclosure, the electronic device 100 displays the first image provided through the first application related to the first mode and the second image obtained through the camera on the display 120. can do.

And, while the first image and the second image are displayed, if a user input for performing a search is received through the first application, the electronic device 100 may perform a search using the first application.

Referring to FIG. 5, when you search for “morning stretching” in the first application, a playlist containing a plurality of images (50-1, 50-2, 50-3) corresponding to the search term is displayed on the display 120. It can be. Additionally, the electronic device 100 may display an image selected by the user among a plurality of images 50-1, 50-2, and 50-3 included in the playlist on the display 120 instead of the existing first image. .

In addition, the electronic device 100 may store information about the “morning stretching” search history, the time when the image selected by the user was played, and at least one keyword corresponding to the image selected by the user in the search term DB.

For example, when “morning stretching” is searched through the first application, the electronic device 100 may store the words “morning” and “stretching” as search history in the search term DB. And, when a user input for selecting the image 50-1 about “stretching in the morning” among the plurality of images 50-1, 50-2, and 50-3 is received, the electronic device 100 selects the image 50-1 for “stretching in the morning”. The time when the video (50-1) was played, the title of the video (50-1) (stretching in the morning), and the keywords (# waking up stretching) included in the video (50-1) can be saved in the search term DB. there is.

In FIG. 5, only the screen provided by the first application is shown on the display 120, but the screen is not limited to this. That is, even while a search is performed in the first mode, the first application screen and the second image provided through the camera may be displayed together.

FIG. 6 is a diagram illustrating a UI for receiving a user input for entering the first mode according to the present disclosure.

In one embodiment according to the present disclosure, when connection with a camera is performed, the electronic device 100 may display a UI 60 on the display asking whether to enter the first mode. Then, when a user input for entering the first mode is received through the UI 60, the electronic device 100 performs a search based on a search word corresponding to the user history through a first application related to the first mode, and , Based on the search results, the first image provided by the first application can be displayed in the first area of the display, and the second image obtained through the camera can be displayed in the second area of the display.

In FIG. 6 , when the electronic device 100 is connected to the camera, the UI 60 is displayed and the first mode is entered. However, the present disclosure is not limited to this. That is, after the electronic device 100 and the camera are connected, the electronic device 100 selects an icon related to the first mode, uses a user's voice command to enter the first mode, or uses an external remote control to enter the first mode. The first mode can also be entered through a user command.

FIG. 7 is a diagram illustrating a UI for receiving a user input for selecting a multi-view mode according to the present disclosure.

In one embodiment according to the present disclosure, the multi-view mode may be implemented as a plurality of modes. For example, the first mode may be a multi-view mode for home training, and the second mode may be a multi-view mode related to dance.

In one embodiment according to the present disclosure, when connection with a camera is performed, the electronic device 100 may display a UI 70 on the display asking whether to enter the first mode or the second mode. Additionally, the electronic device 100 may enter the multi-view mode corresponding to the user input through the UI 70.

For example, when a user input to enter the first mode is received, the electronic device 100 may obtain a search word based on the user's history in the first mode through the first application related to the first mode. Then, the electronic device 100 performs a search using the first application and displays the first image provided in the first area of the display, and displays the second image obtained through the camera in the second area of the display. You can.

For example, when a user input to enter the second mode is received, the electronic device 100 may obtain a search term based on the user's history in the second mode through a second application related to the second mode. Then, the electronic device 100 performs a search using the second application and displays the first image provided in the first area of the display, and displays the second image acquired through the camera in the second area of the display. You can.

Here, the first application may be the most used application in the first mode, and the second application may be the most used application in the second mode. However, the application is not limited to this and the applications to be used in the first mode and the second mode may be preset by the user.

And, in the first mode, a search term may be obtained based on the search history in which the user performed the search in the first mode and the viewing history in the first mode, and in the second mode, the search term in the second mode may be obtained based on the search history in which the user performed the search in the second mode. A search term may be obtained based on search history and viewing history in the second mode. In FIG. 7 , when the electronic device 100 is connected to the camera, the UI 70 is displayed to select the first mode or the second mode, but the present disclosure is not limited to this. That is, after the electronic device 100 and the camera are connected, the electronic device 100 selects an icon related to the first mode or the second mode, uses a user's voice command to enter the first mode or the second mode, or uses the first mode. You can also enter the first mode or the second mode through a user command through an external remote control to enter the mode or the second mode.

Figure 8 is a flowchart for explaining the operation of an electronic device according to the present disclosure.

First, the electronic device 100 may receive a user input for entering the first mode (S810). The user input for entering the first mode is a user input of selecting an icon corresponding to the first mode displayed on the display of the electronic device 100, a user input through a remote control connected to the electronic device 100, and a user's voice command. It may be a variety of user inputs, such as user input through user input and user input for connecting the camera to the electronic device 100.

When a user input for entering the first mode is received, the electronic device 100 may perform a search based on a search word corresponding to the user history through the first application related to the first mode in response to the user input ( S820). Specifically, the user history may include search history and viewing history in the first mode, and the electronic device 100 may obtain a search word based on the user history in the first mode. For example, the electronic device 100 may obtain a search term based on at least one keyword included in the search history in the first mode, the time when the video in the viewing history was played, and at least one keyword corresponding to the video. .

Additionally, the electronic device 100 may identify a first application among a plurality of applications installed on the electronic device 100 based on the viewing history when in the first mode. For example, when in the first mode, the electronic device 100 may identify the most used application as the first application, or the application corresponding to the first mode may be preset by the user.

Then, the electronic device 100 displays the first image provided by the first application in the first area of the display based on the search results, and displays the second image acquired through the camera connected to the electronic device 100 on the display. It can be displayed in the second area (S830). In one embodiment, the electronic device 100 obtains a playlist including a plurality of images corresponding to a search result in the first application and identifies one image among the plurality of images included in the playlist as the first image. You can. For example, the electronic device 100 may identify the image located at the top of the playlist as the first image among a plurality of images included in the playlist. However, the method is not limited to this, and the algorithm for identifying the first video among the plurality of videos included in the playlist may be preset by the user. For example, it may be preset to identify the video with the most views or recommendations among a plurality of videos included in the playlist as the first video.

FIG. 9 is a block diagram for explaining a specific configuration of an electronic device according to the present disclosure.

As shown in FIG. 9, the electronic device 900 according to an embodiment of the present disclosure includes a memory 910, a display 920, a camera 930, a communication interface 940, an input/output interface 960, and a microphone. It may include (970) and a speaker (980). However, this configuration is an example, and of course, in carrying out the present disclosure, new configurations may be added or some configurations may be omitted in addition to these configurations. Since the memory 110, display 120, camera 130, communication interface 140, and processor 150 have been described in detail with reference to FIG. 2, the input/output interface 960, microphone 970, and speaker ( 980) is explained.

The input/output interface 960 can input and output at least one of audio and video signals. In particular, the input/output interface 960 can receive images from an external device that can provide images to the display 920.

Meanwhile, the input/output interface 960 may be HDMI (High Definition Multimedia Interface), but this is only an example, and may include MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), and Thunderbolt. It may be any one of Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). Depending on the implementation, the input/output interface 960 may include a port that inputs and outputs only audio signals and a port that inputs and outputs only video signals as separate ports, or may be implemented as a single port that inputs and outputs both audio signals and video signals.

The microphone 970 may receive an audio signal from outside. The audio signal may include a user command, and the user command may include a command for entering the first mode. However, the user command is not limited to this and may include various voice commands for controlling the electronic device 900.

In one embodiment according to the present disclosure, when a user command for entering the first mode is input from the microphone 970, the processor 950 may enter the first mode based on the input user voice. That is, a user command according to the present disclosure may be input from the microphone 970 through the user's voice. Specifically, the processor 950 may receive a user's voice through the microphone 970, perform voice recognition on the received user's voice, and obtain a user command corresponding to the received user's voice. More specifically, the processor 950 may receive a user voice through the microphone 970, input the received user voice into a learned artificial intelligence model, and obtain a user command corresponding to the received user voice. And, when a user command corresponding to the received user voice is obtained, the processor 950 may perform an operation corresponding to the user command, as described above. That is, in the embodiment described above through the drawings, the first mode is entered by selecting the icon on the UI for entering the first mode, but this is not limited, and the electronic device 900 enters the first mode through the user's voice. You can enter.

Additionally, as an example according to the present disclosure, a search within the first application may be performed by receiving a user's voice from the microphone 970.

Audio data can be output under the control of the speaker 980 processor 950. In particular, in various embodiments according to the present disclosure, the speaker 980 may output audio data provided from the first image provided by the first application.

In the above-described embodiments, it has been described that when entering the multi-view mode according to the present disclosure, an image acquired through a camera and an image provided from an application related to the multi-view mode are displayed together, but the present disclosure is not limited to this. That is, in the present disclosure, as shown in FIGS. 10A, 10B, and 10C, images provided from applications related to the multi-view mode may be displayed together with broadcast images provided from the electronic device 100.

FIG. 10A is a diagram illustrating a broadcast image being displayed on a display of an electronic device according to the present disclosure.

Specifically, referring to FIG. 10A, the electronic device 100 can display a broadcast image on the display. That is, when the electronic device 100 according to the present disclosure is implemented as a TV or smart TV, the electronic device 100 can display a broadcast image on the display.

FIG. 10B is a diagram illustrating a search screen according to the present disclosure being displayed together with a broadcast video according to the present disclosure.

As shown in FIG. 10A, when a user input to enter the multi-view mode is received while a broadcast image is displayed on the display of the electronic device 100, the electronic device 100 operates through an application corresponding to the multi-view mode. A search screen that performs a search can be displayed along with the broadcast video. User input for entering the multi-view mode is, for example, a user input for selecting an icon for entering the multi-view mode displayed on the display of the electronic device 100, and a user input through an external remote control connected to the electronic device 100. and may include various user inputs, such as user input through a user's voice command.

In FIG. 10B, the search screen and the broadcast video are shown together in a PIP method, but the present disclosure is not limited to this and the search screen and the broadcast video can be displayed together in a PBP method or various other methods.

FIG. 10C is a diagram showing that an image corresponding to the search result of FIG. 10B is displayed together with a broadcast image.

In response to the search result shown in FIG. 10B, the electronic device 100 may display a content image corresponding to the multi-view mode along with the broadcast image shown in FIG. 10A through an application corresponding to the multi-view mode, as shown in FIG. 10C.

In Figure 10C, the content video and broadcast video corresponding to the multi-view mode are shown together in a PBP method, but the present disclosure is not limited to this and the search screen and broadcast video can be displayed together in a PIP method or various other methods. .

Since these embodiments can be modified in various ways and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In addition, the above-described embodiments may be modified into various other forms, and the scope of the technical idea of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to completely convey the technical idea of the present disclosure to those skilled in the art.

In the above-described embodiments, the UI for setting the display mode for the source that provides the image to the display 120 according to the present disclosure is shown as being displayed on the display 120 in the horizontal display mode. It is not limited. That is, of course, UIs according to the present disclosure can be displayed on the display 120 in portrait mode.

The terms used in this disclosure are merely used to describe specific embodiments and are not intended to limit the scope of rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present disclosure, expressions such as “have,” “may have,” “includes,” or “may include” refer to the presence of the corresponding feature (e.g., component such as numerical value, function, operation, or part). , and does not rule out the existence of additional features.

In the present disclosure, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in the present disclosure can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (eg, a third component).

On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

The expression “configured to” used in the present disclosure may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware.

Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Meanwhile, various elements and areas in the drawing are schematically drawn. Accordingly, the technical idea of the present invention is not limited by the relative sizes or spacing drawn in the attached drawings.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or similar device using software, hardware, or a combination thereof. According to hardware implementation, embodiments described in the present disclosure include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions. In some cases, embodiments described herein may be implemented with a processor itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, the methods according to various embodiments of the present disclosure described above may be stored in a non-transitory readable medium. These non-transitory readable media can be mounted and used in various devices.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, programs for performing the various methods described above may be stored and provided in non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed on a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or online through an application store (e.g. Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or created temporarily in a storage medium such as the memory of a manufacturer's server, an application store server, or a relay server.

In addition, although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure pertains without departing from the gist of the present disclosure as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present disclosure.

100: electronic device
200: Camera

Claims (18)

In electronic devices,
display;
a memory storing at least one instruction; and
A processor that controls the electronic device by executing at least one instruction stored in the memory,
The processor,
Receiving user input for entering the first mode,
In response to the user input, a search is performed based on a search word corresponding to the user history through a first application related to the first mode,
Based on the search results, the first image provided by the first application is displayed in the first area of the display, and the second image acquired through a camera connected to the electronic device is displayed in the second area of the display. Electronic device that controls the display.
According to paragraph 1,
The user history includes a basic search term corresponding to the first mode and the user's search history and viewing history in the first mode,
The processor,
An electronic device that obtains the search term based on the user history in the first mode.
According to paragraph 2,
The processor,
An electronic device that obtains the search term based on at least one keyword included in the search history, a point in time when a video in the viewing history was played, and at least one keyword corresponding to the video.
According to paragraph 2,
The processor,
An electronic device that identifies a first application among a plurality of applications installed on the electronic device based on the viewing history in the first mode.
According to paragraph 1,
The processor,
Performing a search using the search term in the first application to obtain a playlist including a plurality of videos corresponding to the search term,
An electronic device that identifies one image among a plurality of images included in the play list as the first image.
According to paragraph 1,
The processor,
An electronic device that controls the display to display a third image provided through another camera connected to the electronic device in a third area of the display that is different from the area of the display where the first image and the second image are displayed.
According to paragraph 2,
The processor,
While operating in the first mode, when a user input for executing a plurality of applications is received, a screen provided by a second application different from the first application is displayed on the display where the first image and the second image are displayed. An electronic device that controls the display to display in a fourth area different from the area.
In clause 7,
The processor,
When the screen provided by the second application is displayed in the fourth area, playback of the second video is stopped and the fourth video provided by the second application is played in the fourth area based on the search result. Electronic devices.
According to paragraph 1,
The processor,
When a user input for entering the second mode is received, a search is performed using a search term corresponding to the user history in the second mode through a third application related to the second mode,
An electronic device that controls the display to display an image provided by the third application in the first area and display the second image in the second area based on the search results.
In a method of controlling an electronic device,
Receiving a user input for entering a first mode;
performing a search based on a search term corresponding to user history through a first application related to the first mode in response to the user input; and
Based on the search results, displaying a first image provided by the first application in a first area of the display and displaying a second image acquired through a camera connected to the electronic device in a second area of the display. A control method containing ;
According to clause 10,
The user history includes a basic search term corresponding to the first mode and the user's search history and viewing history in the first mode,
The steps for performing the search are:
A control method further comprising obtaining the search term based on the user history in the first mode.
According to clause 11,
The obtaining step is,
A control method comprising the step of obtaining the search word based on at least one keyword included in the search history, the point in time when the video in the viewing history was played, and at least one keyword corresponding to the video.
According to clause 11,
The steps for performing the search are:
The control method further includes identifying a first application among a plurality of applications installed on the electronic device based on the viewing history in the first mode.
According to clause 10,
The steps indicated above are:
performing a search using the search term in the first application, and obtaining a playlist including a plurality of videos corresponding to the search term; and
A control method comprising: identifying one image among a plurality of images included in the play list as the first image.
According to clause 10,
The control method further comprising displaying a third image provided through another camera connected to the electronic device in a third area different from the area of the display where the first image and the second image are displayed.
According to clause 11,
While operating in the first mode, when a user input for executing a plurality of applications is received, a screen provided by a second application different from the first application is displayed on the display where the first image and the second image are displayed. A control method further comprising displaying a fourth area different from the area.
According to clause 16,
When the screen provided by the second application is displayed in the fourth area, playback of the second video is stopped, and the fourth video provided by the second application is played in the fourth area based on the search result. A control method further comprising:
According to clause 10,
When a user input for entering the second mode is received, performing a search using a search term corresponding to the user history in the second mode through a third application related to the second mode; and
Based on the search results, displaying an image provided by the third application in the first area and displaying the second image in the second area.

Images