KR102396375B1 - Method and device for playing multimedia - Google Patents

Abstract

A device and method for acquiring music information about music played in an electronic device, acquiring a storyline for the music based on the music information, and displaying an image by acquiring one or more images based on the storyline is initiated.

Description

Multimedia playback method and device thereof

The present disclosure relates to a multimedia playback method and a device therefor, and more particularly, to a method and device for displaying an image related to the music while music is being played in an electronic device.

As devices for multimedia reproduction, electronic devices such as digital audio players, smart phones, tablet PCs, and smart TVs are widely used by users. These electronic devices may reproduce various multimedia files such as music, images, and videos.

Also, even if multimedia files are not stored in the electronic device, streaming media may be played in the electronic device through the Internet or the like. With the development of network technology, the types and amounts of digital multimedia that individuals can access through electronic devices are increasing day by day.

When music is played on an electronic device, the electronic device stores images or text related to the music, such as album cover, lyrics, and tag information of the music, so that the user can have a better audiovisual experience and better understand the music. It can be implemented to display.

However, there is a problem that the album cover of the corresponding music is not sufficient to enhance the user's understanding of the corresponding music.

In addition, since the lyrics or tag information of the corresponding music is displayed as text to the user, it is very cumbersome and time consuming to recognize and understand the text from the user's point of view.

The user may watch a video such as a music video of the corresponding music through the electronic device for a rich audiovisual experience and a deeper understanding of the corresponding music. However, when a video file such as a music video of a corresponding music is played in an electronic device, the video files generally have a large file size, so there is a problem of wasting disk capacity of the corresponding electronic device.

When these video files are played by streaming in an electronic device through the Internet or the like, the problem of wasting disk capacity can be solved. However, when the corresponding electronic device is connected to the Internet through a mobile communication network, a problem of increasing the burden of data usage on the user may arise. Even if the burden of data usage on the user is insignificant, the corresponding video file may not be played smoothly depending on the Internet connection speed or connection state. If the corresponding video file is not played smoothly, the music that the user originally intended to listen to may not be played smoothly, which may cause inconvenience to the user.

Accordingly, there is a need for a technology capable of enhancing a user's understanding of corresponding music while providing a rich audiovisual experience to the user while music is being played in the electronic device.

Various embodiments may provide a method and a device for displaying an image related to music while playing music in an electronic device.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure is to obtain music information about music played in an electronic device, and to obtain a storyline for the music based on the music information. It is possible to provide a multimedia playback method comprising the steps of: acquiring one or more images based on the storyline; and displaying the one or more images.

Also, the music information may include meta-information of the music, and the storyline may be obtained based on the meta-information.

Also, the music information may include audio fingerprint information of the music, and the storyline may be obtained based on the audio fingerprint information.

In addition, the step of obtaining one or more images based on the storyline

and selecting one or more keywords based on the storyline, wherein the one or more images are obtained based on the selected one or more keywords.

In addition, the step of obtaining one or more images based on the storyline

obtaining a plurality of candidate images based on the selected one or more keywords; and determining a priority among the plurality of candidate images based on a predetermined criterion, wherein the one or more images are selected from among the plurality of candidate images. It is possible to provide a multimedia playback method, characterized in that it is obtained based on the priority.

In addition, the step of determining the priority among the plurality of candidate images based on a predetermined criterion

and obtaining semantic information of the plurality of candidate images, wherein the priority is determined by a similarity between the semantic information of the plurality of candidate images and a keyword selected based on the storyline. can provide

Also, the storyline may include one or more sentences composed of text, and the one or more images may be obtained based on the one or more sentences.

In addition, the one or more sentences constitute one or more paragraphs based on similarity, the music is divided into one or more music segments corresponding to the one or more paragraphs, and the one or more images are displayed based on the one or more music segments. It is possible to provide a multimedia playback method, characterized in that

In addition, the storyline is the lyrics of the music, the music information includes the lyrics information about the lyrics, and the one or more images can provide a multimedia playback method, characterized in that displayed based on the lyrics information there is.

In addition, based on the storyline, further comprising the step of obtaining emotion information, the one or more images may provide a multimedia playback method, characterized in that obtained based on the emotion information.

In addition, the one or more images may provide a multimedia playback method, characterized in that rendered based on the emotion information.

In addition, the storyline is divided into one or more paragraphs based on at least one of temporal information, spatial information, and person information, the music is divided into one or more music segments corresponding to the one or more paragraphs, and the one or more images can provide a multimedia playback method, characterized in that displayed based on the one or more music segments.

In addition, a second aspect of the present disclosure is to obtain music information about music played in an electronic device, obtain a storyline for the music based on the music information, and display one or more images based on the storyline. It is possible to provide a multimedia playback device comprising a control unit for acquiring and a display unit for displaying the one or more images.

Also, the music information may include meta-information of the music, and the storyline may be obtained based on the meta-information.

Also, the music information may include audio fingerprint information of the music, and the storyline may be obtained based on the audio fingerprint information.

Also, the control unit may select one or more keywords based on the storyline, and the one or more images may be obtained based on the selected one or more keywords.

In addition, the storyline may include one or more sentences composed of a series of texts, and the one or more images may be obtained based on the one or more sentences.

In addition, the control unit may further obtain the emotion information related to the music based on the storyline, and the one or more images may be obtained based on the emotion information.

In addition, the storyline is divided into one or more paragraphs based on at least one of temporal information, spatial information, and person information, the music is divided into one or more music segments corresponding to the one or more paragraphs, and the one or more images can provide a multimedia playback device, characterized in that displayed based on the one or more music segments.

In addition, a third aspect of the present disclosure may provide a computer-readable medium in which a program for executing the method of the first aspect is recorded on a computer.

FIG. 1 is a diagram illustrating an example in which an image related to music is displayed while music is being played in an electronic device, according to an embodiment.
2 and 3 are block diagrams of an electronic device according to an embodiment.
4 is a flowchart of a method of displaying an image related to music while playing music in an electronic device, according to an exemplary embodiment.
5 is a flowchart of a method of displaying an image related to the corresponding music by using a keyword included in a storyline of the corresponding music while music is being played in the electronic device, according to an exemplary embodiment.
6 is a diagram illustrating an example in which an image related to the corresponding music is displayed using a description of the corresponding music while music is being played in the electronic device, according to an embodiment.
7 is a diagram illustrating an example in which an image related to the corresponding music is displayed using lyrics of the corresponding music while music is being played in the electronic device, according to an exemplary embodiment.
8 is a flowchart of a method of displaying an image related to the corresponding music by using a plurality of candidate images related to the corresponding music while music is being played in an electronic device, according to an exemplary embodiment.
9 is a flowchart of a method of rendering and displaying an image related to music while music is being played in an electronic device, according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And, in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

The terms used in this specification are selected as currently widely used general terms as possible while considering functions according to various embodiments, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in this specification should be defined based on the meaning of the term and the contents throughout this specification, rather than the simple name of the term.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and includes one or more other features or It should be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof does not preclude the possibility of addition. In particular, the numbers described in the specification are examples for helping understanding, and the present invention should not be limited by the numbers described in the specification.

In addition, terms such as “…unit” and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

Although "first", "second", etc. are used throughout the specification to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Therefore, it goes without saying that the “first component” mentioned below may also be a “second component” within the spirit of the embodiment.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

Throughout the specification, an electronic device is a device that can play music, and includes a smartphone, tablet, mobile phone, personal digital assistant (PDA), media player, audio player, digital audio player, PMP (Portable Multimedia Player), e-book terminal, and digital broadcasting device. terminals, personal computers (PCs), laptops (laptops), micro servers, global positioning system (GPS) devices, navigation, kiosks, MP3 players, smart TVs, digital cameras and other mobile or non-mobile computing devices, but are not limited thereto doesn't happen

Throughout the specification, music may refer to an audio file in which corresponding music is recorded. The electronic device may play an audio file by driving a multimedia playback application, and by playing the audio file, music recorded in the corresponding audio file may be played. Such a multimedia playback application may be basically installed in the electronic device or may be separately installed by a user.

The format of the audio file includes all formats reproducible through the electronic device. The format of the audio file may include an uncompressed audio format, a lossy compressed audio format, a lossless compressed audio format, and the like.

For example, the uncompressed audio format may be stored with an extension such as WAV or AIFF (Audio Interchange File Format), and the lossy compressed audio format may be stored with an extension such as MP3 or AAC (Advanced Audio Codec). The lossy compressed audio format may be stored with an extension such as FLAC (Free Lossless Audio Codec) or Monkey's Audio APE. However, the format of the audio file is not limited thereto and may have various extensions.

Throughout the specification, a multimedia playback device is an electronic device capable of playing music and/or displaying an image, and among the above-described electronic devices, all electronic devices capable of playing music and/or displaying an image may be referred to as a multimedia playback device. there is.

Throughout the specification, a storyline of music means a narrative description of the corresponding music, and the storyline of music may consist of one or more images, one or more videos, and/or text for narrative description.

1 is a diagram illustrating an example in which an image related to music is displayed while music is being played in an electronic device, according to an embodiment.

Although it is shown that music is played on one and the same electronic device and an image related to the music is displayed in FIG. 1, it may be implemented such that music is played on one electronic device and an image related to the music is displayed on another electronic device. may be

The multimedia playback application 100 may be driven in the electronic device, and music may be played. Here, the reproduction of music means reproduction of an audio file of the corresponding music.

Referring to FIG. 1 , the multimedia playback application includes an image window 110 displaying an image related to the played music, a controller 120 for controlling the multimedia playback application, and an information window displaying music information on the played music. (130) may be included.

Referring to FIG. 1 , when music is played in an electronic device, images 110A, 110B, 110C, and 110D related to the corresponding music may be displayed through the image window 110 . Images 110A, 110B, 110C, and 110D related to the corresponding music may be obtained based on the storyline of the corresponding music.

The electronic device may obtain the storyline of the music from a server external to the electronic device.

By displaying an image obtained based on the storyline of the music, a user who listens to the music may more easily understand the music. In addition, the user may be given a rich audiovisual experience.

Since the electronic device only plays music and displays an image related to the music, there is an effect of consuming less resources of the electronic device than playing a video related to the music. In addition, when the electronic device is connected to the Internet through a mobile communication network and plays music or images by streaming, there is an effect of reducing the user's data usage burden.

2 and 3 are block diagrams of an electronic device according to an embodiment.

As shown in FIG. 2 , the electronic device 1000 includes a display unit 1210 and a control unit 1300 . However, not all of the components shown in FIG. 2 are essential components of the electronic device 1000 . The electronic device 1000 may be implemented by more components than the components shown in FIG. 2 , and the electronic device 1000 may be implemented by fewer components than the components shown in FIG. 2 .

The controller 1300 typically controls the overall operation of the electronic device 1000 to perform a function of the electronic device 1000 .

Specifically, the controller 1300 obtains music information about the electronic device 1000 or music reproduced in an electronic device separate from the electronic device 1000 .

When music is played in the electronic device 1000 , the corresponding music may be output through the sound output unit 1220 of the electronic device 1000 . In this case, the audio file of the corresponding music may be previously stored in the storage unit 1700 of the electronic device 1000 or received from an external server through the communication unit 1500 of the electronic device 1000 and played.

The music information may be previously stored in the storage unit 1700 of the electronic device 1000 or may be obtained by being received from an external server through the communication unit 1500 of the electronic device 1000 .

The music information may be a file name of an audio file in which music is recorded. When the meta information of the corresponding audio file is not recorded, the file name itself of the corresponding audio file may be used as music information to obtain a storyline of the corresponding music.

The music information may be meta information of an audio file in which music is recorded. The meta information may include information about at least one of the title, singer, composer, lyricist, performer, genre, duration, album, composer, music segment, lyrics, and description of the corresponding music. there is. When the meta information of the corresponding audio file is not completely recorded, the meta information of the audio file may be obtained by searching in a search engine or the like based on the file name of the corresponding audio file and/or already recorded meta information. Here, the search engine may provide meta information of the audio file in response to the query.

The music information may be lyric information corresponding to an audio file in which music is recorded. When the audio file is reproduced, the electronic device 1000 may load lyric information corresponding to the corresponding audio file and display lyrics included in the lyric information.

Lyric information may be obtained from a lyric file with an LRC extension. The lyrics information may include information related to the start time of each sentence constituting the lyrics. When the audio file is reproduced and lyrics information corresponding to the audio file is loaded, the electronic device 1000 loads the lyrics information corresponding to the audio file, and based on the start time of each sentence included in the lyrics information It may be implemented so that each sentence is scrolled and displayed. Here, since the lyrics of the music can be displayed according to the playback time of the audio file, the user can more easily understand the lyrics of the music being played.

The music information may be audio fingerprint information of music. The audio fingerprint information of music means information about characteristics of the corresponding music, and may be used for identification and/or search of music.

When music is reproduced in the electronic device 1000 , the electronic device 1000 may obtain audio fingerprint information by analyzing an audio file in which the corresponding music is recorded.

When music is played in an electronic device separate from the electronic device 1000, the electronic device 1000 records the music through the microphone 1620 to generate audio data, and analyzes the audio signal of the generated audio data to Audio fingerprint information may be obtained.

Specifically, the audio fingerprint information may be obtained using a power spectrum generated after converting an audio signal into a frequency domain using a Fast Fourier Transform (FFT) method, without being limited thereto, through various methods. can be obtained.

The electronic device 1000 may transmit the obtained audio fingerprint information to the audio DB, and obtain meta information of the audio file identified by the corresponding audio fingerprint information through DB matching in the audio DB. Here, the audio DB may be stored in the storage unit 1700 of the electronic device 1000 or an external server, and may include audio fingerprint information and meta information of various audio files. The meta information of the audio file includes at least one of the title, singer, composer, lyricist, performer, genre, playing time, album, composer, music segment, lyrics, and description of the music recorded in the audio file. may include information about

The control unit 1300 acquires a storyline for the reproduced music based on the acquired music information.

The storyline of music means a narrative description of the corresponding music, and the storyline of the music may consist of one or more images, one or more videos, and/or text for the narrative description.

The storyline for the music may be previously stored in the storage unit 1700 of the electronic device 1000 or may be obtained by being received from an external server through the communication unit 1500 of the electronic device 1000 .

The storyline of the music may be obtained by searching a search engine or the like based on meta information of an audio file in which the corresponding music is recorded. Here, the search engine may provide a storyline about music in response to a query.

The controller 1300 acquires one or more images based on the acquired storyline.

One or more images may be previously stored in the storage unit 1700 of the electronic device 1000 , or may be obtained by being received from an external server through the communication unit 1500 of the electronic device 1000 .

When the storyline of music consists of one or more images, an image related to the corresponding music may be obtained from among one or more images constituting the storyline of the music.

When a storyline of music consists of one or more videos, an image related to the music may be obtained from frames of one or more videos constituting the storyline of music. A method of acquiring an image related to the music in the electronic device 1000 from among frames of one or more videos constituting a storyline of the music will be described later with reference to FIG. 4 .

When the storyline of music is composed of text, an image related to the music may be obtained based on the text constituting the storyline of the music. A method of acquiring an image related to the music in the electronic device 1000 based on text constituting a storyline of the music will be described later with reference to FIG. 5 .

The display unit 1210 may display one or more acquired images.

An image obtained based on the storyline of the music is displayed through the display unit 1210 of the electronic device 1000, so that a user who listens to the music can understand the music more deeply. In addition, the user may be given a rich audiovisual experience.

Not all of the components shown in FIG. 2 are essential components of the electronic device 1000 . The electronic device 1000 may be implemented by more components than the components shown in FIG. 2 , and for example, as shown in FIG. 3 , the electronic device 1000 according to an embodiment may include the display described above. In addition to the unit 1210 and the control unit 1300, a user input receiving unit 1100, an output unit 1200, a sensing unit 1400, a communication unit 1500, an A/V input unit 1600, and a storage unit 1700 are further added may include

Hereinafter, each component of the electronic device 1000 will be described in detail.

The user input receiving unit 1100 means a means for a user to input data to control the electronic device 1000 . For example, the user input receiving unit 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, red There may be a mechanical tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto. In particular, when the touch pad is configured as a touch screen or a touch screen display by forming a layer structure with the display unit 1210 , the user input receiving unit 1100 may be used as an output device in addition to the input device.

The touch pad may be configured to detect not only a real-touch but also a proximity touch, and for convenience of description in the present specification, both real-touch and proximity touch are may be referred to as “touch”.

As used herein, "real-touch" means an input generated when a pointer is physically touched on the screen, and "proximity-touch" means an input generated when a pointer is touched on the screen. It means an input that is generated while being approached by a predetermined distance from the corresponding screen even if it is not physically touched.

As used herein, the term “pointer” refers to a tool for directly touching or proximally touching a specific part of a displayed screen. Examples thereof include a stylus pen, a finger, and the like.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210 , a sound output unit 1220 , and a vibration motor 1230 . .

The display unit 1210 displays information processed by the electronic device 1000 . For example, the display 1210 may display an execution screen of an operating system driven in the electronic device 1000 and an execution screen of an application driven in the operating system.

On the other hand, when the display unit 1210 and the touch pad constitute a layer structure to form a touch screen, the display unit 1210 may be used as an input device in addition to an output device. The display unit 1210 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display ( 3D display) and an electrophoretic display (electrophoretic display) may include at least one. Also, according to an implementation form of the electronic device 1000 , the electronic device 1000 may include two or more display units 1210 . In this case, two or more display units 1210 may be disposed to face each other using a hinge.

The sound output unit 1220 outputs audio data received from the communication unit 1500 or stored in the storage unit 1700 . Also, the sound output unit 1220 outputs a sound signal related to a function performed by the electronic device 1000 . The sound output unit 1220 may include a speaker, a buzzer, and the like.

The vibration motor 1230 may output a vibration signal. For example, the vibration motor 1230 may output a vibration signal corresponding to the output of audio data or video data. Also, the vibration motor 1230 may output a vibration signal when a touch is input to the touch screen.

The controller 1300 may perform a function of the electronic device 1000 by generally controlling the overall operation of the electronic device 1000 . For example, the control unit 1300 executes the programs stored in the storage unit 1700 , and thus the user input receiving unit 1100 , the output unit 1200 , the sensing unit 1400 , the communication unit 1500 , and the A/V input unit. (1600) and the like can be controlled in general.

The sensing unit 1400 may sense a state of the electronic device 1000 or a state around the electronic device 1000 , and transmit the sensed information to the controller 1300 .

The sensing unit 1400 includes a magnetic sensor 1410, an acceleration sensor 1420, a temperature/humidity sensor 1430, an infrared sensor 1440, a gyroscope sensor 1450, a position sensor ( For example, it may include at least one of a GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , and an illuminance sensor 1490 , but is not limited thereto. Since a function of each sensor can be intuitively inferred from the name of a person skilled in the art, a detailed description thereof will be omitted.

The communication unit 1500 may include one or more components that enable communication between the electronic device 1000 and an external device. For example, the communication unit 1500 may include a short-range communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .

Short-range wireless communication unit 1510 is a Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, near field communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared (IrDA) , an infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, etc., but is not limited thereto.

The mobile communication unit 1520 transmits and receives a wireless signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission/reception of a voice signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an implementation example, the electronic device 1000 may not include the broadcast receiver 1530 .

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , and the like. The camera 1610 may obtain an image frame such as a still image or a video through an image sensor in a video call mode or a photographing mode. The image captured through the image sensor may be processed through the controller 1300 or a separate image processing unit.

The image frame processed by the camera 1610 may be stored in advance in the storage unit 1700 or transmitted to the outside through the communication unit 1500 . Two or more cameras 1610 may be provided according to the configuration of the terminal.

The microphone 1620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 1620 can receive an acoustic signal from an external device or speaker. The microphone 1620 may use various noise removal algorithms to remove noise generated in the process of receiving an external acoustic signal. The microphone 1620 may receive a user's voice answer input corresponding to the problem content provided by the above-described control unit 1300 .

The storage unit 1700 may store a program for processing and control of the controller 1300 , and may store data input to or output from the electronic device 1000 .

The storage unit 1700 includes a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk. The storage unit 1700 may be referred to as a memory.

Programs stored in the storage unit 1700 may be classified into a plurality of modules according to their functions. For example, the programs may be classified into a UI module 1710 , a touch screen module 1720 , a notification module 1730 , etc. there is.

The UI module 1710 may provide a specialized UI, GUI, or the like that interworks with the electronic device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the controller 1300 . The touch screen module 1720 according to an embodiment may recognize and analyze a touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

In order for the touch pad to sense a user's touch, various sensors may be provided inside or around the touch pad. An example of such various sensors is a tactile sensor. A tactile sensor refers to a sensor that senses a touch of a specific object to the extent or higher than that of a human being. The tactile sensor may sense various information such as the roughness of the contact surface, the hardness of the contact object, and the temperature of the contact point.

In addition, there is a proximity sensor as an example of these various sensors. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity without mechanical contact using the force of an electromagnetic field or infrared rays. Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor.

Also, as an example of these various sensors, there is a force touch sensor. Different functions may be performed in the electronic device 1000 according to the amount of pressure that touches the touch pad, so that a gesture input that may be applied through the touch pad may be further diversified.

Gesture input can be implemented in various ways. Among them, “tap” refers to an input method that is applied when the pointer is moved from the touchpad after the pointer is touched once on the touchpad. "tap)" means two times, "multiple tap" means an input method in which three or more touches are applied within a predetermined time, and "long tap" means touch for a predetermined time or longer or until a specific event occurs. It refers to an input method that is applied while maintaining a state, that is, an input method that is tapped for a long time. In addition, "drag" refers to an operation of moving the pointer to another location on the touchpad while the user touches the touchpad using a pointer and the touchpad and the pointer remain in touch. Basically, similar to “drag”, a user touches the touchpad using a pointer, and then moves the pointer to another location on the touchpad while maintaining the touch state with the touchpad and the pointer. However, it means an operation in which the movement speed is relatively faster than that of “drag”.

The notification module 1730 may generate a signal for notifying the occurrence of an event of the electronic device 1000 . Examples of events generated in the electronic device 1000 include signal reception, message reception, key signal input, schedule notification, and the like. The notification module 1730 may output a notification signal in the form of a video signal through the display unit 1210, may output a notification signal in the form of an audio signal through the sound output unit 1220, and the vibration motor 1230 It is also possible to output a notification signal in the form of a vibration signal through

In addition, some or all of the respective configurations of the electronic device 1000 illustrated in FIGS. 2 and 3 may be implemented by at least one hardware processor. For example, some or all of the respective configurations of the electronic device 1000 may be implemented through a separate processor other than the main processor of the electronic device 1000 .

In addition, some of the respective configurations of the electronic device 1000 illustrated in FIGS. 2 and 3 may be implemented by at least one software program. For example, some functions of the electronic device 1000 may be implemented by an operating system program, and some functions may be implemented by an application program. Accordingly, the functions of the electronic device 1000 may be implemented by at least one hardware and at least one software, and the functions of the electronic device 1000 implemented by the software may be implemented by an operating system installed in the electronic device 1000 . and by an application.

The electronic device 1000 according to an embodiment includes a processor, a memory for storing and executing program data, a permanent storage such as a disk drive, a communication port for communicating with an external device, a touch panel, a key, user interface devices such as buttons and the like. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, the computer-readable recording medium includes a magnetic storage medium (eg, read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and an optically readable medium (eg, CD-ROM). ), DVD (Digital Versatile Disc), etc. The computer-readable recording medium may be distributed among network-connected computer systems, so that the computer-readable code may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed on a processor.

4 is a flowchart of a method of displaying an image related to music while playing music in an electronic device, according to an exemplary embodiment.

For convenience of description, it will be described with reference to FIG. 1 . Referring to FIG. 1 , a multimedia playback application 100 may be driven in an electronic device to play music. Here, the reproduction of music means reproduction of an audio file of the corresponding music.

In step S100, the electronic device acquires music information about music played in the corresponding electronic device.

The music information may be previously stored in the electronic device or may be obtained by being received from an external server through the Internet or the like.

The music information may be meta information of an audio file in which music is recorded. The meta information may include information about at least one of the title, singer, composer, lyricist, performer, genre, duration, album, composer, music segment, lyrics, and description of the corresponding music. there is.

For example, as shown in the information window 130 of FIG. 1 , the electronic device may obtain the singer and title (MOBB-Chenzao) of the music. The electronic device may acquire various types of meta information not shown in the information window 130 in addition to the singer and title of the music.

Meanwhile, the electronic device may be implemented to acquire music information about music played in an electronic device different from the corresponding electronic device.

In this case, the music information may be audio fingerprint information, the electronic device may generate audio data by recording the corresponding music, and may obtain audio fingerprint information by analyzing an audio signal of the generated audio data.

In step S110, the electronic device acquires a storyline for the corresponding music based on the music information obtained in step S100.

A storyline for music may consist of one or more images, one or more videos, and/or text.

The storyline of the music may be previously stored in the electronic device or may be obtained by being received from an external server through the Internet.

The storyline of the music may be obtained by searching a search engine or the like based on meta information of an audio file in which the corresponding music is recorded. Here, the search engine may provide a storyline about music in response to a query. For example, a search engine may provide a music video for music in response to a query.

Referring to FIG. 1 , an electronic device may acquire a video related to the corresponding music, for example, a music video, based on a singer and a title (MOBB-CHENZAO) of the music. That is, the storyline of the corresponding music is composed of the music video of the corresponding music.

In step S120, the electronic device acquires one or more images based on the storyline obtained in step S110.

The image may be obtained by being stored in advance in the electronic device or received from an external server through the Internet or the like.

When a storyline of music consists of one or more videos, an image related to the music may be obtained from among frames of one or more videos constituting the storyline.

For example, referring to FIG. 1 , the electronic device may acquire one or more images from frames constituting a music video. In this case, the electronic device may acquire one or more images at predetermined time intervals from among the frames constituting the music video. For example, one or more images may be acquired at 10-second intervals from among the frames constituting the music video. As shown in the information window 130 of FIG. 1 , when the playback time of the music played on the electronic device is 3:59 and the playback time of the music video of the corresponding music is also 3:59, 0 in the music video: 1st frame at 00, 2nd frame at 0:10, 3rd frame at 0:20, ... , and a twenty-fourth frame at 3:50 may be obtained.

In step S130, the electronic device displays one or more images obtained in step S120.

For example, referring to FIG. 1 , the electronic device may display the acquired images 110A, 110B, 110C, and 110D.

The electronic device may be implemented to display the acquired images 110A, 110B, 110C, 110D in accordance with the played music. For example, frames obtained at a predetermined time interval from the music video may be displayed at the predetermined time interval in the frame order in the electronic device.

For example, referring to FIG. 1 , when the playback time of music played in the electronic device is 3:59 and 10 seconds have elapsed after the music is played, the second frame at 0:10 in the music video is displayed in the electronic device. can be displayed. Accordingly, as shown in the information window 130 , the second frame 110A of the corresponding music video may be being displayed at 0:12 playback time. Similarly, the fifth frame 110B, the thirteenth frame 110C, and the eighteenth frame 110D of the music video may be being displayed at 0:46, 1:51, and 2:33 playback time points, respectively.

If the playing time of the corresponding music and the playing time of the music video of the corresponding music are different from each other, the electronic device may display images based on the ratio of the two playing times. For example, when the music video playback time is 7:58, which is twice the music playback time of 3:59, images acquired at 10-second intervals may be displayed on the electronic device at 5-second intervals.

Since the electronic device only plays music and displays an image related to the music, there is an effect of consuming less resources of the electronic device than playing a video related to the music. In addition, when the electronic device is connected to the Internet through a mobile communication network and plays music or images by streaming, there is an effect of reducing the user's data usage burden.

5 is a flowchart of a method of displaying an image related to the corresponding music by using a keyword included in a storyline of the corresponding music while music is being played on the electronic device, according to an exemplary embodiment.

For convenience of description, it will be described with reference to FIG. 6 .

6 is a diagram illustrating an example in which an image related to the corresponding music is displayed using a description of the corresponding music while music is being played in the electronic device, according to an embodiment.

Steps S200, S210, S230, and S240 are similar to steps S100, S110, S120, and S130 described above in FIG. 4, respectively, and thus overlapping descriptions will be omitted.

In step S200, the electronic device acquires music information about music played in the corresponding electronic device.

The music information may be meta information of an audio file in which music is recorded. The meta information may include at least one of a title, a singer, a composer, a lyricist, a performer, a genre, a playing time, an album, a composer, a music segment, lyrics, and a description of the corresponding music.

For example, as shown in the information window 230 of FIG. 6 , the electronic device may obtain the composer and title (Beethoven - Symphony No. 5) of the music. The electronic device may acquire various types of meta information not shown in the information window 230 in addition to the singer and title of the music.

In step S210, the electronic device acquires a storyline for the corresponding music based on the music information obtained in step S200.

A storyline for music may consist of one or more images, one or more videos, and/or text.

The storyline of the music may be previously stored in the electronic device or may be obtained by being received from an external server through the Internet.

The storyline of the music may be stored as meta information of an audio file in which the corresponding music is recorded. The storyline of the music may include a description of the corresponding music and/or lyrics of the corresponding music among the information included in the meta information.

For example, the electronic device may acquire various types of meta information not shown in the information window 230 in addition to the composer and title (Beethoven - Symphony No. 5) of the music, and the corresponding music among the information included in the acquired meta information. explanation can be obtained. In this case, the description of the corresponding music may include a description of each of the music segments constituting the corresponding music.

Music segments are musically distinct parts within one piece of music, and may be parts distinguished by a musical form.

For example, popular music typically includes an intro, verse, pre-chorus, chorus, bridge, outro, sabi, hook ( hook), ad lib, humming, scat, and the like.

For example, a symphony is generally divided into four movements, each in a fast tempo sonata allegro form, a slow tempo ABA or theme and variation form; It may have a minuet or scherzo form of medium/fast tempo, a sonata or rondo form of a fast tempo, and the like. Also, each movement may be divided into a plurality of paragraphs. A music segment of music that is a symphony may be each movement of the corresponding symphony, or may be each paragraph.

In the case of Beethoven's 5th Symphony, it is divided into 4 movements, and the 4 movements may correspond to 4 music segments. The music segment information included in the meta information may include information on a start time of each music segment. For example, the total playing time of the audio file of Beethoven's 5th Symphony is 34:25, and in that audio file, the first movement starts at 00:00, the second movement starts at 8:25, and the third movement starts at 16:00 and the fourth movement starts at 24:44, the start time of each music segment may be 00:00, 8:25, 16:00, and 24:44.

The four movements have the nicknames 'Struggle', 'Hope', 'Doubt', and 'Victory' respectively. Therefore, the storyline of Beethoven's 5th Symphony may include descriptions of the four movements and their aliases.

When a storyline for music includes only a description of the music and/or lyrics of the music, the storyline is composed of text.

On the other hand, the storyline of the music may be obtained by searching in a search engine or the like based on the music information of the corresponding music, for example, meta information of an audio file in which the corresponding music is recorded. Here, the search engine may provide a description of the music and/or lyrics of the music in response to the query. Accordingly, the electronic device may obtain the description of the music and/or the lyrics of the music provided from the search engine.

When the search engine provides various content as well as a description of the music and/or lyrics of the music in response to the query, the electronic device may be implemented to select a storyline for the music based on a predetermined condition among the various content. may be

For example, the predetermined condition may be an order between contents. That is, the highest rated content among various content items may be selected as the storyline for the music.

For example, the predetermined condition may be the number of overlapping content or the overlapping ratio. That is, the content provided by the search engine with the most overlap among various content may be selected as the storyline for the music.

The electronic device may select a storyline for music from a plurality of search engines. Furthermore, the electronic device may be implemented to select a storyline for music based on a predetermined condition among contents provided from a plurality of search engines. For example, the predetermined condition may be the number of overlapping content or the overlapping ratio. That is, content provided with the most overlap by a plurality of search engines may be selected as a storyline for music.

The electronic device may select a storyline for music from among various contents based on a combination of the above-described conditions.

In step S220, the electronic device acquires one or more keywords based on the storyline obtained in step S210.

When a storyline for music includes only a description of the music and/or lyrics of the music, the storyline is composed of text.

A storyline composed of text can be analyzed to obtain keywords. As a result of the analysis of the storyline, the keyword may be selected based on the part-of-speech of the word in the storyline, the number of appearances of the word, context, and the like. For example, a noun, a verb, and/or a frequently used word in the storyline may be selected as the keyword, but the present invention is not limited thereto, and the keyword may be selected based on various criteria.

Based on the storyline composed of text, the electronic device may acquire one or more keywords. For example, when Beethoven's 5th Symphony is played back on the electronic device, the electronic device performs 'Struggle', 'Hope', and 'Doubt' based on the storyline of Beethoven's 5th Symphony. )' and 'Victory' can be acquired.

In step S230, the electronic device acquires one or more images based on the keyword obtained in step S220.

One or more images may be obtained by being stored in advance in the electronic device or received from an external server through the Internet or the like.

When one or more images are stored in advance in the electronic device, the plurality of images are stored in an image database of a storage unit of the electronic device, so that one or more images may be obtained from the image database based on a keyword. Here, the image database may store an image, a title of the image, a description of the image, and the like. The electronic device may search the image database using the obtained keyword. The electronic device may acquire one or more images having a title or description matching the keyword as a result of searching the image database.

When one or more images are received and acquired from an external server through the Internet, the electronic device may acquire one or more images using a search engine. Here, the search engine may provide an image in response to a query based on each keyword.

Referring to FIG. 6 , four images 210A corresponding to each of the keywords 'Struggle', 'Hope', 'Doubt', and 'Victory' obtained in step S220. , 210B, 210C, 210D) may be obtained.

In step S240, the electronic device displays the image obtained in step S230.

For example, referring to FIG. 6 , the electronic device may display four acquired images 210A, 210B, 210C, and 210D.

The electronic device may be implemented to display the acquired four images 210A, 210B, 210C, and 210D in accordance with the played music. For example, the four images 210A, 210B, 210C, and 210D may be displayed respectively during each of the four time ranges that divide the total playing time of the reproduced music into quarters.

The electronic device may be implemented to display the obtained four images 210A, 210B, 210C, 210D in accordance with a music segment of the music being played. For example, the music segment of the music being played and the four images 210A, 210B, 210C, and 210D each correspond to each other, and when a music segment is played, an image corresponding to the music segment may be displayed. . That is, the image corresponding to the first music segment may be displayed during a time range from the start time of the first music segment to the start time of the second music segment located next to the first music segment.

Even if there is no music video or a separate explanatory image corresponding to the reproduced music, the electronic device may acquire and display the image by using a storyline including text. Accordingly, a user can more easily understand music, and since the electronic device acquires and displays an image using text, there is also an effect of saving resources of the electronic device.

7 is a diagram illustrating an example in which an image related to the corresponding music is displayed using lyrics of the corresponding music while music is being played in the electronic device, according to an exemplary embodiment.

The storyline of the music may be obtained by searching in a search engine or the like based on the music information of the corresponding music, for example, meta information of an audio file in which the corresponding music is recorded. Here, the search engine may provide a description of the music and/or lyrics of the music in response to the query. Accordingly, the electronic device may obtain the description of the music and/or the lyrics of the music provided from the search engine.

When a storyline for music includes only a description of the music and/or lyrics of the music, the storyline is composed of text. Here, the storyline may include one or more sentences composed of text. One or more sentences may be distinguished based on punctuation marks, such as periods, question marks, and exclamation marks, or may be distinguished through grammatical analysis.

For example, the lyrics of the song "Let's sing together" by a singer named "The Earth" are "I'm in times of trouble" and she comes to me ( She comes to me), Saying looking ahead to, Let's sing together", the lyrics contain 4 sentences.

In the lyrics, each sentence may be distinguished through grammatical analysis. For example, one phrase divided based on one verb may be one sentence.

When the storyline includes punctuation marks, each sentence may be distinguished based on the punctuation marks.

The electronic device may acquire one or more images based on one or more sentences constituting the storyline.

One or more images may be obtained by being stored in advance in the electronic device or received from an external server through the Internet or the like.

When one or more images are stored in advance in the electronic device, a plurality of images are stored in the image database of the storage unit of the electronic device, so that one image is stored in the image database based on each sentence or keyword of each sentence constituting the storyline from the image database. More than one image can be obtained. Here, the image database may store an image, a title of the image, a description of the image, and the like. The electronic device may search the image database by using each sentence constituting the storyline or a keyword of each sentence. The electronic device may acquire one or more images having a title or description matching each sentence constituting the storyline or a keyword of each sentence as a result of searching the image database.

When one or more images are received and acquired from an external server through the Internet, the electronic device may acquire one or more images using a search engine. Here, the search engine may provide an image in response to a query based on each sentence constituting the storyline or a keyword of each sentence.

Referring to FIG. 7 , images 310A, 310B, 310C, and 310D corresponding to each sentence constituting the lyrics of “Let's sing together” of The Earth may be obtained. The electronic device may display the acquired images 310A, 310B, 310C, 310D.

Meanwhile, the meta information may include lyrics information. Accordingly, when an audio file is reproduced and lyrics information corresponding to the audio file is loaded, the electronic device loads the lyrics information corresponding to the audio file, and based on the start time of each sentence included in the lyrics information, each It may be implemented so that the sentences are scrolled and displayed. Furthermore, the electronic device may display an image corresponding to each sentence according to the start time of each sentence. Accordingly, since the lyrics of the music and images corresponding to the lyrics can be displayed according to the playback time of the audio file, the user can more easily understand the music being played.

In an embodiment, one or more sentences constituting a storyline may constitute one or more paragraphs. The electronic device may distinguish a paragraph based on a degree of similarity between sentences constituting the storyline. The similarity between sentences may be calculated by a well-known similarity calculation method.

Currently, known similarity calculation methods can be divided into five categories, namely, a character matching method, a TF-IDF (term frequency-inverse document frequency) vector method, a probabilistic method, a sentence structure method, and a semantic expansion method. The character matching method calculates the similarity of sentences according to the same number of words in two sentences. A typical example is the Jaccard Similarity Coefficient method. In the TF-IDF vector method, a sentence is expressed as a TF-IDF vector, and the cosine value of the two vectors is used as a degree of similarity. The probabilistic method introduces a language model framework and acquires the similarity of two sentences using probability. In the sentence structure method, a sentence is divided into different components according to a matching approach of a sentence template, and a degree of similarity is calculated based on the structure of the component.

For example, when the similarity value of the first sentence is predetermined, the second sentence may have a similarity value according to the similarity calculated using the first sentence and the second sentence. That is, since the similarity is calculated using the self sentence and the previous sentence, all sentences may have a similarity value. The similarity values of all sentences may form a similarity value sequence.

A threshold value may be set for the similarity value sequence. In addition, each similarity value in the similarity value sequence may be compared with a threshold value. When a certain similarity value is greater than the threshold value, it may be determined that the similarity between the sentence having the corresponding similarity value and the previous sentence is relatively high. When the similarity value is lower than the threshold value, it may be determined that the similarity between the sentence having the corresponding similarity value and the previous sentence is relatively low. When the similarity value is lower than the threshold value, the sentence having the corresponding similarity value belongs to the next paragraph of the paragraph to which the previous sentence belongs, and the sentence having the corresponding similarity value becomes the first sentence of the next paragraph.

The threshold value may be set by a user or an administrator of the backend server, or may be set as a default value. The default value may be obtained by the following statistical method. First, when a plurality of music of various types is selected and a storyline for each music is obtained, the number of paragraphs in each storyline is determined. Then, a similarity value sequence is obtained for an arbitrary storyline including X paragraphs, and the similarity values included in the similarity value sequence are arranged from small values to large values, so that the Xth similarity value is critical for the corresponding storyline. It can be set to a value. An average of the threshold values for each storyline may be determined as a default value.

In one embodiment, the music may be divided into music segments using the following method. First, samples are extracted from music at predetermined time intervals. For example, the predetermined time interval may be a 10 ms interval. Then, the characteristics of the extracted sample are analyzed. For example, the characteristics of the sample may include time domain and frequency domain characteristics, fundamental frequency characteristics, and Mel Frequency Cepstral Coefficients (MFCC) characteristics. The extracted sample is determined to which music segment it belongs by using a classification device. For example, the classification device may be a vector support device. Music sample classification technology based on sample characteristics is a field that is being researched and applied, and will not be repeated here.

By using the lyrics information and music segment information included in the meta information of the music, a sentence and/or a paragraph of the lyrics may be associated with a music segment of the corresponding music. The lyrics information may include information related to the start time of each sentence constituting the lyrics, and the music segment information may include information about the start time of each music segment. Accordingly, sentences having a start time between the start time of a music segment and the start time of the next music segment of the corresponding music segment may be associated with the corresponding music segment. Based on the above-described similarity calculation method, when lyrics are divided into paragraphs, a paragraph may be associated with a music segment of the corresponding music.

Paragraphs of the music's storyline can be associated with music segments in different ways. For example, in the case of popular music, a paragraph of the corresponding popular music may be associated with a music segment in the following way. For convenience of description, it is assumed that the corresponding popular music includes an intro, a verse, a chorus, and a bridge music segment. Since intros and bridges in popular music generally have no lyrics, chorus music segments and bridge music segments may not relate to paragraphs in the storyline. And, if the storyline of music consists of M paragraphs, the last paragraph of the storyline is associated with the chorus music segment of popular music, and M-1 paragraphs excluding the last paragraph in the storyline can be associated with the verse music segment. there is.

A music segment may be divided into sub-segments. In the above example, the verse music segment may be divided into sub-segments. First, samples are extracted at predetermined time intervals from a verse music segment. For example, the predetermined time interval may be a 10 ms interval. All samples extracted from the verse music segment form a sample sequence.

For example, when the similarity value of the first sample in the sample sequence is predetermined, the second sample may have a similarity value according to the similarity calculated using the first sample and the second sample. That is, since the similarity is calculated using the self sample and the previous sample, all samples may have similarity values. The similarity values of all samples may form a sample similarity value sequence.

Sample similarity values included in the sample similarity value sequence are arranged from a small value to a large value, and the M-1th sample similarity value may be set as a threshold value. The threshold value may be set by a user or an administrator of the backend server, or may be set as a default value. A sample similarity value and a threshold value of each sample included in the sample sequence are compared, so that the verse music segment may be divided into M-1 sub-segments. For example, when the sample similarity value of a sample is greater than the threshold value, it may be determined that the similarity between the sample having the corresponding sample similarity value and the previous sample is relatively high. When the sample similarity value is lower than the threshold value, it may be determined that the similarity between the sample having the corresponding sample similarity value and the previous sample is relatively low. When the sample similarity value is lower than the threshold value, the sample having the corresponding sample similarity value belongs to the next subsegment of the subsegment to which the previous sample belongs, and the sample having the corresponding sample similarity value becomes the first sample of the next subsegment. M-1 sub-segments may be associated with M-1 paragraphs except for the last paragraph in the storyline.

Music segments of music may be played continuously in the electronic device. When a certain music segment is reproduced, the electronic device may display an image corresponding to each sentence included in a paragraph related to the corresponding music segment. Since intros and bridges in popular music generally have no lyrics, chorus music segments and bridge music segments may not relate to paragraphs in the storyline. As such, when a music segment not associated with a paragraph of the storyline is played, a default image may be displayed. The default image may be an album cover of played music.

Music may be divided into music segments, and music segments may be divided into sub-segments, and an image corresponding to the sub-segment may be displayed while the sub-segment is reproduced. Accordingly, while the music is being reproduced, the image is displayed more statically, and the user's understanding of the corresponding music can be further improved.

8 is a flowchart of a method of displaying an image related to the corresponding music by using a plurality of candidate images related to the corresponding music while music is being played in an electronic device, according to an exemplary embodiment.

Steps S300, S310, S320, S330, and S360 are similar to S200, S210, S220, S230, and S240 described above in FIG. 5, and thus overlapping descriptions will be omitted.

In step S300, the electronic device acquires music information about music played in the corresponding electronic device.

In step S310, the electronic device acquires a storyline for the corresponding music based on the music information obtained in step S300.

In step S320, the electronic device acquires one or more keywords based on the storyline obtained in step S310.

In step S330, the electronic device acquires a plurality of candidate images based on the one or more keywords obtained in step S320.

The electronic device may acquire a plurality of candidate images corresponding to one keyword.

The plurality of candidate images may be obtained by being stored in advance in the electronic device or received from an external server through the Internet or the like.

When a plurality of candidate images are received and obtained from an external server through the Internet, the electronic device may obtain a plurality of candidate images using a search engine. Here, the search engine may provide a plurality of candidate images in response to a query based on keywords.

The electronic device may obtain a plurality of candidate images from a plurality of search engines.

In step S340, the electronic device determines a priority among the plurality of candidate images obtained in step S330 based on a predetermined criterion.

For example, the predetermined criterion may be an order among a plurality of candidate images. That is, among the plurality of candidate images, a highly rated candidate image has a higher priority than a lowly rated candidate image from the search engine.

For example, the predetermined criterion may be the number of overlapping or overlapping ratios of the candidate image. That is, the more redundantly provided by the search engine among the plurality of candidate images, the higher the priority.

For example, the predetermined criterion may be a degree of similarity between the candidate image and the keyword. Semantic information on the meaning of the candidate image may be obtained through semantic analysis of the candidate image, and the similarity between the semantic information and the keyword may be determined. Semantic analysis and similarity comparison are not described here as they are well researched and applied.

When the electronic device obtains a plurality of candidate images from a plurality of search engines, the electronic device may be implemented to determine a priority among the plurality of candidate images provided from the plurality of search engines based on a predetermined criterion. For example, the predetermined criterion may be the number of overlapping or overlapping ratios of the candidate image. That is, the candidate image provided with the most overlap by a plurality of search engines has a higher priority.

The electronic device may determine a priority among the plurality of candidate images based on the combination of the aforementioned criteria.

In step S350, the electronic device acquires one or more images from among a plurality of candidate images based on the priority determined in step S340.

The electronic device may acquire one image having the highest priority among a plurality of candidate images corresponding to one keyword. That is, one keyword corresponds to one image.

The electronic device may acquire the plurality of images in the order of highest priority among the plurality of candidate images so that the plurality of images correspond to one keyword. The number of images obtained from among the plurality of candidate images may be implemented to be proportional to the playing time of a music segment corresponding to the corresponding keyword.

In step S360, the electronic device displays the one or more images obtained in step S350.

Since an image more suitable for the keyword of the storyline of the music may be displayed, the user who listens to the music may more easily understand the corresponding music.

9 is a flowchart of a method of rendering and displaying an image related to music while music is being played in an electronic device, according to an exemplary embodiment.

Steps S400, S410, S420, and S440 are similar to steps S100, S110, S120, and S130 described above in FIG. 4, respectively, and thus overlapping descriptions will be omitted.

In step S400, the electronic device acquires music information about music played in the corresponding electronic device.

In step S410, the electronic device acquires a storyline for the corresponding music based on the music information obtained in step S400.

In step S420, the electronic device acquires emotion information and one or more images based on the storyline obtained in step S410.

When the storyline of music is composed of text, one or more keywords may be obtained based on the storyline, and emotion information corresponding to each keyword may be obtained through the word sentiment database. The word sentiment database may store a word and sentiment information of the corresponding word as a pair, and through DB matching between each keyword and the words stored in the word sentiment database, the electronic device may acquire sentiment information corresponding to each keyword . For example, since the word “trouble” is negative, the word sentiment database may store the word “trouble” and sentiment information representing “negative” as a pair. Emotion information may be composed of text, code, symbol, number, etc., and various emotions may be expressed by a combination thereof.

When the storyline of music consists of one or more images, semantic information about the meaning of the one or more images can be obtained through semantic analysis of the one or more images, and the emotional information corresponding to the semantic information is a word It can be obtained through a sentiment database. Through DB matching between each semantic information and words stored in the word sentiment database, the electronic device may acquire sentiment information corresponding to each image.

The word sentiment database may be pre-stored in the electronic device or may be pre-stored in an external server.

In step S430, the electronic device renders one or more images obtained in step S420 based on the emotion information obtained in step S420.

Rendering the image means adjusting the hue, saturation, brightness, contrast, and/or transparency of the image or giving a graphic effect to the image in order to generate a specific emotion in the user who has viewed the image.

For example, when the obtained emotion information represents a negative emotion, the electronic device may lower the brightness or the saturation of the obtained image in order to generate a negative emotion in the user who has viewed the image.

For example, when the acquired emotion information represents a positive emotion, the electronic device may increase the brightness or saturation of the acquired image in order to generate a positive emotion in the user who has viewed the image.

In step S440, the electronic device displays one or more images rendered in step S430.

Since the image is rendered and displayed, the user can more emotionally appreciate the storyline of the played music.

According to an embodiment, the storyline may be divided into one or more paragraphs based on at least one of temporal information, spatial information, and person information, and music may be divided into one or more music segments corresponding to one or more paragraphs.

Time information means words related to time in a storyline composed of text. For example, the word related to time may be morning, noon, night, spring, summer, autumn, winter, and the like. When the storyline includes time information "autumn", the sentence including "autumn" belongs to the next paragraph of the paragraph to which the previous sentence of the corresponding sentence belongs.

Spatial information refers to words related to space in a storyline composed of text. For example, a word related to space may be a mountain, a building, a sea, a room, a house, and the like. When the storyline includes spatial information "mountain", the sentence including "mountain" belongs to the next paragraph of the paragraph to which the previous sentence of the corresponding sentence belongs.

Person information refers to words related to a person in a storyline composed of text. For example, the words related to the person may be you, you, me, me, he, her, the name of the person, and the like. When the storyline includes the person information "she", the sentence including "she" belongs to the next paragraph of the paragraph to which the previous sentence of the corresponding sentence belongs.

The divided paragraphs of the storyline may be related to 'gi', 'win', 'before', and 'finish' of the storyline. When the storyline is divided into four paragraphs, the four paragraphs may be associated with each of 'gi', 'win', 'before', and 'finish' of the storyline, respectively. If the storyline is divided into three paragraphs, the first paragraph of the three paragraphs relates to 'gi', the second paragraph relates to 'win' and 'before', and the third paragraph relates to 'finish' can be When the storyline is divided into two paragraphs, the first paragraph may be related to 'gi' and 'win', and the second paragraph may be related to 'before' and 'finish'. When the storyline is divided into 5 or more paragraphs, the first paragraph is related to 'gi', the last paragraph is related to 'finish', and the preceding and following paragraphs of the remaining paragraphs are equally divided into 'win' and 'win', respectively. It can be related to 'before'.

Paragraphs of the music's storyline can be associated with music segments in different ways. For example, in the case of popular music, a paragraph of the corresponding popular music may be associated with a music segment in the following way. For convenience of description, it is assumed that the corresponding popular music includes an intro, a verse, a chorus, and a bridge music segment. The 'gi', 'win', 'before', and 'texture' of the storyline may be associated with intro, verse, chorus, and bridge music segments.

Music segments of music may be played continuously in the electronic device. Since the corresponding music segment corresponds to the associated 'gi', 'win', 'before', or 'gyeol', 'gi', 'win', 'jeon', or 'gyeol' corresponds to each paragraph of the storyline. , when a certain music segment is reproduced, the electronic device may display an image corresponding to each sentence included in a paragraph corresponding to the corresponding music segment.

All documents, including publications, patent applications, patents, etc., cited in this specification may be incorporated into the present invention in the same way as if each cited document was individually and specifically expressed in combination or as a whole in the present invention. .

For the understanding of the present invention, reference signs have been described in the preferred embodiments shown in the drawings, and specific terms are used to describe the embodiments of the present invention, but the present invention is not limited by the specific terms, and the present invention is not limited thereto. may include all components commonly conceivable by those skilled in the art.

The present invention may be represented in functional block configurations and various processing steps. These functional blocks may be implemented in any number of hardware and/or software configurations that perform specific functions. For example, the present invention provides integrated circuit configurations, such as memory, processing, logic, look-up table, etc., capable of executing various functions by means of the control of one or more microprocessors or other control devices. can be hired Similar to how the components of the present invention may be implemented as software programming or software elements, the present invention includes various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, including C, C++ , may be implemented in a programming or scripting language such as Java, assembler, or the like. Functional aspects may be implemented in an algorithm running on one or more processors. Further, the present invention may employ prior art techniques for electronic configuration, signal processing, and/or data processing, and the like. Terms such as 'mechanism', 'element', 'means', and 'configuration' may be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in connection with a processor or the like.

The specific implementations described in the present invention are only examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings illustratively represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections, physical connections that are replaceable or additional may be referred to as connections, or circuit connections. In addition, unless there is a specific reference such as 'essential' or 'importantly', it may not be a necessary component for the application of the present invention.

In this specification and claims, the use of the term 'above' and similar referent terms may correspond to both the singular and the plural. In addition, in the present specification, the invention to which a 'range' is applied includes an invention to which an individual value belonging to the range is applied, and unless there is a description to the contrary, each individual value constituting the range is described in the detailed description same as done Finally, the steps constituting the method according to the present invention may be performed in an appropriate order unless there is an explicit order or description to the contrary. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and unless limited by the claims, the scope of the present invention is limited by the examples or exemplary terms. it's not going to be In addition, those skilled in the art will recognize that various modifications, combinations, and changes can be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Claims (20)

a memory for storing instructions; and
By executing the above instructions:
obtain music information about the music;
obtaining a storyline of the music based on the music information, wherein the storyline is divided into a plurality of paragraphs based on a word for time, person, or space included in the storyline, and the paragraph is divided into a plurality of paragraphs. contains a plurality of sentences corresponding to keywords - ;
dividing the music into a plurality of music segments corresponding to the plurality of paragraphs into which the storyline is divided based on the words for time, person, or space included in the storyline;
acquiring a plurality of images for each keyword of each sentence of the paragraph corresponding to the music segment, and determining the number of the images corresponding to each keyword, the playing time of the music segment corresponding to each keyword set to be proportional to; and
and at least one processor configured to control a display to display the images corresponding to each keyword while the music segment corresponding to each keyword is played. The method of claim 1,
The music information includes at least one of meta information and audio fingerprint information of the music, and the storyline is obtained based on at least one of the meta information and the audio fingerprint information. The method of claim 1,
A plurality of candidate images are obtained based on the plurality of sentences, and the images for each of the keywords are obtained from the plurality of candidate images. The method of claim 1,
the at least one processor is further configured to execute the instructions to obtain emotional information related to the music based on the storyline,
wherein the plurality of images are obtained based on the emotion information. The method of claim 1,
The plurality of sentences constitute the paragraph based on a degree of similarity. The method of claim 1,
The storyline includes the lyrics of the music,
The music information includes lyrics information about the lyrics,
The plurality of images are displayed based on the lyrics information. The method of claim 1,
the at least one processor is further configured to execute the instructions to obtain sentiment information based on the storyline,
wherein the plurality of images are obtained based on the emotion information. The method of claim 1,
The plurality of images are rendered based on emotion information. The method of claim 1,
The at least one processor executes the instructions to:
obtaining a plurality of candidate images based on the plurality of keywords of the plurality of sentences of the paragraph corresponding to the music segment;
further configured to determine priorities among the plurality of candidate images based on a preset criterion,
The plurality of images are obtained from the plurality of candidate images based on the determined priorities. 10. The method of claim 9,
The predetermined criterion includes at least one of an order among the plurality of candidate images and the number of times the candidate image is repeatedly provided. The method of claim 1,
wherein the storyline comprises a music video, and wherein the plurality of images are obtained from one or more images of frames of the music video. obtaining music information about music;
obtaining a storyline of the music based on the music information, wherein the storyline is divided into a plurality of paragraphs based on a word for time, person, or space included in the storyline, and the paragraph is divided into a plurality of paragraphs - contains a plurality of sentences corresponding to the keywords of ;
dividing the music into a plurality of music segments corresponding to the plurality of paragraphs into which the storyline is divided based on the words for time, person, or space included in the storyline;
acquiring a plurality of images for each keyword of each sentence of the paragraph corresponding to the music segment, and determining the number of the images corresponding to each keyword, the playing time of the music segment corresponding to each keyword setting to be proportional to ; and
and displaying the images corresponding to the respective keywords while the music segments corresponding to the respective keywords are being played. 13. The method of claim 12,
The music information includes at least one of meta information of the music and audio fingerprint information, and the storyline is obtained based on at least one of the meta information and the audio fingerprint information. 13. The method of claim 12,
A plurality of candidate images are obtained based on the plurality of sentences, and the images for each of the keywords are obtained from the plurality of candidate images. 13. The method of claim 12,
Further comprising the step of obtaining emotion information related to the music based on the storyline,
The plurality of images are obtained based on the emotion information, multimedia playback method. 13. The method of claim 12,
The storyline includes the lyrics of the music,
The music information includes lyrics information about the lyrics,
The plurality of images are displayed based on the lyrics information. 13. The method of claim 12,
obtaining a plurality of candidate images based on the plurality of keywords of the plurality of sentences of the paragraph corresponding to the music segment;
Further comprising the step of determining priorities among the plurality of candidate images based on a preset criterion,
The plurality of images are obtained from the plurality of candidate images based on the determined priorities. 18. The method of claim 17,
The predetermined criterion includes at least one of an order among the plurality of candidate images and a number of times the candidate images are repeatedly provided. 13. The method of claim 12,
wherein the storyline comprises a music video, and wherein the plurality of images are obtained from one or more images of frames of the music video. A computer-readable recording medium in which a program for implementing the multimedia reproducing method of any one of claims 12 to 19 is recorded.

Images