JP4369434B2 - Multimedia data processing apparatus, multimedia data processing program, and data structure of multimedia content data - Google Patents

Description

  The present invention relates to a multimedia data processing device, a multimedia data processing program, and a data structure of multimedia content data, and more particularly to a multimedia data processing device, a multimedia data processing program, and a multimedia that can easily create multimedia content. It relates to the data structure of content data.

  With the recent spread of communication devices such as mobile phones, communications that transmit created multimedia contents to other users or cause other users to create multimedia contents have spread.

  However, when creating multimedia content, it is often necessary to take a complicated operation procedure, and it may be difficult for some users who are not used to the operation to create the multimedia content.

Therefore, a method aimed at simplifying the user's operation has been considered.
As such a communication tool, Patent Document 1 is a receiving device that extracts and executes an attached script in conjunction with opening an email when it receives an email attached with a script, and is further executed in the receiving device. A receiving device that controls output of processing results to generate a script reflecting the action and output the script to a mail sender is disclosed. By using such a receiving apparatus described in Patent Document 1, it is possible to cause the sender to receive an action desired by the sender.

Further, in Patent Document 2, in order to easily create an e-mail to which multimedia data is added, an e-mail template describing content layout information is prepared, and an e-mail including multimedia according to the layout information of the template is prepared. The method to create is described.
JP 2002-140270 A JP 2002-324036 A

  However, in the case of creating multimedia contents by executing script generation in the receiving device described in Patent Document 1, it is necessary to describe a complex script and transmit it to the receiving device. is there. Further, since the executed script does not guide the operation on the receiver side, there is a problem that the action desired by the sender side may not always be obtained.

  In addition, there has been a problem that it is not easy for a mail message creator to create a mail. That is, it is difficult to create a message simply by combining a script and data prepared by the message creator.

  As a technique for facilitating creation of messages and mails including multimedia data, a technique disclosed in Patent Document 2 that facilitates arrangement of characters and still images using a template in which layout information is described is disclosed. Although it works effectively when creating a mail with multiple images and text mixed in a single large screen viewed on a PC (Personal Computer), the scenes of A, B, C, etc. In addition, there is no mention of creating a mail in which the content of the mail is switched by discontinuously changing the time of the mail screen, and even if the technique disclosed in Patent Literature 2 is used, the time transitions. There is a problem that it is difficult for a user to create multimedia data.

  Furthermore, there is a problem that the contents of a message cannot be rewritten using a variety of functions such as a camera, a microphone, and an address book that are inherent to the terminal in a conventional data processing device such as a mobile phone or a mobile terminal. is there.

  Furthermore, the creator's terminal, which is a conventional data processing device, rewrites a part of the script used to create the multimedia content, and the script is used on the playback side, so that the script can be rewritten with a minimum. There is also a problem that creation and playback are not possible.

  In addition, on the creator's terminal, which is a conventional data processing device, delete script commands or tags used for creating multimedia content that do not require a sending partner or that do not conform to standard procedures. There was also a problem that it was not possible.

  The present invention has been made in view of such problems, and provides a multimedia data processing apparatus, a multimedia data processing program, and a data structure of multimedia content data that can easily create multimedia contents. As a first object, a multimedia data processing device, a multimedia data processing program, and a data structure of multimedia content data that can allow a partner to reliably create and edit desired multimedia content are provided. This is the second purpose.

In order to achieve the above object, according to one aspect of the present invention, a multimedia data processing device acquires a template including a script from a storage device inside the multimedia data processing device or another device via a network. A template acquisition unit that performs selection, a selection unit that selects a desired template from the acquired templates, a script execution unit that reproduces data referenced from the script according to a script included in the selected template, and a reference from the script. The user changeable data discriminating means for discriminating the changeable portion of the data, the data changing means for changing the changeable portion, the data change confirming means for confirming the data change, and the script. Output at least one of the recorded data And a script data output means.

According to another aspect of the present invention, a multimedia data processing device includes a template acquisition unit that acquires a template including a script from a storage device inside the multimedia data processing device or another device via a network; Selection means for selecting a desired template from the selected templates, script execution means for reproducing data referenced from the script according to the script included in the selected template, and the multimedia according to the script included in the template Function selection means for selecting the function of the data processing apparatus and action determination for determining what processing is to be performed on the data referenced from the script using the selected function according to the script included in the template Means, Output and action execution means for executing the constant has been processed, and the data changing means for changing the data referenced by the script, the data change determining means for determining the change of data, at least one of the scripts and modified data Script data output means.

  The template described above describes a time constraint for determining an action to be performed on data referenced from the script, and the action execution means preferably performs the determined process according to the time constraint.

  In addition, the action determination means uses the selected function to obtain any one of still image data, moving image data, audio data, and position information data acquired by GPS (Global Positioning System) or the like. It is preferable to acquire the data to be included and determine what processing is to be performed on the data referenced from the script using the acquired data.

  Alternatively, the action determination means acquires data via the network using the selected function, and determines what processing is performed on the data referenced from the script using the acquired data. Is preferred.

  Alternatively, the determined process to be executed by the action execution unit is preferably a process for accessing a predetermined WEB page.

  Alternatively, the determined process to be executed by the action execution means is preferably a process for calling a predetermined telephone number.

  Alternatively, the action execution means acquires data using the selected function, and includes environment information including any of the specification and state of the selected function, the template, and the data acquired using the selected function. The conversion process is performed on the acquired data according to the environment information, and the conversion process according to the environmental information reduces the amount of data according to the memory capacity limit and the analysis result of the display device or multimedia data. It is preferable to include conversion of the corresponding screen size, color reduction, or data format matching.

  Alternatively, the action execution means acquires data using the selected function, and includes environment information including any of the specification and state of the selected function, the template, and the data acquired using the selected function. Accordingly, the acquired data is subjected to a conversion process, and the conversion process according to the environment information is preferably a process for changing the acquired data to predetermined alternative data.

  Alternatively, the action execution means obtains data using the selected function, and saves the environment information including any of the specification and state of the selected function, the template, and the data obtained using the selected function. Accordingly, it is preferable to determine whether or not it is necessary to use the acquired data.

  Alternatively, immediately before executing the determined process, the action execution means performs the process after notifying that the process is to be performed via the user interface, accepting confirmation for the process, and performing the process. It is preferable to perform any one of accepting the choice of whether to do.

  Further, transition information of the media data reproduction state is described in the template, and it is preferable that the script execution unit manages the transition of the media data reproduction state.

  Further, the script execution means may perform the transition of the media data playback state when the event determination means acquires the event using the selected function according to the transition information of the media data playback state described in the template. More preferred.

  Further, the event is more preferably a time event including at least one of arrival of a predetermined time and expiration of a timer.

  Also, transition information of media data playback state is described in the template together with time constraints, and it is preferable that the script execution means manages the transition of the media data playback state in terms of time.

  In addition, transition information of media data playback state is described in the template, and the multimedia data processing apparatus generates thumbnail generation means for generating thumbnails of media data included in the template, and the generated thumbnails based on the transition information. It is preferable to further include preview means for displaying the information.

  The multimedia data processing apparatus further includes a template acquisition unit that acquires a template including a script, a selection unit that selects a desired template from the acquired templates, and a data change determination unit that determines a data change. It is preferable.

  Further, it is more preferable that the template acquisition unit acquires a template stored in a storage device inside the multimedia data device.

  Alternatively, it is more preferable that the template acquisition unit acquires a template from another device via a network.

  Alternatively, it is more preferable that the template acquisition unit acquires a template from a message received via a network.

  Moreover, it is preferable that the multimedia data processing apparatus further includes a limiting unit that limits data change according to a script.

  The multimedia data processing apparatus preferably further comprises sequence management means for managing a sequence described in the script and changeable data presenting means for indicating data that can be changed corresponding to the sequence.

  The multimedia data processing apparatus preferably further comprises sequence management means for managing a sequence described in the script and operation guide presenting means for presenting an operation guide for changing data corresponding to the sequence. .

  Furthermore, it is more preferable that the script execution means further includes a reproduction time control means for controlling the reproduction time of the changed data according to a predetermined rule when performing a data change operation.

  Further, the script execution means further includes media time selection means for determining and selecting a place where execution or stop can be performed among data referenced from the script, and the script execution means is selected from the data. When executing a spot, it is preferable to pause at that spot.

  Furthermore, the location selected by the media time selection means is more preferably a location where changeable media data is switched.

  Further, the script execution means further includes a sequence change point determination means for determining a change point of the future sequence closest to the present, and the script execution means skips data reproduction until the next determined sequence change point. Is more preferable.

  Alternatively, the script execution unit further includes a sequence change point determination unit that determines a change point of the past sequence that is closest to the current time, and the script execution unit skips reproduction of data up to the determined past sequence change point. More preferably.

  In addition, when the data change operation is performed, the multimedia data processing apparatus preferably adds the start position or the end position of the changed section of the data to the predetermined position of the data.

  Moreover, it is preferable that a script data output means deletes and outputs at least one part of the script contained in a template, and the data referred from a script.

  Furthermore, at least one of the script included in the template and the data referenced from the script is described in XML (Extensible Markup Language), and the script data output means refers to the script corresponding to the description of a predetermined tag. More preferably, the data to be deleted is deleted.

  The script data output means preferably outputs at least one of the script and the changed data to a storage device inside the multimedia data processing apparatus.

  Moreover, it is preferable that a script data output means outputs at least one of a script and changed data to another apparatus via a network.

  Further, it is more preferable that the script / data output means outputs at least one of the script and the changed data to another device using electronic mail.

  Alternatively, it is more preferable that the script data output means outputs at least one of the script and the changed data to another device using MMS (Multimedia Messaging Service).

  According to still another aspect of the present invention, the multimedia data processing apparatus displays a template list acquisition unit that acquires a list of templates configured to include a script, a template list display unit that displays a template list, and Template selection means for selecting one or more desired templates from the template list, and message sending means for sending the selected template as a message to another apparatus are provided.

  Furthermore, it is preferable that the script includes at least one of reference information to data to be reproduced, a data reproduction method, and action information for specifying a function executed in the data reproduction terminal.

  The action information is at least one of a video or still image imaging function, a function of acquiring text information input by a user, a sound recording function, a function of acquiring position information, and a function of acquiring data. It is preferable to identify one function.

  The multimedia data processing apparatus preferably further comprises script execution means for reproducing data referenced from the script according to the script.

  The multimedia data processing apparatus further includes a communication partner function acquisition unit that acquires a function of the communication partner device, and the template list acquisition unit includes a template list that can be executed with the functions included in the acquired function of the communication partner device. It is preferable to obtain

  Alternatively, the multimedia data processing apparatus further includes a communication partner function acquiring unit that acquires a function of the communication partner device, and the template list display unit is a template list that can be executed with the functions included in the acquired function of the communication partner device. Is preferably displayed.

  According to still another aspect of the present invention, the multimedia data processing apparatus is provided via a template acquisition unit that acquires a template including a script, a program acquisition unit that acquires a program from a broadcasting station, and the Internet. Content acquisition means for acquiring the Internet content to be executed, at least of data referenced from the script according to the script included in the template, the program acquired by the program acquisition means, and the Internet content acquired by the content acquisition means Script execution means for reproducing one.

  Further, the template includes program guide information, and the multimedia data processing apparatus further includes reproduction data selection means for selecting one or more programs or Internet contents based on the program guide information, and the script execution means is selected. It is preferable to play a program or Internet content.

According to still another aspect of the present invention, a multimedia data processing program is a program that causes a computer to execute a template process including a script, and includes a storage device or a network inside the multimedia data processing device. The template acquisition step for acquiring the template from another device, the selection step for selecting the desired template from the acquired template, and the data referenced from the script are reproduced according to the script included in the selected template. A script execution step, a user changeable data determination step for determining a user changeable portion of data referenced from the script, and data for changing the changeable portion by using the function of the multimedia data processing device Further comprising the steps, to execute the data change determination step of determining a change of data, and a script data output step of outputting at least one of the scripts and the changed data.

According to still another aspect of the present invention, a multimedia data processing program obtains a template from a storage device inside the multimedia data processing device or another device via a network, and the obtained template A program for causing a computer to execute a selection step for selecting a desired template from the above and a processing of a template including the selected script, and reproducing data referenced from the script according to the script included in the template A script execution step, a terminal function selection step for selecting a function of the multimedia data processing device in accordance with the script included in the template, and data selected from the script using the selected function. And action determining step of determining whether to perform the processing as the action execution step of executing the determined processing, the data changing step of changing the data referenced from the script, the data change determination step of determining the change of data And a script data output step for outputting at least one of the script and the changed data.

  Further, the template describes a time constraint for determining an action to be performed on data referenced from the script, and the action execution step preferably performs a process determined according to the time constraint.

  Further, transition information of the media data playback state is described in the template, and the script execution step preferably manages transition of the media data playback state.

  Further, the transition information of the media data playback state is described in the template together with the time constraint, and the script execution step preferably manages the transition of the media data playback state in terms of time.

  Also, transition information of media data playback state is described in the template, and a thumbnail generation step for generating a thumbnail of media data included in the template and a preview step for displaying the generated thumbnail based on the transition information are provided. Further, it is preferable to execute it.

  The multimedia data processing program further executes a template acquisition step for acquiring a template including a script, a selection step for selecting a desired template from the acquired templates, and a data change determination step for determining data change. It is preferable to make it.

  Further, it is preferable that the multimedia data processing program further executes a restriction step for restricting data change according to the script.

  The multimedia data processing program preferably further executes a sequence management step for managing a sequence described in the script and a changeable data presentation step for presenting data that can be changed in accordance with the sequence.

  The multimedia data processing program may further execute a sequence management step for managing a sequence described in the script and an operation guide presenting step for presenting an operation guide for changing data corresponding to the sequence. preferable.

  Furthermore, when performing a data change operation, it is more preferable that the script execution step further executes a reproduction time control step for controlling the reproduction time of the changed data according to a predetermined rule.

  The script execution step further executes a media time selection step for determining and selecting a portion that can be executed or stopped from the data referenced from the script, and the script execution step is a selection of the data. It is preferable to pause at the above-mentioned place when executing the given place.

  Furthermore, it is more preferable that the location selected in the media time selection step is a location where changeable media data is switched.

  Furthermore, the script execution step further executes a sequence change selection step for determining a change point in the future sequence closest to the present, and the script execution step skips data reproduction until the next determined sequence change point. Is more preferable.

  Alternatively, the script execution step further executes a sequence change selection step for determining a past sequence change point closest to the present time, and the script execution step skips the reproduction of data up to the latest past sequence change point determined. More preferably.

  In addition, when the data change operation is performed, the multimedia data processing program preferably adds the start position or the end position of the changed section of the data to the predetermined position of the data.

  According to still another aspect of the present invention, the data structure of multimedia content data is multimedia content processed in a data processing apparatus including playback means for playing back media data and input means for accepting an input operation from a user. An input operation description indicating a data structure of data, a reproduction description portion indicating media data reproduced by reproduction means of the data processing device, an input operation received by the input means of the data processing device, and a process corresponding to the input operation The multimedia content data is processed, the reproduction means of the data processing device reproduces the predetermined media data, the input means of the data processing device accepts the predetermined input operation, and the input operation description portion Based on this, the data processing apparatus is caused to execute processing according to a predetermined input operation.

  According to still another aspect of the present invention, the data structure of multimedia content data is multimedia content processed in a data processing apparatus including playback means for playing back media data and input means for accepting an input operation from a user. An input operation description indicating a data structure of data, a reproduction description portion indicating media data reproduced by reproduction means of the data processing device, an input operation received by the input means of the data processing device, and a process corresponding to the input operation The multimedia content data is processed by the processing unit, the schedule description unit for managing the valid time of the media data reproduced by the reproducing means of the data processing device and the valid time of the input operation received by the input means. Data processing at a predetermined time based on the schedule description part The predetermined media data is reproduced by the reproduction means of the device, the predetermined input operation is received by the input means of the data processing device at the predetermined time, and the data processing device is determined at the predetermined time by the input operation description part. The process according to the input operation is executed.

  Moreover, it is preferable that the process corresponding to the input operation accepted by the input unit of the data processing apparatus is a process of changing the multimedia content data.

  Furthermore, the change process applied to the multimedia content data is more preferably a process for replacing a part of the multimedia content data.

  Alternatively, the change process added to the multimedia content data is more preferably a process of adding data obtained by replacing a part of the multimedia content data to the content data.

  Alternatively, the change process added to the multimedia content data is more preferably a process of adding data obtained by replacing a part of the multimedia content data in the subsequent predetermined process to the multimedia content data.

  The multimedia content data is preferably data written in a script language.

  Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same.

[First Embodiment]
FIG. 1 is a diagram showing a configuration of a data communication system including a multimedia data processing apparatus according to the first embodiment. Referring to FIG. 1, the data communication system according to the present embodiment includes a template providing server (hereinafter simply referred to as a server) 2 that provides a template described in a script language, and reproduction and editing of the provided template. The server 2 and the playback / editing apparatus 1 communicate with each other via the Internet, a public line network, or a broadcasting network. Although not shown in FIG. 1, the server 2 and the playback / editing apparatus 1 may communicate with each other via a dedicated line such as a LAN (Local Area Network) or may perform wireless communication. . As shown in FIG. 1, the playback / editing apparatus 1 corresponds to a mobile phone, a PDA (Personal Digital Assistants), a personal computer, and the like. In the following description, the playback / editing apparatus 1 is the mobile phone 1. It will be described as a thing.

  Note that the playback / editing apparatus 1 that is a multimedia data processing apparatus includes information communication equipment, video / audio equipment, home appliances, amusement equipment, and the like.

  As information communication devices, information devices such as personal computers and home servers, and communication devices such as telephones, especially portable information communication devices such as mobile phones, PDAs (Personal Digital Assistants), notebook computers, and wearable computers. Is applicable.

  Also, as video / audio equipment, digital still cameras as image recording equipment, digital video cameras and HDD (Hard Disk Drive) recorders as video recording equipment, DVD (Digital Versatile Disk) recorders, video tape recorders, video playback equipment DVD player and video player, voice recorder as voice recording device, silicon audio recorder or player as music recording / playback device, MD (Mini Disk) recorder or player, CD (Compact Disk) recorder or player, broadcast data reception For example, a television receiver or a radio, which is a playback device, is applicable.

  In addition, examples of home appliances include refrigerators, washing machines, air conditioners, vacuum cleaners, air purifiers, and microwave ovens.

  Further, the amusement device corresponds to a home use or a portable game device.

  The reproduction / editing apparatus 1 that is a multimedia data processing apparatus includes the information processing, communication, image recording / reproduction, video recording / reproduction, audio recording / reproduction, music recording / reproduction, and broadcast data reception / reproduction. A device having a plurality of functions may be used, and any device other than those listed above may be used as long as the device can reproduce or edit multimedia data.

  In the present embodiment, the case where the playback / editing apparatus 1 is the mobile phone 1 will be described in detail.

  Note that in this embodiment, media data refers to any of image (still image) data, moving image data, audio data, music data, text data, and animation data, streaming data or broadcast data received from the outside, or It refers to the following multimedia data obtained by integrating them.

  In the present embodiment, multimedia data refers to data obtained by integrating one or more media data, and the data format may be described in a human-readable text format or can be directly read by a human. There may be no binary format.

  In the present embodiment, the script refers to the layout information on the screen for each media data included in the multimedia data and the start of playback on the time axis in order to display, play, and edit the above-described multimedia data. This refers to a description format that defines attributes such as end timing, playback speed, and volume. Also, control information such as activation of an external application, acquisition of information from inside and outside the apparatus, and link transition to other multimedia data can be described in the script.

  Further, in the present embodiment, the template is multimedia data that includes the above-described script and can be played back or edited, and refers to the media data in the script triggered by a user operation or the like. Is a form of multimedia data that can add, change, and delete media data. The template may include one or more media data referenced from the script.

  Next, FIG. 2 shows a specific example of the hardware configuration of the mobile phone 1 which is a playback / editing device included in the data communication system.

  Referring to FIG. 2, mobile phone 1 included in the data communication system of the present embodiment includes an input / output unit 140 that is an interface with a user, a CPU (Central Processing Unit), and the like. 1 includes a control unit 120 that controls the entire system 1, a communication unit 110 that communicates with other devices, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and is executed by the control unit 120. And a storage unit 130 for storing the program, intermediate data of the program, data received from another computer, and the like.

  Further, the input / output unit 140 includes a key code input device 142 including numeric buttons such as “1” and “2” and direction buttons such as “R” and “L”, and an LCD ( A display 144 such as a liquid crystal display), a microphone 146 that inputs sound, a speaker 148 that outputs sound, and a camera 149 that captures and inputs an image.

The control unit 120 includes a timer therein.
In such a mobile phone 1, the processing for reproducing and editing the template obtained by the user from the server 2 is realized by the hardware of the mobile phone 1, the software and the template stored in the storage unit 130 and executed by the control unit 120. Is done. Since the operation of the mobile phone 1 itself shown in FIG. 2 is well known, detailed description thereof is omitted here.

  Further, the server 2 included in the data communication system is a server constructed by a general personal computer or the like, and detailed description thereof will not be given here.

  Next, FIG. 3 shows a program stored in the storage unit 130 of the mobile phone 1 and a flow of data processing in the mobile phone 1. The data processing flow shown in FIG. 3 is realized in the mobile phone 1 by the control unit 120 executing each program stored in the storage unit 130. That is, the template received from the server 2 by the communication unit 110 is played and edited by the control unit 120 executing a program stored in the storage unit 130.

  Specifically, referring to FIG. 3, first, control unit 120 executes a playback / editing engine program 131 to process a template including a script and media data received by communication unit 110. The functional configuration of the playback / editing engine program 131 here will be described later.

  Next, the control unit 120 causes a UI (User Interface) content display program 132 to be executed based on the processing result of the playback / editing engine program 131. By executing the UI content display program 132 to process the template, the UI content in the template is displayed on the display 144.

  Further, the control unit 120 executes the image capturing program 133 based on the processing result in the reproduction / editing engine program 131 and receives an image input from the camera 149 and various instructions from the key code input device 142. In addition, when shooting a moving image, an audio input from the microphone 146 is also accepted.

  Further, the control unit 120 executes the image data display / reproduction program 134 based on the processing result in the reproduction / editing engine program 131 and reproduces the image data included in the template on the display 144. If the image data includes sound, the sound is played back by the speaker 148.

  Further, the control unit 120 executes the audio data reproduction program 135 based on the processing result in the reproduction / editing engine program 131 and causes the speaker 148 to reproduce the audio data included in the template.

  Further, the control unit 120 executes the audio data recording program 136 based on the processing result in the reproduction / editing engine program 131, and receives the audio input from the microphone 146 and various instructions from the key code input device 142. .

  Further, the control unit 120 executes the animation data display / reproduction program 137 based on the processing result of the reproduction / editing engine program 131 and reproduces the animation data included in the template on the display 144.

  Further, the control unit 120 executes the text data display / reproduction program 138 based on the processing result in the reproduction / editing engine program 131 and reproduces the text data included in the template on the display 144.

  Further, the control unit 120 executes the text editing program 139 based on the processing result in the reproduction / editing engine program 131, and accepts text information and various instructions from the key code input device 142.

  Further, the control unit 120 executes a WWW (World Wide Web) browser program 151 based on the processing result of the playback / editing engine program 131 to play back the WWW content.

  Further, the control unit 120 executes the WWW browser program 151 based on the processing result in the playback / editing engine program 131 and acquires data from the server 2 via the communication unit 110.

  Further, the control unit 120 executes the mail program 152 based on the processing result in the reproduction / editing engine program 131, receives text information and various instructions from the key code input device 142, and transmits mail.

  Further, the control unit 120 executes the call program 153 based on the processing result in the reproduction / editing engine program 131 to start a call.

  Further, the control unit 120 executes the position acquisition program 154 based on the processing result in the reproduction / editing engine program 131 and acquires the current position of the terminal from the GPS (Global Positioning System) 150.

  Further, the control unit 120 executes the address book program 155 based on the processing result in the reproduction / editing engine program 131 and acquires the mail address and telephone number stored in the storage unit 130.

  Further, the control unit 120 executes the file acquisition program 156 based on the processing result of the playback / editing engine program 131 and acquires content data of a specific media type stored in the storage unit 130.

  Further, the control unit 120 executes the above-described image shooting program 132, audio data recording program 136, text editing program 139, WWW browser program 151, position acquisition program 154, address book program 155, and file acquisition program 156. Based on the received information, the playback / editing engine program 131 is executed to process a template including a script and media data.

  Furthermore, the functional configuration of the above-described playback / editing engine program 131 is shown in FIG. The control unit 120 executes the playback / editing engine program 131 stored in the storage unit 130, whereby each function shown in FIG.

  Referring to FIG. 4, first, script reading unit 1311 reads a script included in the template received by communication unit 110. Then, the script reading unit 1311 passes the read script to the syntax analysis unit 1312.

  Next, the syntax analysis unit 1312 analyzes the syntax of the script passed from the script reading unit 1311. Then, the syntax analysis unit 1312 passes the analysis result to the execution control unit 1313 and the time management unit 1314.

  Next, the time management unit 1314 creates a time table based on the analysis result passed from the syntax analysis unit 1312. Then, the time management unit 1314 passes the created time table to the execution control unit 1313.

  In addition, the event management unit 1315 detects the occurrence of an event. Then, the event management unit 1315 passes the detection result to the execution control unit 1313.

  Next, the execution control unit 1313 is based on the time table passed from the time management unit 1314, the detection result of event occurrence passed from the event management unit 1315, and the analysis result passed from the syntax analysis unit 1312. Run the script. Further, the execution control unit 1313 instructs the script writing unit 1316 to rewrite the script based on the detection result of the event occurrence passed from the event management unit 1315.

  The script writing unit 1316 rewrites the script based on an instruction from the execution control unit 1313, and passes the rewritten script to the execution control unit 131. Then, the rewritten script is executed by the execution control unit 1313.

  In the present embodiment, the case where the mobile phone 1 processes the template provided from the server 2 will be described. However, some or all of the script and media data included in the template are stored in the storage unit 130 of the mobile phone 1. In this case, each unit shown in FIG. 3 or FIG. 4 processes a script or media data read out and acquired from the storage unit 130.

  Next, a script included in a template provided from the server 2 to the mobile phone 1 and processed by the mobile phone 1 will be described.

  FIG. 5 is a diagram schematically showing the data structure of the script. Referring to FIG. 5, the script processed by mobile phone 1 in the present embodiment is characterized by comprising a drawing information description portion and a user navigation information description portion.

  In the drawing information description part, reference information to media data such as image (still image) data, moving image data, audio data, music data, text data, animation data, streaming data, broadcast data, etc. File name, URL (Uniform Resource Locators), broadcast station frequency, channel, etc.) are described.

  It should be noted that by using a description format such as a data URL defined in RFC2397, the entity of the media data itself may be described in the drawing information description part instead of the reference information to the media data.

  In the user navigation information description portion, a script definition input event for executing link movement, application activation, help display, and the like, an action (operation), and an execution condition are described.

  Examples of the above actions include capturing a video or still image by camera activation, acquiring user input text data by starting a text editor, acquiring voice data by starting a voice recorder, acquiring a media data file by starting a file selection application, Automatic acquisition of time information, location information, address book, schedule information, and other information from inside or outside, activation of device applications such as WWW browsers and mailers, device-specific functions such as calling, melody playback, and lighting Calling or link transition to execution of another script, transmission of a control signal for controlling other devices such as TV channel switching, air conditioner temperature setting, and the like.

  Furthermore, both the drawing information description part and the user navigation information description part include a time-series description. That is, the reference information to the media data is described in the drawing information description part in time series, and the event is described in the user navigation information description part in time series.

  With this configuration, the script contains media data such as image (still image) data, music data, text data, animation data, streaming data, and broadcast data one after another in time series. Multimedia data that can be played back can be described.

  Furthermore, since events can be described in time series, events such as camera activation can be executed at a predetermined time or time zone while playing back a script. Further, by describing such a camera activation event so as to be associated with an action for replacing a predetermined image, when the user presses a predetermined key during a time zone in which the predetermined image is to be displayed, When the reproduction is temporarily stopped and the camera is activated immediately, and the photographing operation is executed here, the application can be realized in which the reproduction of the script is resumed and the currently photographed image is displayed at a predetermined position.

  As described above, events that are activated in time series and the corresponding actions are explicitly defined in the script, so that multiple functions can be used using functions such as cameras, microphones, and address books on mobile devices. Even when creating multimedia data in which the media data is reproduced in time series, the operation procedure can be simplified and the convenience for the user is enhanced.

A specific script will be described with reference to FIGS.
With reference to FIG. 6, first, the content of the action is defined in the 12th to 15th lines of the header portion (2nd to 16th lines) of the script. That is, referring to FIG. 6, the 12th to 15th lines include, as action contents, unique identification information (action id) for each action, an execution format (type) of the action, and action contents ( dest) and an action execution area (Image) are defined.

  More specifically, in the twelfth line of FIG. 6, the action that is the identification information act_movie replaces the information displayed in the Image area and the file stored in the movie (replace). The meaning is defined.

  In the fifth line of FIG. 6, a hint about how to use the script in the mobile phone 1 is described. That is, it is defined that the script is a template for navigating the content editing operation. In addition, it is possible to define that the content is content only for viewing without navigating the editing operation.

  Next, referring to FIG. 7, the paragraphs in the 18th to 39th lines of the main body (the 17th line to the 40th line) of the script describe media data and actions to be executed in parallel. Has been. Further, the 19th and 37th lines describe that these media data and actions are executed in time series. Furthermore, the paragraphs of the 20th to 25th lines, the 26th to 31st lines, and the 32nd to 36th lines respectively include 0 second to 8 seconds, 8 seconds to 15 seconds, and 15 seconds. Media data and actions executed in seconds to 19 seconds are described.

  More specifically, lines 21 to 23 in FIG. 7 describe that image data, text data, and audio data are reproduced at times 0 to 8 seconds, respectively. The row describes that the “movie activation” UI is displayed and an action of identification information act_movie is executed when the select key is operated. The 30th line describes that the “voice recording” UI is displayed, a notification is received from the timer two seconds later, and the action of the identification information act_speech is executed. Note that in the 28th to 30th lines, the name described after “src =” is the source ID representing the event generation source. The UI display may be described in a script as shown in FIG. 7, or a table that prescribes the correspondence between the activated application and the UI displayed at that time as shown in FIG. The table may be stored in the storage unit 130 of the mobile phone 1 and automatically read and displayed when the control unit 120 of the mobile phone 1 executes the reproduction / editing engine program 131.

  As described above, the script processed by the mobile phone 1 according to the present embodiment includes a drawing information description portion that describes drawing information and user navigation information, a user navigation information description portion that are executed in time series, It is comprised from these.

  In the present embodiment, a case where a human-readable script in a text format is processed by the mobile phone 1 will be described, but the script may be in a binary format that cannot be directly read by a human.

  In this embodiment, a case where a batch of scripts is processed by the mobile phone 1 will be described. However, the script is divided into a drawing information description section including reference information to media data and a user navigation information description section. When the media data is replaced or inserted, only the script in which the drawing information description part including the reference information to the media data is described may be edited.

  Specifically, in FIG. 70 and FIG. 71, each script divided into a drawing information description part and a user navigation information description part, the script in which the drawing information description part is described, and user navigation The script in which the information description part is described is shown.

  Specifically, the script in which the drawing information description part shown in FIG. 70 is described includes a still image top. jpg and text top. txt is displayed, and the still image image. jpg, music data bgm. amr, text message. It is described that txt is reproduced.

  Also, the script in which the user navigation information description part shown in FIG. 71 is described is image.of media data in which reference information is described as a reference destination in FIG. jpg and message. This is a script that realizes the function of replacing and inserting txt.

  FIG. 72 shows a specific example of screen display when the script shown in FIG. 70 and FIG. 71 described above is parsed and executed by the mobile phone 1 according to the present embodiment. First, as shown in FIG. 72 (A), according to the description of FIG. 71, an image file selection form is displayed together with a character string of “SELECT PHOTO”, and a character string input form is displayed together with a character string of “ENTER BIRTHDAY MESSAGE”. The FIG. 72 (B) shows a situation where the user has selected an image file and entered text on the screen shown in FIG. 72 (A). When the “send” button is selected and executed on the screen shown in FIG. 72B, the currently selected image file is image. As jpg, the input text is message. It is incorporated into the template as txt and played back. That is, during the period from 0 second to 2 seconds from the start of playback of the script, as shown in FIG. jpg and text top. txt is displayed, and then between the start 2 seconds and 8 seconds, image.txt is displayed as shown in FIG. jpg and message. txt is displayed and the music data bgm. amr is played back. The image.image displayed in FIG. jpg is an image selected by the user on the screen shown in FIG. txt is the text entered on the screen shown in FIG.

  In this way, by dividing the script into the drawing information description part and the user navigation description part, and deleting the user navigation description part from the script and sending it attached to an email as described later In addition, it is sufficient to attach only the drawing information description part, and there is an advantage that the process of analyzing the script and considering which part should be deleted can be omitted.

  The relationship between the script including the drawing information description part and the script including the user navigation description part is such that, for example, a script including a description part linking both scripts, such as the script shown in FIG. 73, is prepared. It is preferable to treat a set of a script and a media data group referenced from a script including a drawing information description part as a template.

  In addition, by describing a link to the script including the corresponding user navigation description part in the script including the drawing information description part, the user navigation can be performed at a predetermined timing while reproducing the script including the drawing information description part. It is also possible to move to the execution of a script including a description portion and shift to an input screen as shown in FIG.

  Next, script playback and editing processing in the mobile phone 1 according to the present embodiment will be described with reference to the flowchart of FIG. The processing shown in the flowchart of FIG. 9 is realized by reading and executing the playback / editing engine program 131 shown in FIG. 4 from the storage unit 130 by the control unit 120 of the mobile phone 1.

  Referring to FIG. 9, first, after initialization is performed in step S <b> 11, in step S <b> 12, the communication unit 110 acquires a script from the server 2 via the Internet.

  Next, in step S <b> 13, the control unit 120 executes the playback / editing engine program 131 and reads the script acquired by the script reading unit 1311.

  In step S14, the syntax analysis unit 1312 performs syntax analysis on the read script. Note that the syntax analysis method here is not limited in the present invention, and may be a general syntax analysis method.

  Next, in step S15, the time management unit 1314 creates a time table as shown in FIGS. 10 to 12 based on the syntax analysis result. FIG. 10 is a diagram showing a specific example of a time table for media drawing processing created by the time management unit 1314 based on the drawing information description portion of the script shown in FIGS. 6 and 7, and FIGS. FIG. 8 is a specific example of an event processing time table and a UI allocation status display time table created based on the user navigation information description portion of the script shown in FIG. 6 and FIG. That is, in step S15, the time management unit 1314 responds to the correspondence between the script execution time and the media data reproduced at that time, the script execution time, and the event generated at that time, based on the script analyzed in step S14. The correspondence between the processing and the correspondence between the script execution time and the allocation of the UI operated at that time is created in a time table format as shown in FIGS.

  Specifically, the time table of the media drawing process shown in FIG. 10 is based on the drawing information description part of the script shown in FIGS. FirstImage.mpg), text 1 (FirstText.txt), and music 1 (FirstSound.smf) are played back, and text 2 (SecondText.txt) is played back for 8 to 15 seconds from the start of script execution. A time table indicating that still image 3 (ThirdImage.jpg), text 3 (ThirdText.txt), and music 3 (ThirdSound.smf) are played for 15 to 19 seconds from the start of script execution. .

In addition, when the text included in the script or the text input by the user is displayed in the same area as other images and videos, attributes such as color and font are set in order to improve the visibility of the text. It is preferable to adjust automatically. The simplest implementation of such automatic adjustment is that when a specific color attribute expression is a RGB ternary set, the color attribute distance √ ((ΔR) ² + (ΔG) ² + When (ΔB) ² ) is smaller than a predetermined threshold value, the complementary color of the color attribute of the background color (the color having the one's complement of each RGB component value of the background color as an RGB component) is set as the text color attribute. It is obtained by doing.

  Further, the event processing time table shown in FIG. 11 is based on the user navigation information description portion of the script shown in FIGS. 6 and 7, and when the Select key is operated during 0 second to 8 seconds of script execution start. When the moving image shooting application that is the image shooting program 133 is activated and the Select key is operated within 8 to 15 seconds from the start of script execution, for example, the moving image shooting application that is the image shooting program 133 and the text editing program 139 A screen for selecting one of the character input applications is displayed on the display 144, and the selected program is started, and the voice data recording program 136 is recorded by a timer between 8 to 15 seconds from the start of script execution. To launch the input application It is to time table.

  The UI allocation status display time table shown in FIG. 12 is based on the user navigation information description portion of the script shown in FIGS. 6 and 7, and the Select key is set between 0 second and 8 seconds from the start of script execution. In order to indicate to the user that the moving image shooting application, which is the image program 133, is started when operated, “movie shooting” is displayed on the display 144 as an explanation of the Select key, and the script execution starts between 8 seconds and 15 seconds. The Select key is displayed on the display 144 to indicate to the user that when the Select key is operated, either the moving image shooting application as the image shooting program 133 or the character input application as the text editing program 139 is activated. “Camera shooting” and “ In order to indicate to the user that the voice input application, which is the voice data recording program 136, is started by the timer between 8 seconds and 15 seconds from the start of script execution, the display 144 is started by the timer. 4 is a time table indicating that “recording” is displayed as an explanation of

  Next, in step S16, the execution control unit 1313 executes the script based on the syntax analysis result and the time table. At the time of script execution, the execution control unit 1313 instructs the UI content display program 132 to display a UI that changes with the script execution time on the display 144 based on the UI allocation status display time table. The script execution processing here will be described later with reference to specific examples based on the scripts shown in FIGS.

  In step S17, when event management unit 1315 detects the occurrence of an event (YES in S17), in step S18, execution control unit 1313 executes processing corresponding to the event. The processing according to the event here will be described later with a subroutine.

  The events detected in step S17 described above are mainly fixed, such as a system event such as an end arrival event that reaches the end of the execution point when script execution is executed in time series, and a menu display instruction. There are three types of events: fixed input events for executing general processing and script definition input events for instructing link movement, application activation, and help display. Specifically, the fixed input event and the script definition input event correspond to a user instruction input from the key code input device 142 during script execution. For this reason, during the execution of the script, the execution of the script is continued until some event such as an instruction input occurs from the key code input device 142.

This completes the script playback and editing process on the mobile phone 1.
Note that if the above playback and editing process is interrupted by a call program, etc. in the middle of the process, or is interrupted due to user convenience or battery consumption, the playback and editing point at that time and the script being changed Are stored in the storage unit 130 and the process is interrupted, and when the process is resumed later, the playback and editing time points saved from the storage unit 130 and the script being changed are read, and from the playback and editing time points before the interruption, It is preferable to be configured to be able to return.

  Further, reproduction and editing processing when the script is described in SMIL (Synchronized Multimedia Integration Language) will be specifically described with reference to the flowchart of FIG.

  Referring to FIG. 79, first, in mobile phone 1, a script described in SMIL included in a template is parsed and stored in a data structure (S2401).

  A specific example of the data structure storing the analysis result in step S2401 is shown in FIG. As shown in FIG. 80 (B), in step S2401, a DisplayContent class that holds attribute values such as media type, file name, and display area for each piece of media data that is referred to by reference information included in the script. Instances are created, and a set of instances is integrated and managed in the Display Scheduler shown in FIG.

  Further, FIG. 81 shows the analysis result of the script in step S2401. FIG. 81 specifically shows the analysis results of the scripts shown in FIGS. 6 and 7, and 11 instances of the DisplayContent class are created as the script analysis results. This number is the number of media data referred to by the reference information finally described in the script.

  Of the 11 instances, the media data file designated for the instances 5, 6 and 7 is empty, and in the initial state shown in FIGS. 6 and 7, the corresponding media data is referred to. This indicates that it has not been performed. Reference information is described by accepting designation of media data in an editing process described later, and the media data is referred to. Alternatively, guidance display can be performed by designating a predetermined file. For example, taking instance 5 as an example, since it has been analyzed that the action content is imaging and insertion by camera activation, the camera illustration image can be guided by specifying the camera illustration image (Camera.jpg). Is displayed. More simply, a character string that is the value of EvDesc (such as “camera activation”) may be displayed as a guidance display on the bottom line of the screen.

  Then, the script is reproduced while following the data structure as shown in FIG. 80 (S2402).

  When the media data insertion or replacement process described below is performed (YES in S2403), the corresponding value is updated in the data structure storing the analysis result in step S2401 (S2404). Rewriting processing such as media data insertion and replacement processing will be described in detail later.

  A specific example of the analysis result of the script updated in step S2404 is shown in FIG. The editing result shown in FIG. 82 is a result of rewriting processing performed on the scripts shown in FIGS. 6 and 7 in the initial state shown in FIG. More specifically, the case where the above-described rewriting process is executed and media data “CameraPict01.jpg”, “EditText01.txt”, and “Voice01.amr” are added to the script in the initial state is shown. Reference information is described in the corresponding instances 5, 6, and 7. As for the image data “FirstImage.mpg”, since the file name is defined on the script side, the movie file obtained as a result of movie shooting is also held with this name. In other words, the file entity is overwritten and changed without rewriting the file name.

  The processes in steps S2402 to S2404 described above are repeatedly executed until the end point of the script execution point is reached (NO in S2405). When the end point is reached (YES in S2405), the script reproduction process is terminated and mail is sent. It is determined whether or not to transmit with attachment (S2406).

  If script editing such as media data insertion or replacement processing has not been completed (NO in step S2406), the process returns to step S2402, and the script is played from the beginning of the execution point.

  On the other hand, when the editing is completed and the mail is attached and transmitted (YES in S2406), a script described in SMIL is generated from the data structure shown in FIG. 82 (S2407). Then, the script described in SMIL generated in step S2407 is attached to the mail and transmitted or stored (S2408). Note that the mail attachment process in step S2408 will also be described later.

  A specific example of the script transmitted in step S2408 is shown in FIG. The script shown in FIG. 83 is a script that has undergone a rewrite process for adding media data that is image data, text data, and audio data to the script in the initial state shown in FIG. 6 and FIG. Specifically, the part indicated by the underline in FIG. 83 is the reference information of the media data inserted by the rewriting process.

  Further, when the script is transmitted, the editing function which is the user navigation information description part may be deleted and transmitted. A specific example of the script from which the editing function has been deleted is shown in FIG. The script shown in FIG. 84 performs an editing function after rewriting processing for adding media data as image data, text data, and audio data to the script in the initial state shown in FIGS. More specifically, the correspondence between the action part defined by <action id = ...> and the event and action described by <event ref = ...> The attached part has been deleted. It is shown underlined in FIG.

  Note that when the script described in FIG. 84 is generated by writing the script described in SMIL from the data structure shown in FIG. 82 in step S2407, the event and action included in the data structure are deleted when the editing function is deleted. Ignore attributes related to. Or, when writing a script including an editing function, “insert” is specified as the action content (Act Type = “insert”), but the media data reference information has already been described by the above processing. Therefore, the action content is changed to “replace” (Act Type = “replace”). It is shown underlined in FIG.

  Next, processing according to the event executed in step S18 will be described with reference to the flowchart of FIG.

  Referring to FIG. 13, when event occurrence detection information is received from event management unit 1315 in step S17 described above, execution control unit 1312 temporarily stops the script being executed in step S21.

  If the event detected in step S17 is an event indicating that the execution of the script has reached the end of the execution point (YES in S22), the execution control unit 1313 executes the script in step S23. Is finished, the process is returned to step S16 in FIG. 9, and the execution of the script is started again. That is, the script execution is returned to the start point of the execution point again, and the script execution is repeated from the beginning. In the present embodiment, the script included in the acquired template is repeatedly executed. However, the script may be terminated by executing it once or a predetermined number of times. Alternatively, an option of repeating or ending script execution may be presented on the screen, and a repetition or termination of script execution may be instructed by a user input such as a button operation.

  If the event detected in step S17 is an event other than an event indicating that the execution of the script has reached the end of the execution point (NO in S22), the execution control unit 1313 is detected in step S24. The processing content is determined based on the event, and the determined processing is executed in steps S25 to S27. The processing content determination processing in step S24 and each processing shown in steps S25 to S27 will be described later with reference to flowcharts.

  Thus, the process according to the event in step S18 is terminated, and the process returns to the reproduction / editing process shown in the flowchart of FIG.

  Next, the process content determination process in step S24 will be described with reference to the flowchart of FIG.

  Referring to FIG. 14, when the occurrence of an event is detected in step S17 described above, execution control unit 1313 determines the time parameter value (S31) at that time, that is, the parameter representing the script execution time and the source ID (S32). That is, the event occurrence source is determined with reference to the identification information of the event occurrence source and the event occurrence processing time table (S33) created by the time management unit 1314. Furthermore, from the determined event and the event occurrence processing time table, the processing content associated with the event when the event occurs is determined. When there are a plurality of determined processes (YES in S34), in step S35, the execution control unit 1313 displays a selection screen on the display 144 or the like, and from among the plurality of processes, Accept selection of process starting from. For example, as shown in FIG. 54, “Invoke Camera” and “Local Folder” are presented in the list displayed on the selection screen, and when “Invoke Camera” is selected, the camera is started, When “Local Folder” is selected, the data folder is accessed.

  Alternatively, in step S35, the execution control unit 1313 may execute a plurality of determined processes one by one in order. In that case, the process is executed in the order in which the schedule description of the event corresponding to the process appears. More specifically, taking the example between the 8th and 15th seconds of the script of FIG. 7, when the select key is pressed, “camera activation” described in line 28 according to the description order is First, it is executed, and then “text input” described in line 29 is executed. Note that a configuration may be employed in which the execution priority order of a plurality of processes is described in a script, and the plurality of processes are executed in that order.

  Alternatively, when there is a shortcut means for executing each of the plurality of processes in step S35, a configuration corresponding to the shortcut means designated by the user may be directly executed. More specifically, taking the example between the 8th and 15th seconds of the script of FIG. 7, “camera activation” described in line 28 is executed when the camera activation key is pressed, When the text input key is pressed, “text input” described in line 29 may be executed.

  Thus, the process corresponding to the event in step S24 is terminated, and the process is returned to the event process shown in the flowchart of FIG.

  Furthermore, referring to the event time table in step S33 described above will be described with reference to the flowchart of FIG.

  Referring to FIG. 27, when the referenced source ID is “Timer” (“Source ID” = Timer in S131), the corresponding processing is performed with reference to the Timer column and the time parameter value of the event processing time table. Judgment is made (S132).

  On the other hand, when the referenced source ID is “UI” (“source ID” = UI in S131), the UI column of the event processing time table, the type ID, that is, the identification information specific to the key operated when the event occurs, The time parameter value is referred to (S133), and the corresponding process is determined (S134).

  As the identification information unique to the key described in the event processing time table and the script, the identification information of the hardware key of the terminal may be used directly, or a logical symbol is used as shown in FIG. A table defining the correspondence between symbols and hardware keys is stored in the storage unit 130 of the mobile phone 1 in advance, and when the execution control unit 1313 refers to the event time table, the table is read and the operated hardware key The event may be determined by referring to the logical symbol.

  The event time table reference in step S33 is ended as described above, and the process returns to the process content determination process shown in the flowchart of FIG.

  If it is determined in the above process that the event detected in step S17 is an event for executing the menu process, the process shown in FIG. 15 is executed in step S25. That is, referring to FIG. 15, first, in step S41, execution control unit 1313 displays a menu screen as shown in FIG. 16 on display 144, and accepts a selection from the user. Specifically, referring to FIG. 16, preview processing of the script, storage processing in the data folder, processing of storing media data referenced from the script in an individual media file, the script and the media data in an email Processing to attach, processing to send the script and media data by MMS (Multimedia Messaging Service), processing to edit the slide during playback of the script, processing to display the properties of the script, processing to terminate the application, etc. Accept selection. In step S42, processing corresponding to the selection from the user is executed. A slide represents a part of a script executed in parallel as described in lines 20 to 25 in FIG. That is, the script shown in FIG. 7 includes three slides of 20 to 25 lines, 26 to 31 lines, and 32 to 36 lines that are sequentially reproduced.

  Further, when “slide editing” is selected on the menu screen as shown in FIG. 16, the execution control unit 1313 displays a menu screen for editing the slide as shown in FIG. Then, the selection of the slide deletion process or the slide rewrite prohibition process is accepted.

  Note that the script may describe whether slide editing is permitted or prohibited, and if it is prohibited, the slide editing option may not be presented and the slide editing may be disabled. it can.

  More specifically, FIG. 68 shows an example of a script that describes whether such slide editing is permitted or prohibited, and the flowchart of FIG. 69 shows processing for determining permission to execute slide editing in step S42. Will be described. That is, referring to FIG. 69, when a meta description (name = "readOnly") of prohibition of editing as shown in line 4 of FIG. 68 is described in the script (YES in S2201), the attribute describing the prohibition Refer to (content). If the attribute is “yes” (YES in S2202), the editing of the slide in step S42 is prohibited. On the other hand, if the attribute is not “yes” (NO in S2202), the editing of the slide in step S42 is permitted. In the first place, even if the script does not have a meta description that prohibits editing (NO in S2201), the editing of the slide in step S42 is permitted.

This completes the description of the slide edit permission determination based on the specific example.
The process in step S42 when the slide deletion is selected on the slide editing menu screen as shown in FIG. 36 in step S41 described above will be described with reference to the flowchart of FIG.

  That is, referring to FIG. 35, when the process for deleting the slide during playback of the script is selected in step S41 described above, the execution control unit 1313 displays the time parameter value, the media rendering process time table, and With reference to the event processing time table (S171 to S173), the media drawing processing and event processing that are valid at that time are determined. Then, the schedule description portion of the media rendering process and the event process determined to be valid at that time is deleted from the script, and the script is rewritten (S174).

  The process in step S42 in the case of executing the process of deleting the slide is ended as described above, and the process is returned to the menu process shown in the flowchart of FIG.

  The processing in step S42 when the slide rewrite prohibition is selected in the slide editing menu screen as shown in FIG. 36 in step S41 described above will be described with reference to the flowchart of FIG.

  That is, referring to FIG. 37, when the rewrite prohibition process of the slide during reproduction of the script is selected in step S41 described above, the execution control unit 1313 refers to the time parameter value and the event process time table at that time. (S181, S182), and the event processing that is valid at that time is determined. Then, the schedule description portion of the event process determined to be valid at that time is deleted from the script, and the script is rewritten (S183).

  The process in step S42 in the case of prohibiting the rewriting of the slide is finished as described above, and the process returns to the menu process shown in the flowchart of FIG.

  If it is determined in the above process that the event detected in step S17 is an event for executing the link movement process, the process shown in FIG. 17 is executed in step S27. That is, referring to FIG. 17, first, in step S51, the execution control unit 1313 refers to a parameter indicating a link destination included in the script, and interrupts execution of the script in step S52. In step S53, a link destination script is acquired based on the parameter referred to in step S51. Thereafter, the process returns to step S13 of the process shown in FIG. 9, and the acquired script is read and executed in the same manner.

  If it is determined in the above process that the event detected in step S17 is an event for executing the application execution process, the process shown in FIG. 18 is executed in step S26. That is, referring to FIG. 18, first, execution control unit 1313 causes display 144 to display a screen for selecting whether to start an application. If activation of the application is selected (YES in S61), the execution control unit 1313 activates a predetermined application in step S62. Next, when it is necessary to acquire predetermined data from the application (YES in S63), it is acquired in step S64. If it is necessary to rewrite the script being executed (YES in S65), the script writing unit 1316 executes a script rewriting process in step S66.

  Furthermore, the execution control unit 1313 resumes the reproduction of the script after the above-described script rewriting process is completed. The playback of the script can be started from the playback start time of the script, from the playback time of the script at that time, or from the start time of the rewritten section. Any of the above configurations may be adopted, and a configuration in which the user selects from the above options may be adopted.

  Note that the above restart time point may be dynamically determined according to a predetermined rule set for the terminal. Similarly, other times related to playback such as script playback time and end time may be determined. The script shown in FIG. 67 is used to execute a script according to the rule that “if the remaining playback time of the script is 10 seconds or less, restart from the current playback position of the script, otherwise restart from the start of the script”. The processing for determining the resumption point of reproduction will be specifically described.

  Referring to FIG. 67, first, the execution control unit 1313 refers to the current playback position of the script (S2101). This reproduction position indicates the reproduction position at the time when the pause is performed in step S21.

  Next, the execution control unit 1313 refers to the total playback time of the script (S2102). The total playback time of the script is acquired from the schedule description part of the script. For example, if the script is as shown in FIG. 7, in step S2102, lines 20 to 25, lines 26 to 31 and lines 32 to 36 are executed in time series. The total playback time is calculated, and the total playback time of the script is obtained as 19 seconds.

  Next, the execution control unit 1313 calculates the remaining playback time of the script by subtracting the current playback position of the script from the total playback time of the script acquired in step S2102 (S2103).

  If the remaining playback time of the script calculated in step S2103 is 10 seconds or less (NO in S2104), the execution control unit 1313 sets the playback start time as the current playback time of the script. If not (YES in S2104), the execution control unit 1313 sets the playback start time as the script start time.

This is the end of the description of the process for determining the specific script resumption point.
Next, application activation in step S62 described above will be described using the flowchart of FIG.

  Referring to FIG. 24, when a predetermined application is registered in the storage unit 130 of the mobile phone 1 (YES in S101), the execution control unit 1313 displays the playback time and area size described in the script, etc. Is started as a startup parameter (S102). If the predetermined application is not registered in the storage unit 130 of the mobile phone 1 (NO in S101), and if data acquisition from the application is necessary in the script (YES in S103), the file acquisition program 156 is used. Then, the local file acquisition application is activated (S104).

  If the predetermined application is not registered in the storage unit 130 of the mobile phone 1 in step S101, the predetermined application is downloaded from the server 2 via the communication unit 110 and stored in the storage unit 130 of the mobile phone 1. However, it may be configured to start.

  Further, the operation in step S102 described above will be described with reference to the flowchart of FIG. 66 by taking a case where the application to be activated is an image capturing application as a specific example.

  That is, referring to FIG. 66, first, the execution control unit 1313 obtains the size of the drawing area in which the media data to be replaced is displayed from the script (S2001). The size of the drawing area is generally described in the header portion of the script. Specifically, assuming that the script is shown in FIG. 6, numerical values 176 and 120 specified for the width (width) and height (height) of the drawing area are acquired from the description of the eight lines in FIG.

  Next, the execution control unit 1313 determines, based on the size of the drawing area acquired in step S2001, the smallest shooting size that can include the drawing area in the image shooting application (S2002). Specifically, when the image capturing application can select width 600 × height 480, width 120 × height 160, or width 120 × height 128 as the capturing size, in step S2002, the capturing size width It is determined that 600 × height 480 can include the size of the drawing area of width 176 × height 120 in the specific example.

  Then, the execution control unit 1313 designates the shooting size in the image shooting application as width 600 × height 480 based on the determination result in step S2002, and starts the image shooting application (S2003).

  If the application to be activated is the voice recording application by performing the same processing as described above, recording can be performed for the time described in the script. In addition, when the application to be started is a file acquisition application, it is possible to prevent selection of a file larger than the remaining available size so that the total file size of the message does not exceed the upper limit of the sendable size. . Further, when the application to be activated is a file acquisition application that acquires a moving image, it is possible to prevent selection of a file having a reproduction time longer than the reproduction time described in the script.

  Thus, the application activation in step S62 is terminated, and the process returns to the application execution process shown in the flowchart of FIG.

  Next, the data acquisition in step S64 described above will be described using the flowchart of FIG.

  Referring to FIG. 25, first, execution control unit 1313 receives data from the application activated in step S62 (S111). If the acquired data does not match the playback time and area size described in the script (NO in S112), the execution control unit 1313 converts the acquired data to match the description of the script (S113). .

  In step S113, the media data referred to by the reference information previously described in the script and the acquired data may be combined to generate a new script or media data. In the case where the acquired data is media data having a reproduction time, this generation process can be described as follows using FIG.

  That is, referring to FIG. 56, as shown in the left diagram of FIG. 56, in this specific example, “original media data” is described to have a reproduction time of 5 seconds and to be reproduced from the initial time t1. Yes. The “acquired data” is assumed to be a moving image taken for 5 seconds from time t2 (= t1 + 3), which is the time parameter value of the event referred to in step S31. In this case, when the user instructs to replace the “original media data” portion whose playback time corresponds to the time when the acquired data was captured with an appropriate user interface, “ The content and reproduction time of “data reproduced after rewriting” are data shown in the lower part of the left diagram of FIG. One method for realizing such a process is that the execution control unit 1313 instructs to capture a predetermined portion of the original media data, which overlaps the acquisition time and the reproduction time of the acquired data. Obtained by rewriting and saving with the editing program 133. In another implementation method, a script writing unit 1316 that receives an instruction from the execution control unit 1313 generates and writes a script that expresses a new start time and playback time of each of original media data and acquired data. It is. According to the latter implementation method, the script written by the script writing unit 1316 in the above specific example is the script shown in the right diagram of FIG. In addition, it is preferable to instruct by an appropriate user interface or the like whether or not to add to the generated data a portion from time t3 to time t4 that does not overlap the original media data.

  In step S113, for example, a part of the acquired data may be extracted by performing a process of extracting only the audio data from the moving image data, and new media data may be generated.

  In step S112 and step S113 during the above processing, when the size of the memory area occupied by the acquired data exceeds a predetermined value, for example, the data size such as color reduction of the image data or reduction of the sampling rate of the audio data You may perform the process which reduces.

  In addition, when the acquired data does not conform to the description of the script in step S112 during the above processing, predetermined media data (default data) stored in the storage unit 130 of the mobile phone 1 is used instead of the acquired data. May be.

  Thus, the data acquisition in step S64 is terminated, and the process returns to the application execution process shown in the flowchart of FIG.

  Next, the script rewriting process in step S66 described above will be described using the flowchart of FIG.

  Referring to FIG. 26, execution control unit 1313, when the sum of the media data and the acquired data included in the script is equal to or less than the prescribed value stored in storage unit 130 (YES in S121), script writing unit 1316. Then, the script rewriting process is executed (S122). Otherwise (NO in S121), the execution control unit 1313 displays on the display 144 that the script cannot be rewritten (S123).

  In step S122 described above, the execution control unit 1313 can also change the time point at which the acquired data is written using the time parameter value of the event referred to in step S31. For example, as script rewriting processing, as shown in FIG. 57, when a key operation is performed 4 seconds after the start of reproduction and a camera application is activated to rewrite media data, the key operation is performed. Shooting is started from the time (fourth second), and the moving image 2 acquired from the camera application after the photographing is finished is reproduced from the time when the key operation is performed (fourth second), and the reproduction end time of the moving image 1 as the original media data Write in the interval up to (8th second). In this example, the media data is written at the time of the time parameter value of the event. However, it can be written at a time after or before the time parameter value of the event.

  This completes the script rewriting process in step S66, and returns the process to the application execution process shown in the flowchart of FIG.

  This completes the script playback and editing process on the mobile phone 1 in the data communication system.

  More specifically, a case where the mobile phone 1 in the data communication system executes the script reproduction and editing processing shown in FIGS. 6 and 7 will be described with reference to the flowcharts of FIGS. 19 and 20.

  Referring to FIG. 19, when the 20th line of the script shown in FIG. 7 is executed, the syntax analysis unit 1312 of the playback / editing engine program 131 first analyzes the script, and in step S71, the execution control unit 1313 instructs the image data display / reproduction program 134 to display and reproduce the moving image data “First Image.mpg” in the Image area in 0 second to 8 seconds. Then, in step S72, the image data display / reproduction program 134 displays and reproduces the moving image data “First Image.mpg” on the display 144 in 0 second to 8 seconds, and further includes sound data included in the data. Plays the sound data from the speaker 148.

  Next, when an event that the user operates the select key between 0 second and 8 seconds occurs, the process described in the 24th line of the script shown in FIG. 7 is shown in FIG. Execute the process. That is, referring to FIG. 20, when event management unit 1315 detects the occurrence of an event that the select key has been operated, in step S81, execution control unit 1313 displays an event processing time table whose specific example is shown in FIG. The corresponding processing content (action) is determined with reference to FIG. That is, referring to the event processing time table, in step S82, the execution control unit 1313 activates the corresponding image capturing program 133 and the audio data recording program 136.

  Next, in step S <b> 83, the image capturing program 133 displays a UI necessary for image capturing on the display 144, and generates image data using the camera 149 and the key code input device 142. At the same time, the voice data recording program 136 uses the microphone 146 to generate voice data. A data synthesis program (not shown) synthesizes the generated image data and audio data. In this way, new data is acquired in step S83.

  When the above process ends, the execution control unit 1313 ends the image capturing program 133 and the audio data recording program 136 in step S84.

  In step S85, the playback / editing engine program 131 receives the synthesized data, and the script writing unit 1316 rewrites the script so that the data is displayed in the Image area in 0 second to 8 seconds. .

  Thus, the process on the 24th line of the script shown in FIGS. 6 and 7 is terminated, and the newly generated script is reproduced and edited.

  When the mobile phone 1 of this data communication system executes the script shown in FIGS. 6 and 7 and executes the editing process, the above process is executed line by line.

  Such script playback / editing processing will be described with reference to specific display screen transitions shown in FIG. 21, FIG. 22, and FIG. FIG. 21 and FIG. 22 are diagrams showing specific display screen transitions in order, and FIG. 85 shows a specific example of these display screens in one figure.

  FIG. 21A is a specific example of a screen that displays a list of available templates. On the screen shown in FIG. 21 (A), four templates of templates “I'm here now”, “Look at this!”, “Large time”, and “This is good!” Are displayed in a selectable manner. In this example, only the name of the template is displayed, but it is more preferable that a specific image of a message that can be created using the template is displayed as a thumbnail image together with the name.

  The templates displayed in a list are usually templates stored in a memory in the terminal, but may be displayed that can be downloaded from an external device or server through a network connection.

  In this template list screen, all acquired or acquirable templates may be displayed, or only some templates may be displayed. When performing such a display, as for how to select which template to display, a method of carefully selecting only frequently used templates by a user operation, a method of displaying only recently used templates, a message, etc. In the case of transmission, if a transmission partner is designated in advance and the function of the transmission partner terminal is known, a method of displaying only a template that can be interpreted and executed by the transmission partner terminal is considered. In this way, if the template is selected and displayed according to the function of the transmission partner terminal, for example, when the transmission partner terminal is a mobile phone that does not have a camera, the camera is activated to display a photo on the spot. Templates that are intended to be photographed and sent can be excluded from the list.

  Even when only a part of the templates is displayed as a list, it is preferable that all other acquired or acquirable templates can be displayed by a predetermined user operation. In addition, when the functions of the sending partner terminal are known, the templates that have been found that not all described functions can be interpreted and executed by the partner terminal are displayed separately or selected by distinguishing colors and fonts. Sometimes a warning screen may be displayed to alert the user, or the selection operation may be disabled.

  Next, a specific description will be given of processing in a case where the template “pretty” is selected on the screen shown in FIG.

  Referring to FIG. 21, first, a script “pretty” which is a predetermined navigation frame is selected, and a reproduction process is executed. In the mobile phone 1, the above processing is executed and the script included in the template is analyzed, and the execution of the script is started as shown in FIG.

  UI parts that change in time series with script execution and that navigate the user are displayed on the display 144 as shown in the screen of FIG. Specifically, a character string “camera activation” is displayed on the display 144 when the second frame of the execution screen shown in FIG. If the user does not perform any operation in this state, after the character string “camera activation” is displayed for a predetermined time, the script is executed to the end shown in the screen of FIG. Returning to the starting screen of FIG. 21B, the reproduction is repeated.

  Although a character string is displayed as a UI component in this example, an icon image such as a camera or a moving image may be displayed, or voice or music may be played back.

  When the user performs a predetermined key operation at the timing when the character string “camera activation” shown in FIG. 21C is displayed on the display 144, the image capturing program 133 is executed, and FIG. As shown in FIG. 21F, the camera 149 can shoot a moving image. Then, as shown in FIG. 21G, the template “pretty park” is rewritten so that moving image data shot in the time zone in which the character string “camera activation” is displayed is reproduced. That is, after the user has taken a picture by navigating the UI parts, the execution of the script is returned to the beginning of the script or when the UI part of “camera activation” is displayed on the display 144, as shown in FIG. As shown in FIG. 22B, a moving image shot as shown in FIG. 22B is displayed during the time period when the UI component of “camera activation” is displayed on the display 144 when the template “pretty much” is executed first. Played. In this case, when it is determined at the time of script execution that the moving image data is registered in the time zone in which the “camera activation” UI component is displayed, the “camera activation” UI component is displayed in that time zone. Is preferably not displayed. Or, as shown in FIG. 22B, it is preferable to change the UI so that a “re-shoot” UI part is displayed during the time period.

  Thereafter, the execution of the automatically rewritten script is repeated until an end operation is performed.

  Next, “Preview” is selected from the menu, and a preview execution process is executed. In the preview execution, as shown in FIG. 40, the edited script is played back in a state where display and playback of UI parts are all invalidated. That is, the character string “re-shoot” shown in FIG. 22B is not displayed, and the image shooting program 133 is not executed even if the user performs a predetermined key operation. The preview process will be described later.

  After confirming the display of the edited script in the preview, “Email attachment” is selected from the menu, and the script and media data are transmitted to the other mobile phone 1. The mail attachment process will be described later.

  Furthermore, as another specific example of the present embodiment, a case where the mobile phone 1 in the data communication system executes the reproduction of the script shown in FIGS. 29 and 30 will be described with reference to the flowcharts of FIGS. explain.

  Referring to FIG. 31, when the 19th line of the script shown in FIG. 30 is executed by the mobile phone 1 in the present embodiment, the script is first analyzed by the syntax analysis unit 1312 of the playback / editing engine program 131. The execution control unit 1313 instructs the image data display / reproduction program 134 to display and reproduce the image data “ProductImage.jpg” in the top area in 0 to 10 seconds (S141). Then, the image data display / reproduction program 134 displays and reproduces the image data “ProductImage.jpg” on the display 144 in 0 to 10 seconds.

  Next, when the 20th line of the script shown in FIG. 30 is executed, similarly in step S142, the execution control unit 1313 sets the middle display area for the text display reproduction program 134 in 0 to 10 seconds. An instruction is given to display and reproduce the text data “ProductDesc.txt” (S142). Then, the text data display / reproduction program 138 displays and reproduces the text data “ProductDesc.txt” on the display 144 in 0 to 10 seconds.

  Next, when an event that the user operates the select key between 1 second and 4 seconds occurs, the processing described in the 21st line of the script shown in FIG. 30 is shown in FIG. Execute the process. That is, with reference to FIG. 32, when the event management unit 1315 detects the occurrence of an event that the select key has been operated, the execution control unit 1313 refers to the event processing time table and corresponds to the processing content (action). Is determined (S151). Then, the execution control unit 1313 refers to the event processing time table and activates the WWW browser program 151 that is a corresponding application (S152).

  Next, the WWW browser program 151 acquires WWW content via the communication unit 110 and displays it on the display 144 (S153).

  When the above process ends, the execution control unit 1313 ends the WWW browser program 151 (S154).

  Thus, the process on the 21st line of the script shown in FIGS. 29 and 30 is finished, and the script reproduction process is executed.

  When the mobile phone 1 of this data communication system executes the script reproduction process shown in FIGS. 29 and 30, the above process is executed line by line. Such script reproduction processing will be described with reference to specific display screen transitions shown in FIG.

  Referring to FIG. 33, the mobile phone 1 of the data communication system acquires a script from the communication unit 110 and executes a reproduction process. The mobile phone 1 executes the above process, analyzes the script, and starts executing the script.

  When the execution of the script is started, UI parts that change in time series with the execution of the script and that navigate the user are displayed on the display 144 as shown in FIG. Specifically, an icon image of “WWW browser” is displayed on the display 144 during the reproduction shown in FIG. If the user does not perform any operation in this state, the state automatically changes to the state shown in FIG. 33C and the icon image of “call” is displayed on the display 144. Further, when the user does not perform any operation in this state, the state automatically changes to the state shown in FIG. 33D, and the icon image of “mail transmission” is displayed on the display 144.

  When the user performs a predetermined key operation during the time period shown in FIG. 33B in which the icon image of “WWW browser” is displayed on the display 144, the WWW browser program 151 is executed and is transmitted via the communication unit 110. The predetermined WWW content can be acquired and reproduced on the display 144. When the WWW browser program is terminated after browsing the WWW content, script playback is resumed from the previous playback position.

Thereafter, the reproduction of the script is repeated until an end operation is performed.
Further, preview processing in the mobile phone 1 when “Preview” is selected on the menu screen shown in FIG. 16 as a specific example will be described with reference to the flowchart of FIG. That is, referring to FIG. 23, the preview process is substantially the same as the script reproduction and editing process shown in FIG. 9, but in the preview process, in step S93 based on the analysis result in step S92. Only the time table for media data drawing processing is created based on the drawing information description part of the script. In step S94, the portion based on the user navigation information description portion of the script is not reproduced, but the portion based on the drawing information description portion is executed. When the event management unit 1315 detects the occurrence of an event (YES in S95), the execution control unit 1313 executes a process corresponding to the event (S96). The processing according to the event here will be described later with a subroutine. In the preview process, the script definition input event is invalid, and only the system event and the fixed input event are processed in step S96.

This completes the script preview process in the mobile phone 1.
Next, processing according to the event executed in step S96 will be described with reference to the flowchart of FIG.

  Referring to FIG. 38, first, upon receiving event occurrence detection information from event management unit 1315 in step S95 described above, execution control unit 1312 pauses the script being executed (S191).

  If the event detected in step S95 is an event indicating that the execution of the script has reached the end of the execution point of the script (YES in S192), the execution control unit 1313 executes the execution of the script. (S193), the process returns to step S94 in FIG. 23, and the script is started again. That is, the script execution is returned to the start point of the execution point again, and the script execution is repeated from the beginning.

  On the other hand, when the event detected in step S95 is an event other than an event indicating that the execution of the script has reached the end of the execution point (NO in S192), the execution control unit 1313 detects the detected event. The processing content is determined based on (S194).

  When the processing content determined in step S194 is “menu” (“processing” = menu in S194), the execution control unit 1312 executes the menu processing shown in FIG. 15 (S195). The procedure for executing the menu process is the same as that for performing the reproduction / editing process. However, as shown in FIG. 39, a menu screen on which the editing process is an option instead of the preview process is presented on the display 144. When the editing process is selected, the reproduction editing process shown in FIG. 9 is executed. When the playback / editing process is started from the menu, it may be started from the start time of the script, or may be started from the script execution time at the time of the preview process.

  On the other hand, when the processing content determined in step S194 is “skip” (“processing” = skip in S194), the execution control unit 1312 refers to the media rendering processing time table and executes the media rendering processing to be executed next. And the script execution point is updated at the start time of the media drawing process (S196).

  Here, as the movement destination of the script execution point, in addition to the start point of the media drawing process to be executed next, the start point of the media drawing process to be executed next and the following and the skip process are selected. It may be an execution point when a predetermined time elapses when script execution is continued without being performed.

  Next, the mail attachment process in the mobile phone 1 when the process for attaching the script and the media data to the mail is selected on the menu screen shown in FIG. 16 will be described with reference to the flowchart of FIG. .

  Referring to FIG. 34, first, execution control unit 1313 causes display 144 to display a screen for selecting whether to transmit as reproduction-only multimedia data. If it is not selected to attach the reproduction-only multimedia data on the screen (NO in S161), the execution control unit 1313 uses the multimedia data in the other mobile phone 1 that receives the multimedia data. A hint as to whether or not to use is set (S165). In other words, in step S165, the execution control unit 1313 sets whether the multimedia data is to be used as a template for navigating the content editing operation or is simply browsing content (contents). . Then, the process proceeds to step S163.

  On the other hand, if the screen for selecting whether or not to transmit as reproduction-only multimedia data is selected to attach the reproduction-only multimedia data (YES in S161), the execution control unit 1313 displays the script user. The navigation information description part is deleted (S162).

  Next, the execution control unit 1313 writes a script in the storage unit 130 (S163). Then, the execution control unit 1313 sets the above script and multimedia data as an attached file and starts the mail program (S164).

  In the above mail attachment process, if it is selected to attach the reproduction-only multimedia data (YES in S161), the multimedia data template before editing may be attached to the mail together with the reproduction-only multimedia data. Good. In addition, in step S162 of the mail attachment process described above, if there is a description part that has been previously determined that the mobile phone 1 to be received cannot be processed, the description part may be deleted from the script.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to the drawings.

  Since the configuration of the data communication system including the multimedia data processing apparatus according to the second embodiment of the present invention is the same as that of the first embodiment shown in FIG. 1, description thereof will not be repeated here. . The configuration of the mobile phone 1 in the second embodiment and the flow of programs and data processing stored in the storage unit 130 of the mobile phone 1 are also shown in FIGS. 2 and 3, respectively. Since this is the same as the case of, the description here will not be repeated.

  First, the mobile phone 1 which is the multimedia data processing apparatus according to the second embodiment plays a script including the playback order information of the media data, writes the media data, changes it, and plays the media data. A case where a script including order information is created will be described.

  FIG. 41 shows a specific example of a screen display when a template and a script are changed using the template. The template shown in FIG. 41 is composed of three scenes 1 to 3. The transition of these scenes can be considered when the time information is described in the script in the template and follows it, according to the switching time set on the terminal, or when the user explicitly switches the scene. It is not particularly defined in the invention. In this specific example, the still image of scene 2 shown in FIG. 41 (B) and the text of scene 3 shown in FIG. 41 (C) are shown in FIG. 41 (E) and FIG. 41 (F), respectively. Shows a case where correction is performed by replacing the user.

  Further, a specific example of a script for realizing the screen display of FIG. 41 is shown in FIG. The script shown in FIG. 47 is a script indicating that scenes representing a collection of media data to be played back in parallel are played back sequentially. That is, referring to FIG. 47, descriptions in lines 15 to 17, lines 18 to 21, and lines 22 to 25 represent scene 1 to scene 3, respectively.

  More specifically, in the scene 2 described in the 18th to 21st lines, the 19th line describes reproduction of an image, and the 20th line describes reproduction of text. These images and text belong to the scene 2 described by the 18th to 21st lines and are reproduced in parallel.

  Furthermore, the time displayed after “dur =” in the script represents the time for reproducing the corresponding scene, and after that time has elapsed, the playback is switched to the next scene. In the present embodiment, the script is shown as using such a description method. However, the description method of the script is not limited in the present invention, and any method can be used as long as sequence information for reproducing media can be described. It does n’t matter if you

  Note that the sequence refers to a temporal order relationship during multimedia data playback. A sequence is defined by being surrounded as a child element by a specific section on the playback time axis (specifically, “scene”, “slide”, and more specifically, surrounded by <par> tags and <seq> tags in the SMIL language. One or more (sections). Understand the order relationship of each section on the time axis, and create and maintain a data structure that can be used as a clue to start / stop playback, load / unload media data in each section (media data included) Thus, the sequence is managed. Specifically, the sequence is managed by using a time table as shown in FIGS. The time at which each section starts and ends on the playback time axis is called a sequence change point. In the specific example of the UI allocation status display time table shown in FIG. 7, the 0th, 8th, 15th, and 19th seconds correspond to the sequence change points.

  Further, FIG. 42 is a block diagram showing a functional configuration of the playback / editing engine program 131 of the mobile phone 1 which is the multimedia data processing apparatus according to the present embodiment. The control unit 120 executes the playback / editing engine program 131 stored in the storage unit 130, whereby each function shown in FIG.

  Referring to FIG. 42, functions of playback / editing engine program 131 of mobile phone 1 in the present embodiment are a template storage unit 1000, a selection unit 1001, a script / data temporary storage unit 1002, and a script execution unit 1003. A user change data determination unit 1004, a terminal function selection unit 1005, a terminal function execution unit 1006, a data change unit 1007, a script / data change confirmation unit 1008, a script / data output unit 1009, an output script / A data storage unit 1010 and a data embedding unit 1011 are included.

  The template storage unit 1000 stores one or more templates received and passed by the receiving unit 1015. The template stored in the template storage unit 1000 usually includes both a script as shown in FIG. 47 and sample media data (still image or text) used by the script. May be managed separately in the form of a file, etc., may exist in a server connected via a network, or may be in other forms as long as it is accessible when using a template.

  Note that it is preferable to be able to prohibit or permit the description of the script included in the template to take out only the media data included in the template from the template and perform operations such as separate transfer or storage. Furthermore, it is preferable that a plurality of templates are normally stored in the terminal, and a template that the user wants to use can be added from a server or a memory card connected via a network.

  The selection unit 1001 accepts a template selection from the user in the list of templates displayed in a selectable manner. The template selection screen in FIG. 21 is a typical example.

  The script / data temporary storage unit 1002 temporarily stores the script and media data included in the template. Since the script includes a rewritable part, the template is usually copied and stored in the script / data temporary storage unit 1002 which is this temporary storage area. However, there is no need to make a copy of the media data if the media data itself is not modified, such as rewriting the file name.

  The script execution unit 1003 is a core part of the present invention. In the present invention, the script is for realizing display in which scenes are switched. The script execution unit 1003 preferably manages the scene and time during script playback by creating a table in advance.

  The user change data determination unit 1004 determines whether the media included in the template can be changed. Most of the media included in the template are not rewritable, but by rewriting the reference information that refers to the media such as the media file name, multimedia data desired by the user can be generated in the end. it can.

  Media data included in a template is usually specified and described in the form of a file name in a script. The user change data determination unit 1004 determines whether the media data can be changed. For example, in the case of the scene 2 shown in FIG. 41B, since it is possible to replace still images on the terminal side, the user change data determination unit 1004 determines that still image replacement is possible. Even when a still image file is specified in the script, it is possible that the data of the still image file cannot be changed.

  Further, in the case of a script managed by time, the user change data determination unit 1004 selects the time during which the media data is played, and rewrites the playback time itself, or further adds 4 to three scenes. It is preferable to allow the user change data determination unit 1004 to determine that the first scene is added.

  The terminal function selection unit 1005 selects a terminal function necessary for changing the part selected by the user change data determination unit 1004 from the functions of the mobile phone 1. For example, when the part selected by the user change data determination unit 1004 is a still image, the terminal function selection unit 1005 selects a function for replacing a still image. For example, when the mobile phone 1 is a camera-equipped terminal, it is preferable that the terminal function selection unit 1005 can also select a function for starting the camera.

  The terminal function execution unit 1006 acquires media data using the terminal function selected by the terminal function selection unit 1005. Then, the terminal function execution unit 1006 outputs the acquired media data to the data change unit 1007.

  The data change unit 1007 replaces the media data input from the terminal function execution unit 1006 with the original media data referenced from the script. At this time, what was a still image in the template may be changed to another type of media data such as a moving image or text after the change.

  At this time, if the reference information to the media data is information described in the file name in the script, the changed media data is held in the file format, and the name is described in the script. When the media data after the file name or after the change is in the form of a file and is managed with a name, the part specifying the media data in the script is rewritten with the file name. There is a need.

  The script / data change confirming unit 1008 accepts confirmation from the user that all the script and media data changes have been completed. Then, the script / data change confirming unit 1008 outputs to the script / data output unit 1009 that the change of the script and the media data is confirmed.

  The script / data output unit 1009 outputs the changed script and media data. The script / data output unit 1009 typically transmits it to another terminal or server in the form of an electronic mail such as MMS (Multimedia Messaging Service). Then, the script / data output unit 1009 outputs the script and media data output to another terminal or the like to the output script / data storage unit 1010 and the data embedding unit 1011.

  The output script / data storage unit 1010 holds the script and media data output from the script / data output unit 1009. The data embedding unit 1011 extracts media data from the output script and media data, and embeds it in the template.

  Next, processing in the playback / editing engine program of the mobile phone 1 which is the multimedia data processing apparatus according to the second embodiment will be described with reference to the flowchart of FIG. The process shown in the flowchart of FIG. 43 is also realized by reading and executing the reproduction / editing engine program 131 shown in FIG. 42 by the control unit 120 of the mobile phone 1 from the storage unit 130.

  Referring to FIG. 43, first, selection unit 1001 reads out a template stored in template storage unit 1000 or a template acquired from another device, displays the list on display 144 in a selectable manner, and displays the user's list. The selection is accepted (S1100). In step S1100, the selection unit 1001 preferably displays, for each template, the first or representative screen of the template as a thumbnail and the template title.

  Next, the script execution unit 1003 executes reproduction / editing processing for editing in accordance with a user instruction while reproducing the template selected in step S1100 (S1101). Details of the reproduction / editing process in step S1101 will be described later. Furthermore, although it is common for playback and editing to have a mode that can be switched by the user, it is preferable that editing and playback can be performed simultaneously.

  Next, the script / data change confirmation unit 1008 confirms whether or not the script change by the user is completed (S1102). When the script change is completed (YES in S1102), the script / data output unit 1009 outputs either or both of the confirmed script and data (S1103).

This completes the template reproduction and editing process in the present embodiment.
Next, the reproduction / editing process in step S1101 described above will be described with reference to the flowchart of FIG.

  Referring to FIG. 46, first, script execution unit 1003 switches the scene to be reproduced according to the description of the script (S1200). Although the processing in step S1200 is not limited in the present invention, a typical processing method is to first create a time table or a switching table as shown in FIG. A method to migrate is mentioned.

  FIG. 58 shows a specific example of the media management table when the script shown in FIG. 47 is played back / edited. More specifically, referring to FIG. 58, “Still Image 1” is displayed in “Scene 1” (FIG. 58A), and “Still Image 2” and “Text 2” are displayed in “Scene 2” (FIG. 58). 58 (B)), “Scene 3” indicates that “Still Image 3” and “Text 3” are reproduced (FIG. 58C). In step S1200, the data changing unit 1007 refers to such a media management table and sequentially performs playback / editing in units of scenes.

  Next, the script data output unit 1009 outputs data using the function of the terminal (S1201). The data output in step S1201 may be any form such as a still image, moving image, music, text, or the like. Note that the output processing in step S1201 will be described in detail later with a subroutine.

  Next, the script execution unit 1003 determines whether the mode is the correction mode or the reproduction mode (S1202). When a new template is created based on an existing template, it may be unnecessary to switch to the playback mode only by executing the correction mode. On the other hand, when a mail or message sent from another terminal is used as a new template, it is usually in the playback mode, and it is preferable that the mode can be changed to the correction mode.

  If the correction mode is set in step S1202 (YES in S1202), an input is further obtained from the user, and the user change data determination unit 1004 determines what input is made (S1203).

  If the result of determination in step S1203 is that the input from the user is an input for instructing playback of the previous scene (“previous scene” in S1203), the script execution unit 1003 displays the scene before the script. Processing is performed (S1204).

  Alternatively, when the result of determination in step S1203 is that the input from the user is an input that instructs script change processing (“change processing” in S1203), the data changing unit 1007 performs a specific change operation. (S1205). Note that the change operation in step S1205 will be described in detail later with a subroutine.

  Further, after the change in step S1205, the script / data change confirmation unit 1008 determines whether or not to obtain an input from the user and confirm again (S1206). As a result of the determination in step S1206, when the script after the change is confirmed again (“re-output” in S1206), the process returns to step S1201, and the script / data output unit 1009 changes the same scene in step S1201. Output the data. This is effective when a scene cannot be confirmed only by changing the still image, such as when a still image and music are output simultaneously.

  If the script is not in the correction mode in step S1202 (NO in S1202) and the reproduction of the script is completed (YES in S1207), the reproduction and editing process is terminated.

  Next, the data output processing in step S1201 described above will be described in detail with reference to the flowchart of FIG.

  Referring to FIG. 44, first, the user change data determination unit 1004 determines whether there is change data in the scene (S1110). The determination process in step S1110 will be described in detail later with a subroutine.

  Next, the terminal function selection unit 1005 selects the change data changing means determined in step S1110 (S1111). For example, even if the script is the same, the functions that can be used for reproduction differ depending on the terminal as shown in FIG. FIG. 49 is a diagram illustrating that different functions are selected for the same script depending on the terminal. In selecting this function, not only the hardware of the terminal but also the difference depending on the software of the terminal can be considered. In this case, in step S1111, it is preferable to select a function that can be used in accordance with the terminal, such as still image file selection, video camera selection, and the like. Note that the change data changing means selection processing in step S1111 will be described in detail later with a subroutine.

  Next, the data changing unit 1007 displays the change guidance on the display 144 as an operation guide (S1112). The change guidance displayed in step S1112 is a guide function for selecting a file to be replaced when, for example, still images are replaced, and typically indicates display of a still image file selection button. In step S1112, the data changing unit 1007 dynamically generates such guidance, so that the user can be easily informed how the change can be made.

  Next, the script execution unit 1003 executes scene output together with the change guidance displayed in step S1112 (S1113).

  Then, the script data temporary storage unit 1002 displays, for each change data, a set of input events corresponding to the change means (application) selected in step S1111 and the change guidance generated in step S1112 in the scene event table. (S1114). In step S1114, specifically, the script / data temporary storage unit 1002 is activated by the “F1” key event for the change data “still image 2” as shown in the specific example of the scene event table of FIG. Can be changed between the “image capture” application and the “file acquisition” application activated by the “F2” key event, and the change data “text 2” is the “file acquisition” application activated by the “select” key event. Remember that you can change it.

  Next, the changing process in step S1205 will be described with reference to the flowchart of FIG.

  Referring to FIG. 65, first, the data changing unit 1007 refers to the scene event table (S1501). If the user input event input in step S1203 is not registered in the scene event table (NO in S1502), the process ends.

  If the user input event input in step S1203 is registered in the scene event table (YES in S1502), the script execution unit 1003 determines and activates the corresponding application from the scene event table (S1503). Then, the data change unit 1007 determines the corresponding user change data from the scene event table, acquires the media data from the started application, and rewrites the user change data with the media data acquired from the started application (S1504). ). More specifically, when the key “F1” is pressed, the “image capture” application is activated in step S1503. When the user finishes taking a still image with the “image capture” application, the user change data “still image 2” is rewritten with the newly captured data in step S1504.

  In the above example, when the user performs an operation of inputting media data such as an image, sound, video, and text, there are a plurality of locations where the input media media can be inserted into the template. In some cases, the media data may be inserted into two or more locations. Alternatively, the media data input by the user may be divided and a part thereof may be inserted at a plurality of locations.

  In addition, when media data with a time axis such as voice, music, video, animation, etc. is inserted into a template, if the playback time of each media data is longer than the time specified in the template in advance, Insertion of media data may be prohibited, the playback speed of each media data may be increased to match the specified time, or playback may end halfway after the specified time has passed, or at the specified time Only the part that fits may be played back. Alternatively, the time rule of the template may be changed so that each media data is completely reproduced.

  In addition, when inserting media data such as text, still image, video, voice, music, animation, etc., the above example selects the file according to the inserted media data and activates the application such as the camera. However, media data may be freely selected, and the media data may be converted into media data that fits in a template. For example, only sound may be extracted from the captured moving image and embedded in the template, or the first frame of the moving image may be extracted as a still image and embedded in the template.

  In addition, when entering the above text, the fixed text stored in the device, the text temporarily saved on the clipboard, or the address book, schedule, and transmission / reception that can be acquired as text data stored in the device It is preferable to read data such as e-mail.

  Next, the determination process in step S1110 will be described with reference to the flowchart of FIG.

  That is, with reference to FIG. 59, the user change data determination unit 1004 refers to the currently reproduced scene and the media management table (S1301, S1302), and determines the media data of the scene. For example, when the mobile phone 1 is currently playing “Scene 2” of the script shown in FIG. 47, the user change data determination unit 1004 refers to the media management table shown in FIG. And “text 2” are determined as change data.

  Next, the change data selection unit selection process in step S1111 described above will be described with reference to FIG. FIG. 60 is a table that records the correspondence between the data acquisition program stored in the mobile phone 1 and the media types that the mobile phone 1 supports, and is a table that is stored in the storage unit 130 of the mobile phone 1. More specifically, in the table shown in FIG. 60, an “image capture” application is registered in the mobile phone 1, and “still image” and “moving image” can be acquired, and an “audio data recording” application is registered. “Audio” can be acquired, and “File acquisition” application is registered and “Still image”, “Movie” and “Text” can be acquired.

  In step S1111 described above, the terminal function selection unit 1005 refers to the table, and selects a usable changing unit based on the media type of the change data determined in step S1110. For example, when the media type of the change data is “still image”, an application of “image capture” and “file acquisition” is selected.

  FIG. 53 shows a specific example of screen transition display when a script is executed on the mobile phone 1. Referring to FIG. 53, in scene 2, the process of step S1113 is executed to display a still image and text, and the process of step S1112 is executed to display a change guidance. That is, in the scene 2 of FIG. 53, characters (guidance) “You can attach a picture here” are displayed, and when the corresponding button is pressed, still image data in the data folder is displayed as a thumbnail and displayed as a list. In this list display screen, it is more preferable that file names, data size information, and the like are written together.

  In addition, when two or more media such as still images and music are output in the same scene, the mobile phone 1 is modified with “You can attach a picture here” and “You can attach a music here” A method of selecting either still image data input or music data input by a user operation such as displaying a plurality of guides and pressing a shortcut key, or switching in time, and first “You can attach a picture here” guide , Then display the “You can attach a music here” guide or the “You can attach data here” guide indicating that some media data can be inserted, and press the corresponding button. next In the next screen, the menu of “Attach a picture” and “Attach a music” is selectably displayed, and the user selects this to move to still image data input or music data input. Any method such as a method for performing the method may be used.

  In addition, when the correctable media data and the uncorrectable media data can be distinguished, it is preferable that the mobile phone 1 distinguishes and indicates to the user what can be corrected in the scene. This can be determined by describing whether the media data can be changed in the script.

  More preferably, the mobile phone 1 has a function of jumping to the next time or returning to the previous time for each scene. In normal playback, even if scene 1 is displayed for 8 seconds, the “next scene” function is used so that when the user presses the button after 1 second, the remaining 7 seconds are skipped. E-mail can be created and confirmed.

  More preferably, when the user indicates an intention to edit, the mobile phone 1 has a function of stopping reproduction at that time. For example, it is preferable to stop the playback timer when a user selects a still image on the still image selection screen by pressing a button when the still image is replaced in scene 2 of FIG. When resuming playback after changing the data, it may be possible to start from the playback time that was stopped when the editing operation was performed, start playback from the beginning of the whole, or restart from the first time of the scene. It is done. The beginning of the scene is the beginning of scene 2, that is, when the editing button is pressed at 13 seconds where the scene 2 is scheduled to be reproduced from the 8th to 15th seconds, that is, 8 seconds. Point to your eyes.

  In the template editing process described above, when one or more media data such as text, still image, video, sound, music, and animation is inserted into the template, the template is different from the template currently selected and edited by the user. You may want to change to editing the template. In this case, when changing to any template, the media data inserted so far may be discarded or inserted into the template after the change. In this case, it may be impossible to insert the media data inserted before the change into the template after the change, but in this case, such a change to the template may not be permitted. However, media data that cannot be inserted may be discarded, and only media data that can be inserted may be inserted. In addition, when media data that has been inserted into the template before the change can be inserted into a plurality of locations in the template after the change, it may be inserted into one of the plurality of locations or may be inserted into a plurality of locations. Good. Also, changes to other templates may not be completely permitted in the first place.

[Modification]
Next, a description will be given of a process of playing back a script including media data playback order information and action execution order information, executing an action, and creating a script including media data playback order information.

  50 and 61 show specific examples of scripts including media data playback order information and action execution order information. The script description shown in FIG. 50 defines an action and a drawing area in the same manner as the script description shown in FIG. The script description shown in FIG. 61 defines media data playback and action execution order information in the 17th to 33rd lines. Lines 18 to 32 sequentially execute scenes 1 to 3 defined in lines 19 to 21, lines 22 to 26, and lines 27 to 31, respectively. It is shown that.

  Lines 22 to 26 define execution of the second scene. More specifically, the 23rd line describes that the image indicated by “SecondImage.jpg” is displayed in the “Image” area. Line 24 describes that the text indicated by “Second Text.txt” is displayed in the “Text” area. Line 25 describes that the action referred to by “act_image” is executed when the “select” key event of “UI” is detected.

  Further, FIG. 48 is a block diagram showing a functional configuration of the reproduction / editing engine program 131 of the mobile phone 1 in a modification of the second embodiment. 48, the same reference numerals as those in FIG. 42 are the same as those shown in FIG. 42, and therefore description thereof will not be repeated here. 48, the functions of the playback / editing engine program 131 of the mobile phone 1 in the modification of the second embodiment are the template storage unit 1000, the selection unit 1001, and the script / data temporary storage unit. 1002, script execution unit 1003, terminal function selection unit 1012, action determination unit 1013, action execution unit 1014, data change confirmation unit 1008, script / data output unit 1009, output script / data storage unit 1010 And a data embedding unit 1011. Here, components different from those in FIG. 42 will be described.

  The terminal function selection unit 1012 selects a terminal function according to the description of the script. In the script, for example, the terminal function is written in the form of dest in the action statement in the 12th to 15th lines of FIG. Then, the terminal function selection unit 1012 outputs the selection result to the action execution unit 1014.

  The action determination unit 1013 determines an action according to the description of the script. The determination of the action in the action determination unit 1013 is realized by, for example, interpreting what is described in the type of the 12th to 15th lines of the script shown in FIG. Then, the action determination unit 1013 outputs the determination result to the action execution unit 1014.

  The action execution unit 1014 processes the data obtained using the function selected by the terminal function selection unit 1012 based on the action determined by the action determination unit 1013. The action execution unit 1014 executes various actions such as image synthesis, music splicing, special effects, etc., as well as insertion and replacement, by executing the actions explicitly described.

  Note that the processing in the playback / editing engine program of mobile phone 1 in the modification of the second embodiment is similar to the processing shown in FIG. 43, and therefore description thereof will not be repeated here. Also, since the playback and editing processing of the navigation frame, which is a script, in the mobile phone 1 in the modification of the second embodiment is almost the same as the processing shown in FIG. 46, the same processing is here. Will not be repeated.

  In this modification, in the process shown in FIG. 46, first, an action management table as shown in FIG. 63 is created in addition to the media management table as shown in FIG. FIG. 63 is a diagram showing a specific example of the action management table when the script shown in FIG. 61 is reproduced / edited. More specifically, referring to FIG. 63, there is no action that can be executed in “scene 1” (FIG. 63A), and “still image embedding action” can be executed in “scene 2” (FIG. 63). 63 (B)), “scene 3” indicates that “text embedding action” can be executed (FIG. 63C). The mobile phone 1 according to the modification refers to such an action management table, and performs reproduction / editing sequentially in units of scenes.

  Next, the data output processing in step 1201 in the modification will be described in detail with reference to the flowchart of FIG.

  Referring to FIG. 62, first, the action determination unit 1013 refers to the action management table to determine an action that can be executed in the scene (S1401). Next, the action determination unit 1013 displays an action execution guidance on the display 144 (S1402). The action execution guidance is an operation guide function for starting execution of an action. The action execution guidance display process in step S1402 is the same as the change guidance display process in step S1112 of the first embodiment, and therefore description thereof will not be repeated. Then, the script execution unit 1003 executes scene output together with the guide displayed in step S1402 (S1403).

  FIG. 45 shows a specific example of display of screen transition when a script is executed on the mobile phone 1 in the modification. Referring to FIG. 45, in scene 2 shown in FIG. 45 (B), the process in step S1113 is executed to display a still image, and the process in step S1112 is executed to display a change guidance. Yes. That is, in the scene 2 shown in FIG. 45 (B), the letters (guidance) “You can attach a picture here” are displayed, and when the corresponding button is pressed, the camera application is started as shown in FIG. 45 (D). Is done.

  Note that, as shown in FIG. 49, all the descriptions dedicated to each terminal having different usable functions may be described in the script, and only the description about the functions that can be used by the terminal during template reproduction may be validated. Examples of these functions include a 3D liquid crystal display function and a display display function of two or more screens.

  Alternatively, a configuration may be adopted in which all reference information to media data having different contents for each attribute of the target user is described in a script so that only media data suitable for the attribute of the target user is valid when the template is used. As this attribute, for example, the language used by the user, the area where the user lives, and the like are particularly mentioned.

  In the above-described embodiment, a method for displaying a UI component indicating that the camera is to be activated on the screen and activating the camera by a predetermined key operation has been described. However, as shown in FIG. Is displayed in-line, and a UI part indicating that shooting is to be performed is displayed for a predetermined time (FIG. 52D). When a predetermined key operation is performed during this time period, shooting is performed by the camera. The video may be played back immediately. At this time, when a predetermined time for displaying the “photographing” UI component has passed without performing the photographing, the next reproduction process in the time zone may be advanced. Alternatively, playback may be paused at the end time of the time period, and may wait until a “shoot” operation is performed. Alternatively, the reproduction of the time period may be repeated by returning to the beginning of the time period. This repetition process may be performed indefinitely unless the “shoot” operation is performed or interrupted, or after a predetermined number of repetitions, if the “shoot” operation is not completed, You may make it go to.

  Note that, in the message creation process in the above embodiment, the total data size of the message may exceed the upper limit of the transmittable size. As a countermeasure for this, when the user performs a message transmission operation, an oversize warning may be issued and transmission may not be performed. Alternatively, each time the user inserts media data, the total size may be checked one by one, and if the limit is exceeded, a warning may be issued at that point to invalidate the insertion operation. Alternatively, if the size of media data that is a candidate for insertion is known in advance, it is determined whether or not the total of the total message size up to the present and the size of the media data exceeds the maximum size that can be transmitted. However, selection may not be permitted for media data exceeding the limit.

  Note that FIG. 50 and FIG. 61 described above are specific examples of scripts used only on the creation side. The twelfth to fifteenth lines are used only on the device side that creates and transmits the message. The other parts are also necessary for playback on the device receiving the message. When such a script is output by the script / data output unit 1009, notation contained in the script that is not required by the receiving device or a command that cannot be transmitted in the communication procedure is finally deleted or commented out. Should be output. In such a case, the script / data output unit 1009 preferably includes notation rules to be deleted, and these are deleted.

  Note that the script / data output unit 1009 is now used so that the receiver can download the template itself from the server for the purpose of creating another message using the same template. It is more preferable to send the template download source address.

  Furthermore, it is preferable that the script / data processing unit 1009 converts the script into a format such as HTML, and transmits the converted script together with the original script. Thus, even if the receiving side cannot process the original script at all, if the converted script can be processed, the receiving side can perform reproduction.

  Further, the script and media data before the deletion process are stored in the output script / data storage unit 1010 so as to be convenient when editing based on the template and re-editing and transmitting the transmitted message. Also good. For the same purpose, as shown in FIG. 55, a table indicating the correspondence between the transmitted message and the used template is held. For example, when the message “1221” is to be re-edited, the above table is searched and the template is searched. It is also possible to automatically select “pretty time” and embed the media data included in the message “1221” in the template “pretty time” in the data embedding unit 1011 before editing.

  In order to further facilitate editing, it is preferable to create a thumbnail for each scene. FIG. 51 shows an example of the display. This can be easily realized by interpreting the entire script included in the template and creating the entire time table or sequence table as shown in FIG. By preparing thumbnails for each scene, the thumbnail list is displayed based on the script so that playback can be started from the playback execution point of the scene triggered by the user selecting an arbitrary thumbnail. Then, when rewriting the script, it becomes possible to easily rewrite only the desired data without reproducing from the beginning of the script.

  By executing such a script on the mobile phone 1, multimedia contents can be easily created. In other words, multimedia contents can be easily created by an intuitive method simply by performing a predetermined operation in accordance with guidance information displayed by changing the time series by executing this script. Also, even if a template has already been inserted with media data, the media data can be easily replaced simply by performing a predetermined operation in accordance with the guidance that changes in time series by executing the script. The multimedia content desired by the user can be easily created.

  Furthermore, by sending such a template to another mobile phone 1 and requesting a reply, it is possible to perform a desired operation on the sender side on the sender side. For example, by sending a text message for requesting insertion of a new video to the template “substantial time” as shown in FIG. 21 to another mobile phone 1, the user of the mobile phone 1 is notified. On the other hand, it is possible to instruct execution of editing work for inserting new moving image data into the template.

[Third Embodiment]
Next, a case where the multimedia data processing apparatus is a television receiver (hereinafter abbreviated as a television) will be described as a third embodiment. That is, the template according to the third embodiment describes the layout and time information of the television screen, and the television program is handled as one of the media data.

  FIG. 74 is a diagram showing a configuration of a data communication system including a television which is a multimedia data processing apparatus in the third embodiment. Referring to FIG. 74, the data communication system provides television 6 as a multimedia data processing device, template providing server 2 that provides a template described in a script language, and television program information and program related information. The EPG server 3 is configured to include the WWW server 4 that provides Internet content, and the broadcast station 5 that provides programs to the television 6 via broadcast waves.

  The configuration of television 6 as a multimedia data processing device is similar to the configuration shown in FIG. 4 in the first embodiment, and therefore description thereof will not be repeated here. Furthermore, the television 6 includes a main area 7401 and sub areas 7402 and 7403 in the display unit.

  75, 76, and 77 show specific examples of scripts included in a template that is handled by the television 6 that is the multimedia data processing apparatus according to the third embodiment. FIG. 75 is a specific example showing a script including a drawing information description part that refers to media data, and FIG. 76 is a specific example showing a script including a program guide information description part that describes program information according to a time zone. FIG. 77 is a specific example showing a script including a description relating both scripts shown in FIGS. 75 and 76. The script shown in FIG. 76 corresponds to the script including the user navigation information description part in the first embodiment, but in this embodiment, the operation of the user who views the television 6 is performed. Is a script that plays a role of automatically determining a program to be displayed and reproduced according to a time zone. Program guide information is obtained by analyzing the script.

  Next, the flow of data processing in the television 6 is shown. Regarding the script reading and execution process and the event process, the general flow is the same as the process shown in FIG. 9 and will be described with reference to FIG.

  Referring to FIG. 9, first, television 6 acquires each script shown in FIGS. 75, 76, and 77 (S12), and reads these scripts (S13). Further, the syntax of each read script is analyzed (S14), and a time table is created (S15). Thereafter, the script is executed according to the created time table (S16).

  More specifically, since the image file on the WWW server is designated in lines 14 and 15 in FIG. 75, the corresponding file is acquired from the WWW server 4 in accordance with the description in step S12 described above, and in step S16. Is displayed in the sub-areas 7402 and 7403 which are positions designated by the areas “right” and “left”.

  Also, line 13 in FIG. 75 describes that video content “tvcontents” is reproduced and displayed in main area 7401 (main). Such video content “tvcontents” is described in the script description shown in FIG. Therefore, it is determined movie data or TV channel. The description on line 5 in FIG. 76 indicates that the script shown in FIG. 76 specifies media data of a part whose id in the script shown in FIG. 75 is “tvcontents”.

  Next, the flow of determination processing for determining the video content “tvcontents” in the television 6 will be described with reference to the flowchart of FIG.

  Referring to FIG. 78, first, when it is analyzed that “video id =“ tvcontents ”” is specified in the drawing information description portion of the script on TV 6 (YES in S2301), FIG. 76 shows. The program information corresponding to the current time is acquired with reference to the program guide information which is the analysis result of the script (S2302).

  76, the program of the TV channel number 4 is shown from 0:00 to 2 o'clock, and the line 14 to 21 is shown as 20th from 2 o'clock to 4 o'clock. There is a description that the Internet content specified by the URL described in the row corresponds to the video content “tvcontents”. Therefore, if the current time is between 0 o'clock and 2 o'clock, the video content “tvcontents” is determined to be a TV program of channel number 4 from this description, so in step S2303 the type of the program information (TV The television 6 receives the corresponding broadcast wave from the broadcast station 5 and displays it at a predetermined position according to the determination result (S2304).

  On the other hand, if the current time is between 2 o'clock and 4 o'clock, the video content “tvcontents” is determined from this description to be the Internet content specified by the URL described in the 20th line of FIG. In step S2303, the type of the program information (which is Internet content) is determined, and the television 6 acquires the content data from the WWW server 4 and displays it at a predetermined position according to the determination result (S2305).

  In the script shown in FIG. 76, the program description portion after 4 o'clock is omitted, but there is a similar description for the time, and the same processing is performed even if the current time is other than the above time. It is.

  Through the above processing, in the television 6, the media data acquired from the WWW server 4 is reproduced and displayed in the sub areas 7402 and 7403, and the program table obtained by analyzing the script shown in FIG. 76 is displayed in the main area 7401. A television program or Internet content designated according to the time zone based on the information is acquired from the broadcast station 5 or the WWW server 4 and reproduced and displayed.

  In the present embodiment, the still image data acquired from the WWW server is displayed in the sub-areas 7402 and 7403. The display of such areas also includes the 14th and 15th lines in FIG. The description is the same as the description in the 13th line and combined with the description of the script shown in FIG. 76, or it is related to media data switching and events with time as described in the first and second embodiments. By adding a description, broadcast waves and program-related information are acquired and displayed from the EPG server 3, WWW server 4, and broadcast station 5 according to the time zone, and UI components that accept user operations are displayed and displayed. An action corresponding to a user operation performed on the UI component may be executed.

  Further, as such an action, information related to a program being played back in the main area 7401 by displaying a button such as TV channel switching, volume and brightness adjustment, and displaying a button such as “detailed information”. Link transition to WEB content that displays, or if the TV 6 has a recording function or is connected to a recording device, perform a recording operation, record a screen shot of the screen, and send it to another user For example, when the telephone function is linked, a call is made to a predetermined number.

  Since the data communication system including the television 6 which is the multimedia data processing apparatus in the present embodiment has the above-described configuration, it is mainly for content producers that combine television programs and Internet content. There is an advantage that content data can be created easily.

  In the first to third embodiments and the second embodiment, the template provided from the server 2 is processed by the mobile phone 1 or the television 6 whose specific example is shown in FIG. However, the processing device for processing the template is not limited to a device such as the mobile phone 1 in which functions are integrally configured. That is, each function constituting the mobile phone 1 and the television 6 includes an acquisition device that acquires a template such as a receiver, a media data input device such as a camera that acquires an image, and a media data output device such as a television receiver. Each may be an independent device. Even in such a case, data and control signals are exchanged by performing communication via a dedicated line or wireless communication with each other, and the above-described script reproduction and editing processing is executed in the data processing device, and the acquisition device It is possible to edit the multimedia content using the media data or the like input by the media input device by processing the script acquired in step (b). Then, the template acquired by the media data output device or multimedia data obtained by editing the template can be reproduced.

  Furthermore, a script execution method for executing such a script can be provided as a program. Such a program is stored in a computer-readable recording medium such as a flexible disk attached to the computer, a CD-ROM (Compact Disk-Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), and a memory card. And can be provided as a program product. Alternatively, the program can be provided by being recorded on a recording medium such as a hard disk built in the computer. A program can also be provided by downloading via a network.

  The provided program product is installed in a program storage unit such as a hard disk and executed. The program product includes the program itself and a recording medium on which the program is recorded.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a figure which shows the structure of the data communication system in this Embodiment. It is a block diagram which shows the specific example of the hardware constitutions of the mobile telephone 1 which is a reproduction | regeneration / editing apparatus contained in a data communication system. FIG. 4 is a diagram showing a program stored in a storage unit 130 of the mobile phone 1 and a data processing flow in the mobile phone 1. 3 is a block diagram showing a functional configuration of a playback / editing engine program 131. FIG. It is the figure which showed the data structure of the script typically. It is a figure which shows the specific example of a script. It is a figure which shows the specific example of a script. It is a figure which shows the specific example of the table which prescribes | regulates correspondence with a starting application and UI displayed at that time. 4 is a flowchart showing script playback and editing processing in the mobile phone 1; It is a figure which shows the specific example of the time table of a media data drawing process. It is a figure which shows the specific example of an event processing time table. It is a figure which shows the specific example of UI allocation status display time table. It is a flowchart which shows the process according to the event performed by step S18. It is a flowchart which shows the determination process of the processing content in step S24. It is a flowchart which shows the menu process in step S25. It is a figure which shows the specific example of a menu screen. It is a flowchart which shows the link movement process in step S27. It is a flowchart which shows the application execution process in step S26. It is a flowchart which shows the process at the time of performing the 20th line of the script shown by FIG. It is a flowchart which shows the process at the time of performing the 23rd line of the script shown by FIG. It is a figure which shows the specific example of the transition of the display screen at the time of script execution. It is a figure which shows the specific example of the transition of the display screen at the time of script execution. 6 is a flowchart showing script preview processing in the mobile phone 1; It is a flowchart which shows the application starting process in step S62. It is a flowchart which shows the data acquisition process in step S64. It is a flowchart which shows the script rewriting process in step S66. It is a flowchart which shows the event time table reference process in step S33. It is a figure which shows the specific example of the table which prescribes | regulates a correspondence with a logic symbol and a hardware key. It is a figure which shows the specific example of a script. It is a figure which shows the specific example of a script. It is a flowchart which shows the process at the time of performing the 19th line of the script shown by FIG. It is a flowchart which shows the process at the time of performing the 21st line of the script shown by FIG. It is a figure which shows the specific example of the transition of the display screen at the time of script execution. 4 is a flowchart showing script mail attachment processing in the mobile phone 1; 6 is a flowchart showing a script slide deletion process in the mobile phone 1; It is a figure which shows the specific example of the menu screen at the time of a slide deletion process. 6 is a flowchart showing a script slide rewrite prohibition process in the mobile phone 1; It is a flowchart which shows the event process in step S96. It is a figure which shows the specific example of the menu screen at the time of a preview process. It is a figure which shows the specific example of the screen transition at the time of a preview process. It is a figure which shows the specific example of the display screen before template rewriting, and the screen transition after template rewriting. It is a block diagram which shows the function structure of the reproduction | regeneration / editing engine program 131 in the form of a 2nd Example. It is a flowchart which shows the process in a reproduction | regeneration / editing engine program. It is a flowchart which shows the data output process in step S1201. It is a figure which shows the specific example of the screen transition of rewriting by a camera starting application. It is a flowchart which shows the reproduction | regeneration and edit process in step S1101. It is a figure which shows the specific example of a script. It is a block diagram which shows the function structure of the reproduction | regeneration / editing engine program 131 in the modification of the form of a 2nd Example. It is a figure which shows typically that the function selected even if it is the same script according to a terminal differs. It is a figure which shows the specific example of the script used with the terminal by the side of preparation. It is a figure which shows the specific example of the thumbnail display for every scene. It is a figure which shows the specific example of the screen transition of rewriting by an inline camera application and an inline text input application. It is a figure which shows the specific example of the screen transition of rewriting by a file acquisition application. It is a figure which shows the specific example of the screen for selection for receiving selection of the process started from now among several processes. It is a figure which shows the specific example of the table which hold | maintains a response | compatibility with the transmitted message and the template used for creating a message. It is a figure which shows the example which synthesize | combines the media data and acquisition data which are referred with the reference information of a script at the time of the time parameter value of an event. It is a figure which shows the example which writes acquisition data at the time of the time parameter value of an event. It is a figure which shows the specific example of a switching table. It is a flowchart which shows the determination process in step S1110. It is a figure which shows the specific example of the table which records a response | compatibility with the data acquisition program stored in the mobile telephone 1, and the media type which the mobile telephone 1 supports. It is a figure which shows the specific example of the script used with the terminal by the side of preparation. It is a flowchart which shows the data output process in step S1201. It is a figure which shows the specific example of an action management table. It is a figure which shows the specific example of a scene event table. It is a flowchart which shows the change process in step S1205. It is a flowchart which shows the image pick-up application starting process in step S102. It is a flowchart which shows the process which determines the restart time of script reproduction | regeneration. It is a figure which shows the specific example of the script which described whether slide edit was permitted or prohibited. It is a flowchart which shows the process which determines permission to perform edit of a slide by step S42. It is a figure which shows the specific example of the script containing a drawing information description part in the case of dividing | segmenting and comprising a script into a drawing information description part and a user navigation description part. It is a figure which shows the specific example of a script containing a user navigation description part in the case of dividing | segmenting and comprising a script into a drawing information description part and a user navigation description part. It is a figure which shows the specific example of the screen transition at the time of executing a script. It is a figure which shows the specific example of the script containing the description which relates two scripts. It is a figure which shows the structure of the data communication system comprised including the television which is the multimedia data processing apparatus in 3rd Embodiment. It is a figure which shows the specific example of the script containing the drawing information description part which describes drawing information in 3rd Embodiment. It is a figure which shows the specific example of the script containing the program schedule information description part which describes the program information according to a time slot | zone in 3rd Embodiment. It is a figure which shows the specific example of the script containing the description which relates two scripts. It is a flowchart which shows the flow of the determination process of the content which should be received and reproduced | regenerated. It is a flowchart which shows the flow of a reproduction | regeneration and edit process in case a script is described by SMIL. It is a figure which shows the specific example of the data structure which stores the analysis result of a script. It is a figure which shows the specific example of the analysis result of a script. It is a figure which shows the specific example of the analysis result of the updated script. It is a figure which shows the specific example of the script which performed the rewriting process which adds media data. It is a figure which shows the specific example of the script which deleted the edit function after performing the rewriting process which adds media data. It is a figure which shows the specific example of the transition of the display screen at the time of script execution.

Explanation of symbols

  1 mobile phone, 2 navigation frame providing server, 3 EPG server, 4 WWW server, 5 broadcast station, 6 TV, 110 communication unit, 120 control unit, 130 storage unit, 131 playback / editing engine program, 132 UI content display program, 133 Image shooting program, 134 Image data display reproduction program, 135 sound data reproduction program, 136 voice data recording program, 137 animation data display reproduction program, 138 text data display reproduction program, 139 text editing program, 140 input / output unit, 142 key Code input device, 144 display, 146 microphone, 148 speaker, 149 camera, 150 GPS, 151 WWW browser program, 152 mail program, 1 3 Call program, 154 Location acquisition program, 155 Address book program, 156 File acquisition program, 1000 Template storage unit, 1001 selection unit, 1002 Script data temporary storage unit, 1003 Script execution unit, 1004 User change data determination, 1005, 1012 Terminal function selection unit, 1006 terminal function execution unit, 1007 data change unit, 1008 script / data change confirmation unit, 1009 script / data output unit, 1010 output script / data storage unit, 1011 data embedding unit, 1013 action determination unit, 1014 Action execution unit, 1015 reception unit, 1311 script reading unit, 1312 syntax analysis unit, 1313 execution control unit, 1314 time management unit, 1315 event management unit, 1 16 script written out section, 7401 the main area, 7402,7403 sub-region.

Claims (17)

A template acquisition means for acquiring a template including a script from a storage device inside the multimedia data processing device or another device via a network;
Selecting means for selecting a desired template from the obtained templates;
Script execution means for playing back data referenced from the script according to the script included in the selected template;
Sequence management means for managing a sequence described in the script;
Of the data referenced from the script, user changeable data determination means for determining a user changeable portion;
Changeable data presenting means for indicating data changeable corresponding to the sequence;
Data changing means for changing the changeable part;
Data change confirmation means for confirming the change of the data;
A multimedia data processing apparatus comprising script data output means for outputting at least one of the script and the changed data. The multimedia data processing apparatus according to claim 1, wherein transition information of a media data playback state is described in the template, and the script execution unit manages transition of the media data playback state. The script execution means acquires the transition of the media data reproduction state according to the transition information of the media data reproduction state described in the template, and the action determination means acquires an event using the selected function. The multimedia data processing apparatus according to claim 2 , wherein Wherein are described with transition information of the media data playback state time constraints on the template, the script executing means, transition temporally manage the media data reproducing state, according to any one of claims 1 to 3 Multimedia data processing device. Transition information of media data playback state is described in the template,
Thumbnail generation means for generating thumbnails of media data included in the template;
The thumbnails the generating further comprises a preview means for displaying on the basis of the transition information, the multimedia data processing device according to any one of claims 1-4. In accordance with the script, further comprising limiting means for limiting a change of the data, the multimedia data processing device according to any one of claims 1-5. The script execution means further comprises time selection means on the medium for determining and selecting a place where execution or stop can be performed among data referenced from the script,
The script execution unit, when executing said selected portions of said data, pausing at the points, the multimedia data processing device according to any one of claims 1-6. The multimedia data processing according to any one of claims 1 to 7 , wherein when the data change operation is performed, a start position or an end position of the changed section of the data is added to a predetermined position of the data. apparatus. A program for causing a computer to execute a template process including a script,
A template acquisition step of acquiring the template from a storage device inside the multimedia data processing device or another device via a network;
A selection step of selecting a desired template from the acquired templates;
A script execution step of reproducing data referenced from the script according to the script included in the selected template;
A sequence management step for managing a sequence described in the script;
Of the data referenced from the script, a user changeable data determination step for determining a user changeable portion;
A changeable data presentation step for presenting data that can be changed in correspondence with the sequence;
A data changing step of changing the changeable part using a function of a multimedia data processing device;
A data change confirmation step for confirming the change of the data;
A multimedia data processing program for executing a script data output step for outputting at least one of the script and the changed data. 10. The multimedia data processing program according to claim 9 , wherein transition information of media data playback state is described in the template, and the script execution step manages transition of the media data playback state. The multimedia according to claim 9 or 10 , wherein transition information of a media data reproduction state is described in the template together with a time constraint, and the script execution step temporally manages the transition of the media data reproduction state. Data processing program. Transition information of media data playback state is described in the template,
A thumbnail generation step of generating thumbnails of media data included in the template;
The thumbnails said generating, said transition further to execute a preview step of displaying based on the information, the multimedia data processing program according to any of claims 9-11. A template acquisition step of acquiring a template including the script;
A selection step of selecting a desired template from the acquired templates;
The multimedia data processing program according to any one of claims 9 to 12 , further comprising a data change confirmation step for confirming the change of the data. The multimedia data processing program according to any one of claims 9 to 13 , wherein a restriction step of restricting the change of the data is further executed according to the script. Before SL corresponding to the sequence to perform another operation guide presenting steps of presenting the operation guide for performing data changes, the multimedia data processing program according to any of claims 9-14. The script execution step further executes a time selection step on the media for determining and selecting a portion that can be executed or stopped from the data referred to by the script,
The multimedia data processing program according to any one of claims 9 to 15 , wherein the script execution step pauses at the location when executing the selected location of the data. The multimedia data processing according to any one of claims 9 to 16 , wherein when the data change operation is performed, a start position or an end position of the changed section of the data is added to a predetermined position of the data. program.

Images