FR2864406A1 - Audio or video flow sub-titling method, involves coding control flow containing sub-title part, as resultant flow by production server, such that resultant flow is transmitted to broadcasting and web servers and is called by user terminal - Google Patents

Abstract

The method involves generating a part of a sub-title (PST) of an audio or video flow by a sub-title generator (GS). The part is transmitted to a production server (SP) and is inserted in a control flow (FC). The control flow is coded as a resultant flow (FR) by the server (SP), such that the resultant flow is transmitted to broadcasting and web servers (SD, SW) and is called by a user terminal (T) in a multi-media document (DM). An independent claim is also included for a system of sub-titling in real time an audio stream in a multimedia document requested by a user terminal.

Description

2864406 1

  The present invention relates to a method and a system for subtitling in real time an audio or video stream in a multimedia document.

  Currently, the captioning of a video clip broadcast in a rich media rich media document is completely embedded in the video frames. Therefore it is impossible to act only on the subtitle independently of the video.

  The subtitle of the video sequence is usually produced by an input operator. A high-speed keyboard, called the "Velotype" keyboard, combines the advantages of shorthand and typing based on syllabic writing. The operator greatly increases the input speed with 37 keys divided into three groups initial consonants, vowels and final consonants. The operator produces a syllable by simultaneously striking several keys. A computer part of the keyboard places the characters of the syllable in the right order.

  The present invention aims to broadcast in real time from a multimedia document at least one subtitle of a continuous audio or video stream so that the subtitle is an independent multimedia object and thus capable of specific processing as as a modification of the font of the subtitle characters.

  An advantage of the invention is the generation of the subtitle on a geographical site remote from that of the transmitter of the audio or video stream and those where the generated subtitle is processed.

  To achieve this objective, a method for subtitling in real time an audio stream produced by a source in a multimedia document requested by a user terminal through a network from a server means comprising a call of at least one resulting stream in the multimedia document is characterized in that it comprises the following steps: generating a subtitle portion of the audio stream in real time in a subtitle generating means receiving the audio stream of the source; transmitting at least a portion of subtitle generated to a means of production; inserting at least a portion of subtitle transmitted into a control stream in the production means; and coding the control flow into the resulting flow in the production means so that the resulting flow is transmitted to the server means and called by the user terminal in the multimedia document.

  The invention thus generates in real time a subtitle of an audio or video stream and modifies in real time a control stream comprising the subtitle generated so that after decoding the resulting stream called, the subtitle generated and contained in the stream resulting in the terminal requesting the multimedia document.

  The subtitling method may further include encoding and multiplexing the audio stream and the control stream into the resulting stream, or multiplexing the audio stream and control stream into the resulting stream that is encoded, in the generating means that receives the audio stream from the source, and demultiplexing and decoding resulting streams into coded audio and control streams that are decoded, or decoding the resulting stream that is demultiplexed into the audio stream and the stream in order to reproduce the audio stream simultaneously with the playback of the subtitle portion; or encoding and multiplexing a video stream including the audio stream and an image stream and the control stream into the resulting stream, or multiplexing the video stream and the control stream into the resulting stream that is encoded, in the production means that receives the video stream from the source comprising the audio stream and the image stream and demultiplexing and decoding the resulting stream into coded video and control streams that are decoded, or decoding the resulting stream which is demultiplexed into the video stream and the control stream, to reproduce the video stream simultaneously with the playback of the subtitle portion.

  The generation of a subtitle portion includes either manual subtitling of the audio stream using a high-speed input keyboard, or automatic audio stream subtitling based on speech recognition of the stream. audio.

  Generation of a subtitle portion may also include at least one translation of the subtitle portion, into a translated subtitle portion transmitted to the production means for insertion into the control stream.

  The sub-title portion transmitted to the production means may include a speaker identifier of a portion of the audio stream corresponding to the subtitle portion.

  The invention also relates to a system for subtitling in real time an audio stream produced by a source for carrying out the method of the invention.

  The system is characterized by comprising: lo - means receiving the audio stream of the source for generating at least a subtitle portion of the audio stream in real time; production means for inserting at least a portion of subtitle transmitted by the means for generating it in a control flow; and - means for encoding the control stream comprising at least a subtitle portion into the resulting stream transmitted to the server means and called by the user terminal in the multimedia document.

  The system may further include means connected to the means for encoding and the user terminal through said network and for broadcasting to the terminal the resulting stream called, means in the terminal for decoding the resulting stream called broadcast and means in the terminal to play at least the subtitle portion inserted in the resulting stream called broadcast in the multimedia document.

  In addition, at least two of the following elements are gathered in a single element. the source, the means for generating, the means of production associated with the means for coding, the server means.

  Other features and advantages of the present invention will become more apparent to the

  reading the following description of several

  preferred embodiments of the invention, by way of nonlimiting examples, with reference to the corresponding appended drawings in which: - Figure 1 is a schematic block diagram of a subtitling system according to an embodiment of the invention ; and FIG. 2 is a schematic algorithm of the subtitling method according to the embodiment of the invention.

  It is assumed that a video stream includes a multiplexed image stream and audio stream.

  A user terminal typically requests from a web server a multimedia document DM. As indicated above, the object of the invention is to incorporate at least one subtitle of an audio or video stream into the multimedia document.

  In the invention, the multimedia document DM is for example coded according to the Javascript language and the HTML (HyperText Markup Language) or DHTML (Dynamic HyperText Markup Language) language. The multimedia document may include other elements than the subtitle, such as static images, animations, and sounds.

  With reference to FIG. 1, the subtitling system implementing the subtitling method according to a preferred embodiment of the invention mainly comprises an audio or video source S, at least one terminal T of a US user, at least one GS subtitle generator, an SP production server, an SD broadcast server and a SW web server.

  The source S, the terminal T and the generator GS are connected to an access network RA through telecommunications links LT. For reasons of simplicity, the source S, the terminal T and the generator GS are supposed to be connected to a common access network, whereas in reality they can be connected to different access networks interconnected by interconnected packet networks. Each link LT and the network RA can be conventionally a telephone line and the switched telephone network RTC itself connected to an Internet-based high speed packet transmission network RP. According to other variants, the telecommunications link LT is an xDSL line (Digital Subscriber Line) or an ISDN line (Digital Integrated Services Network) connected to the corresponding access network.

  The SP, SD and SW servers are connected directly to the RP packet network.

  The user terminal T is for example a personal computer connected directly by modem to the link LT. According to other examples, the user terminal T comprises an electronic device or object for personal telecommunications to the user which may be a personal digital assistant PDA. The terminal T may be any other portable or non-portable home terminal such as a microcomputer, telephone, video game console, etc. The terminal T is served by a telephone line and the switched telephone network, as a telecommunications link LT and RA access network, to be connected to the packet transmission network.

  In another example, the user terminal T is for example of the smart TV receiver type. The television receiver cooperates with a remote control display and alphanumeric keyboard also serving as a mouse through an infrared link. Alternatively, the remote control is completed by a more complete wireless keyboard connected by radio to the television.

  According to yet another example, the terminal T is a mobile cellular radio terminal, the telecommunications link LT is a radio communication channel, and the access network RA is the fixed network of a radio communication network, for example of the type GSM (Global System for Mobile Communications) with a GPRS service (General Packet Radio Service), or UMTS type (Universal Mobile Telecommunications System).

  For the sake of clarity, a single user terminal is shown in Figure 1, while a multitude of user terminals may request the multimedia document DM within the saturation limits of the system.

  User terminals and access networks are not limited to the above examples and may be constituted by other known terminals and access networks.

  The user terminals are equipped with a multimedia media player LM which is for example the "Windows Media Player" of Microsoft Corporation. There is no distinction between an internet browser used for the call of a multimedia document and the media player for reading and displaying the streams, and these two entities are considered in the multimedia player LM.

  The audio or video source S delivers in real time and continuously a FA or analog or directly digital FV video stream. The source S is composed of an audio or video data acquisition system and a terminal connected to the access network RA and providing audio or video data exchanges. When the delivered FA audio or FV video is analog, an analog-to-digital converter converts the analog stream into a digital stream. The digital stream is then coded so that the source S sends the coded stream to the subtitle generator GS and the production server SP through the networks RA and RP.

  The subtitle generator GS comprises for the implementation of the invention in particular a high-speed input keypad Velotype CVE, a seizure operator VE trained to use the keyboard Velotype, an HP speaker or any other means of playback of an audio stream and a subtitling terminal TS.

  The source S and the generator GS are connected through the access network RA. If the stream coming from the source S is coded, the caption terminal decodes it. When the stream coming from the source S is a video stream, the generator GS extracts an audio stream FA and an image stream F1 from the video stream FV generally by demultiplexing the video stream. The image flow FI extracted from the video stream FV is processed to be displayed on the screen of the terminal TS of the input operator VE.

  With the Velotype CVE keypad, the VE operator inputs text corresponding to the speech or speech of the speaker included in the audio stream output by the HP speaker through an input interface of the word processor type. The text entered by the operator is the subtitle ST of the audio or video stream. The terminal TS transmits the subtitle ST to the production server SP through the networks RA and RP.

  In one variant, the subtitling terminal TS comprises two distinct terminals, one ensuring the reception of the stream coming from the source S and the other the transmission of the subtitle ST to the production server SP. The two terminals are, for example, a mobile cellular radio terminal for receiving and a computer for transmitting.

  In another variant, the subtitle generator is an automatic subtitle generation system based on speech recognition of speech or speech included in the audio stream FA.

  In another variant that can be combined with the previous variant, the subtitle generator manually translates via the operator, or automatically the subtitle in a language different from the language of a word or words in the audio stream.

  In another variant, the subtitling system comprises several subtitle generators GS each subtitling in a different language, these generators can be manual or automatic. The automatic generators then include voice recognition coupled with a translation.

  The production server SP comprises in particular a processing unit UT and an audio-video encoder EC implementing a control flow.

  If an incoming stream FA or FV is already coded by the source S, the coder EC decodes or acts directly on the coded incoming stream. In the case of an incoming FV video stream, the EC encoder demultiplexes the video stream into an audio stream FA and an image stream FI.

  The EC coder according to the invention also implements a complementary stream, called the FC control stream, which includes commands which are notably related to the temporal and content characteristics of the coded audio or video stream delivered in real time. The control stream FC carries, in addition to commands, data of a type other than audio or video data, for example text data. Commands synchronize this data of another type with each other and with the audio and / or video streams. In the present invention, the data of another type is the generated subtitle ST entered by the operator VE and the audio or video streams can carry no data.

  The audio-video encoder EC encodes the audio stream FA or video FV and the control stream FC for a real-time or non-real-time broadcast of coded streams according to configuration parameters such as a coding type, a compression type, a bandwidth, frame rate, image resolution, etc. The EC audio-video encoder can be analogous to the Windows Media Encoder from Microsoft Corporation implementing a "Script Command" command stream, or the RealNetworks RealNetworks Producer encoder implementing a "RM Events" command flow.

  The streaming server SD, called streaming server, receives from the coder EC in the production server SP a stream FR resulting from the multiplexing of the coded audio or video stream and the coded control stream. The SD server is intended to broadcast continuously a resulting stream broadcast FRD composed of coded and multiplexed audio or video streams and control. The FRD stream is broadcast in real time or offline so that the terminal of the user T continuously downloads and displays the contents of the streams composing the FRD stream.

  As seen above, the encoding of the FA or FV and FC flows is not performed by the broadcast server SD but by the production server SP.

  The web server SW is a classic web server responding to requests from user terminals by sending the DM multimedia document through the RA and RP networks. Other elements such as statistical images, animations and sounds can be stored in a memory of the web server.

  In another variant, the source S and the production server SP are nearby and connected by a wireless short-range radio link type WiFi or Bluetooth, or a conventional cable connection. In this variant, the audio or video stream is not necessarily subject to coding / decoding between the source and the production server. The source S and the production server are then considered as gathered in a single element.

  In yet another variant, the source S and the GS subtitle generator are nearby and connected by a wireless short-range radio link type WiFi or Bluetooth, or a conventional cable connection. In this variant, the audio or video stream is not necessarily subject to coding / decoding between the source and the production server. The source S and the subtitle generator GS are then considered as gathered in a single element.

  The two previous variants can be combined together.

  The subtitling method mainly comprises steps E0 to E12 shown in FIG.

  In step E0, an administrator of the production server SP initializes the configuration parameters of the EC encoder.

  In step E1, the subtitling terminal TS and the production server SP connect to the source S. In step E2, the source S continuously transmits in real time the audio stream FA or video FV. subtitling terminal TS and the production server SP.

  After step E2, the processing unit UT applies in step E31 coder status requests ReqC every N seconds to the coder EC as long as the site SP is in use to determine a current state EtC of the coder . N is a predetermined integer, for example equal to 1.

  If the encoder EC does not respond, then the current state of the encoder is said to be inactive N. If the encoder is active, it responds to a request ReqC by transmitting the current state EtC in a RepC response to the processing unit UT in step E32.

  The possible states of the encoders are: - inactive N, - ready to code P, - during coding C. The distinction between the state ready to code P and the state being coded C gives more precision to the VE operator who can, if necessary, notify the administrator of the production server SP that the coder EC is ready to code when it should be coding. But this distinction is not indispensable.

  Following step E2 and in parallel with steps E31 and E32, the subtitling terminal TS connects to the production server SP in step E41.

  In step E42, the terminal TS transmits ReqP production status requests to the production server SP every M seconds to determine a current production state EtP of the production server SP. M is a predetermined integer, for example of the order of ten.

  In step E43, the processing unit UT responds to a request ReqP by transmitting to the terminal TS a RepP response including the current production state EtP equivalent to the current state of the encoder EtC, determined regularly in steps E31 and E32 every N seconds.

  Following the cyclical steps E31-E32 and E42-E43, the subtitling terminal TS informs in step E5 the operator VE of the current state of production EtP by an alert message displayed on the screen of the terminal TS.

  When the current state of production is the state ready to code P, the operator VE starts or continues to enter the subtitle ST simultaneously, in step E6. When the current state of production is different from ready to code P, the operator VE waits for the next message in step E51 until a new message informs him that the current state of production is the same. ready state to code P. The operator VE decides at the moment of his choice 35 the transmission of a last part of subtitle PST input to the production server SP by the subtitling terminal TS by actuating a particular key from his keyboard approximately every X seconds in step E7. The last part of the PST input subtitle is the subtitle entry part since the last subtitle part was sent. In one variant, the subtitle portion is sent following an entry of a delimiter character by the operator, such as a comma or a point. In another variant, the captioning terminal automatically sends every X seconds a portion of subtitle PST, X being a predetermined integer, for example of the order of 5.

  The subtitling terminal TS continuously stores in a buffer the subtitle entered by the operator VE. Each part of the subtitle PST is transmitted by the subtitling terminal TS according to the HTTP protocol (HyperText Transfer Protocol) in the form of a small file based on the syntax of the language XML (eXtensible Markup Language). The small file may include a "language" tag to specify the language in which the audio stream FA is subtitled.

  The production server SP thus receives in real time, on the one hand, the audio stream FA or video FV and, on the other hand, the subtitle parts entered PST at least in one language.

  On receipt of a portion of the current PST subtitle, the processing unit UT stores in a buffer memory of the production server SP the current subtitle part, possibly according to its language, and applies commands necessary to the encoder EC to insert in the FC control stream the current subtitle portion in step E8.

  In a variant, the processing unit UT performs a processing on at least a portion of the subtitle PST stored before applying the commands. The processing is for example a reconstitution of complete sentences from at least part of a subtitle and / or a translation of the subtitle part and / or an orthographic correction of the subtitle part, etc. The encoder EC continuously codes and multiplexes the audio stream FA, or the audio stream and the image stream FI included in the video stream FV and the control stream FC in step E9, in real time, so that the production server SP stores and transmits a resultant stream FR to the broadcast server SD in step E10. The resulting stream FR does not always include a component from the control stream. The component stemming from the control stream is present in the resulting stream, particularly if a part of subtitle PST is available to be transmitted to the terminal T, that is to say if the operator VE has commanded the transmission of a part of the subtitle to the SP server.

  In general, the insertion of the subtitle in the control and coding flow and the multiplexing of the streams are done in this order, but can be ordered differently according to the characteristics of the EC encoder.

  The resulting stream FR is stored in a database of the broadcast server SD in order, for example, to store it for a video on demand (VOD) video on demand service. Prior to step E10, the user terminal T conventionally calls a multimedia document DM in the web server SW by connecting to the web server SW and requesting the multimedia document DM for example in a form sent by the server SW. In response to the call of the resulting stream in the DM document, the resulting stream to broadcast FRD is transmitted continuously and in real time by the broadcast server SD to the user terminal T through the networks RP and RA to the network. step E1l. The broadcast server may contain different resulting streams containing subtitles of the audio stream FA in different languages.

  The multimedia player LM of the user terminal T displaying the multimedia document DM demultiplexes the resulting broadcast stream FRD, decodes each demultiplexed stream, then reads each decoded stream called by the code of the multimedia document DM on the user terminal T at the same time. step E12. The player may also display other document elements such as static images, animations, and sounds. The subtitle can thus be advantageously handled independently of the audio stream or the video stream. When the control stream includes the subtitle, the player displays the subtitle with a specific code contained in the multimedia document.

  This specific code is generally a code in Javascript language contained in the HTML or DHTML code of the multimedia document DM. The subtitle is then displayed in the multimedia document by the reader, according to the configuration parameters of the reader chosen by the user of the terminal T. In a restricted variant, the source S transmits the audio stream FA or the video stream FV to the subtitle generator GS without transmitting it to the production server SP which inserts the subtitle portions PST into the control stream FC and which only transmits the control stream as the resultant stream FR, without audio or video stream, to the broadcast server SD.

  The US user of the terminal T then reads the subtitle parts displayed in the terminal without any reproduction of audio or video streams.

  In the variant of several subtitle generators each generating subtitles in different languages, the US user of the terminal can choose the subtitle language independently of the language of the audio stream FA.

  In other embodiments, at least two of the servers SP, SD and SW are combined into a single server. According to another architecture of the subtitling system, the production server, the broadcast server and the web server are gathered in a single server.

  In another embodiment, the subtitle generator introduces, by automatic speaker recognition or by the input operator, into the XML file transmitted to the production server the identification of the speaker at the origin of the part of the audio stream. corresponding to a respective subtitle part, ie the name and / or the first name and / or any identifier of the speaker. This makes it possible to specify to the US user of the terminal T the speakers of the subtitle parts.

  In another embodiment, when the video stream comprises only one image stream, the subtitle generator is then a comment generator and the operator VE a commentator of the images of the video stream.

Claims (1)

18 Claims   1 - Process for subtitling in real time an audio stream (FA) produced by a source (S) in a multimedia document (DM) requested by a user terminal (T) through a network (RA, RP) from server means (SD, SW) and comprising a call of at least one resultant stream (FR, FRD) in the multimedia document, characterized in that it comprises the following steps: - generation (E6) of sub-part title (PST) of the real-time audio stream in a subtitle generation means (GS) receiving the audio stream from the source; - transmission (E7) of at least a portion of subtitle generated to a means of production (SP); inserting (E8) at least one sub-title portion transmitted into a control flow (FC) in the production means; and - encoding (E9) of the control flow (FC) into the resulting flow in the production means so that the resulting flow (FR, FRD) is transmitted to the server means (SD, SW) and called by the user terminal (T) in the multimedia document (DM).   2 - Process according to claim 1, comprising after a transmission (E10-Ell) of the resulting stream called (FR) to the terminal (T) a decoding (E12) of the resulting stream called in the terminal, and a reading (E12) of at least the subtitle part (PST) inserted into the resulting stream called in the multimedia document (DM).   3 - Process according to claim 1 or 2, comprising a coding and a multiplexing (E9) of the audio stream (FA) and the control stream (FC) in the resulting stream (FR), or a multiplexing of the audio stream and the control stream in the resulting stream which is encoded, in the production means (SP) which receives the audio stream (FA) of the source (S), and a demultiplexing and decoding (E12) of the resulting stream into audio streams and encoded control that are decoded, or a decoding of the resulting stream that is demultiplexed into the audio stream and the control stream, to reproduce the audio stream (FA) simultaneously with the playback of the subtitle portion.   The method according to claim 1 or 2, comprising encoding and multiplexing (E9) a video stream (FV) including the audio stream (FA) and an image stream (IF) and the control stream ( FC) to the resulting stream (FR), or a multiplexing of the video stream and the control stream to the resulting stream which is encoded, in the production means (SP) which receives the video stream (FV) of the source (S ), comprising the audio stream (FA) and the picture stream (IF), and demultiplexing and decoding (E12) the resulting stream into encoded video and control streams that are decoded, or decoding the resulting stream that is demultiplexed in the video stream and the control stream, to reproduce the video stream (FV) simultaneously with the playback of the subtitle portion (PST).   A method according to any one of claims 1 to 4, wherein the generation (E6) of a subtitle portion comprises manual subtitling of the audio stream (FA) using an ultra-fast input keyboard (CVE).   The method of any one of claims 1 to 4, wherein generating (E6) a subtitle portion includes automatic audio subtitling (FA) based on voice recognition of the audio stream.   The method according to any one of claims 1 to 6, wherein the generation (E6) of a subtitle portion (PST) comprises at least one translation of the subtitle portion, into a portion of translated subtitle transmitted to the production means (SP) for insertion into the control flow (FC).   The method according to any one of claims 1 to 7, wherein the subtitle portion (PST) transmitted to the production means (SP) includes a speaker identifier of a portion of the corresponding audio stream (FA). to the subtitle part (ST).   9 - System for subtitling in real time an audio stream (FA) produced by a source (S) in a multimedia document (DM) requested by a user terminal (T) through a network (RA, RP) from server means (SD, SW) comprising a call of at least one resulting stream (FR, FRD) in the multimedia document, characterized in that it comprises: - means (GS) receiving the audio stream from the source for generating at least a portion of subtitle (PST) of the audio stream in real time; - production means (SP) for inserting at least a portion of subtitle transmitted by the means for generating in a control stream (FC); and - means (EC) for encoding the control flow (FC) comprising at least a portion of the sub-title into the resulting stream transmitted to the server means (SD, SW) and called by the user terminal (T) in the multimedia document (DM).   System according to claim 9, characterized in that the user terminal (T) comprises means (LM) for decoding the called resultant stream and means (LM) for reading at least the subtitle part (PST). ) inserted into the resulting stream called in the multimedia document (DM).   11 - System according to claim 9, characterized in that it comprises a means (SD) connected to the means for coding (EC) and to the user terminal (T) through said network (RA, RP) and for broadcasting at the terminal (T) the resulting stream called (FR), means (LM) in the terminal for decoding the resulting stream called broadcast (FRD) and means (LM) in the terminal for reading at least the subtitle portion (PST) inserted in the resulting stream called broadcast in the multimedia document (DM).   12 - System according to any one of claims 9 to 11, wherein at least two of the following elements are collected in a single element: the source (S), the means for generating (GS), the means of production (SP) ) associated with the means for coding (EC), the server means (SD, SW).   13 - System according to claim 11, wherein at least two of the server means (SW, SD) for generating and coding (SP, EC) and for broadcasting (SD) are combined into a single server.