DE60133374T2 - METHOD AND DEVICE FOR RECEIVING HYPERLINK TELEVISION PROGRAMS - Google Patents

Abstract

A system and method of receiving information hyperlinked to a television broadcast. The broadcast material is analyzed and one or more regions within a frame are identified. Additional information can be associated with a region, and can be transmitted in encoded form, using timing information to identify the frame with which the information is associated. The system comprising a video source and an encoder that produces a transport stream in communication with the video source, an annotation source, a data packet stream generator that produces encoded annotation data packets in communication with the annotation source and the encoder, and a multiplexer system in communication with the encoder and the data packet stream generator. The encoder provides timestamp information to the data packet stream generator and the data packet stream generator synchronizes annotation data from the annotation source with a video signal from the video source in response to the timestamp information. The multiplexer generates a digital broadcast signal that includes an augmented transport stream from the transport stream and the encoded data packets. A receiver displays the annotation information associated with the video signal in response to a viewer request on a frame by frame basis. A viewer can respond interactively to the material, including performing commercial transactions, by using a backchannel that is provided for interactive communication.

Description

BACKGROUND OF THE INVENTION

The The present invention relates to the field of television broadcasting and in particular the field of hyperlinking in a television program.

transfers of information about Television signals are known in the art. TV shows are unidirectional and offer the viewer no way to interact with the material appearing on a television screen. Spectator reactions to displayed material using a Remote control are known, but are generally based on the choice of one be limited to a list of available transmissions.

In a publication titled "Hyperlinked Video ", published in the SPIE Minutes, Issue 3528, November 1998, by Dakss et al. is the hyperlinking of data to objects proposed that appear in the video. This publication describes how Objects within a video are tracked through multiple frames [frames] can be. The hyperlinking concept described therein, however, referred on the use of the system in a laboratory environment with a played locally, recorded video, so the video for example could be interrupted to display the link content. Alternatively, in the publication suggested to show the linked content after the video has reached an interruption in the plot. The publication does not address the issues of providing hyperlinked Data in the context of a broadcast video stream.

Emotional If you are in the field of broadcasting, you will get many additional ones technical problems. In the laboratory context, these problems occur as such not on, and the publication Although it describes that the system is a future scenario of television usage However, aspects of the broadcasting scenario did not materialize within which simulates the new technical problems as a result the restrictions existing broadcasting standards and bandwidths.

The present invention as defined in the appended claims to these problems.

BRIEF DESCRIPTION OF THE DRAWINGS

To the understanding The invention will now be an embodiment the same, by way of example only, with reference to the accompanying diagrammatic drawings Drawings described, wherein

1A - 1D represent a series of video frames as generated by the system of the invention;

2 Fig. 10 is a block diagram of one embodiment of a hyperlinked video system constructed in accordance with the invention;

2A a block diagram of the data flow in the in 2 shown embodiment of the system is and

2 B is a diagram of a set of masking packets;

2C is a diagram of an initial coded data packet stream;

2D is a diagram of a last encoded data packet stream;

3 a block diagram of an embodiment of the in 1 shown multiplexer system;

4 a block diagram of an embodiment of the in 2 shown digital receiver is;

5A - 5B Diagrams of one embodiment of the data structures are those of the system 2 used for storing annotation data;

5A Fig. 10 is a block diagram of an object property table data structure and a program allocation table data structure;

6 a state diagram of the data flow of an embodiment of the in 2 shown system;

7 the interactions between and among states within the 6 illustrated state machine of an embodiment of the invention;

8A to 8G schematically illustrate various illustrative examples of embodiments of an interactive content icon according to the invention;

9A to 9D illustrate illustrative embodiments of video image compression methods in accordance with the principles of the invention;

10A Fig. 3 shows an example area of a frame and an example mask used to describe a two-dimensional image in terms of mathematical morphology according to the invention be used;

10B shows an example result image of a two-dimensional mathematical morphology analysis and a single result pixel in accordance with the principles of the invention and

11A shows a sequence of example frames and an example mask used to describe a three-dimensional image in terms of a mathematical morphology according to the invention, using time as a dimension;

11B Figure 3 shows an example result framework of a three-dimensional mathematical morphology analysis using time as a dimension and a result pixel in accordance with the principles of the invention;

11C Fig. 10 is a flowchart showing an illustrative process according to an embodiment of the invention with which three-dimensional flood fill is achieved;

12A - 12B 5 is a diagram showing an example application of mathematical morphology analysis in accordance with the principles of the invention that generates an outline of a region; and

13 Figure 3 is a diagram showing three illustrative examples of the evolution of histograms over successive frames according to the invention indicating motion.

DESCRIPTION OF THE PREFERRED Embodiment

Briefly, the invention provides an approach for associating annotation information with objects displayed in the frame of a broadcast video and displaying them on the instruction of a viewer. For example, with respect to 1 Note information in the form of business, price and availability information with a particular shirt 2 that is worn by an actor on a TV show ( 1A ). To achieve this, the shirt becomes 2 first identified by a designer serving a part of the system called an authoring system for the system. The designer identified 3 the shirt 2 in a certain context, for example by coloring the shirt ( 1B ), and the system tracks the position of the shirt 2 in the previous and following frames. The designer also creates the text that makes up the shirt 2 associated annotation data 5 become. Thus, in this example, the annotation data may be the names of stores in which the shirt 2 can be bought, the price of the shirt 2 and the available colors include. The system then marks that with the shirt 2 Annotation data are associated, for example, by edges 4 of the shirt 2 within the frame in a different color ( 1C ).

When the broadcast is transmitted from the broadcast area of the system to a viewer, not only the video is transmitted, but also the mask, which is the shirt 2 edged, and the note data, the shirt 2 accompany. The receiver area of the system at the viewer's site receives this data and displays the video frames along with masks surrounding the objects having associated annotation data. In this example, the shirt is 2 outlined in the video frame. When the viewer of the transmitted video wishes to see the annotation data, he or she simply uses the control keys on a standard portable remote control to notify the receiver area of the system that the annotation data display is desired. The system then displays the annotation data 5 on the screen along with the object ( 1D ). In this way, marked objects serve as hyperlinks to further information.

In terms of 2 a hyperlinked video broadcasting system constructed according to the invention has a transmission range 10 , a communication channel area 12 and a reception area 14 on. The transmission range 10 has a video source 20 , an authoring tool 24 , a database storage system 28 and a transmitter 32 on. The video source 20 In various embodiments, a video camera, a video disc, a videotape or a video cassette, an input video signal or any other video source known to those skilled in the art. The authoring tool 24 , which is an annotation source, receives video data from the video source 20 and display them so that a video designer can view and manipulate them as described below. Annotation data, for example, the text to be displayed with the video image, is stored in an object database 28 stored and sent to the transmitter 32 sent to the communication channel area 12 to be sent to the system.

The transmitter 32 has a video encoder 36 , a data packet stream generator 40 and a multiplexer (MUX) 44 on top of the signals from the video encoder 36 and the data packet stream generator 40 for sending over the communication channel 12 combined. As is known to those skilled in the art, the video encoder can 36 each encoder for generating a transport stream, for example an MPEG or MPEG2 encoder. The data packet stream generator 40 encodes additional data, as described below, which should accompany the video data as it is sent to the viewer. Of the Data packet stream generator 40 generates coded data packets. The multiplexer 44 generates an extended transport stream.

The communication channel area 12 not only shows the transmission medium such. As cable, infrastructure for terrestrial transmission, microwave connection or satellite connection, but also each buffer that receives the video data until they are from the reception area 14 be received. Such caching may include video disc, tape or cartridge, memory or other storage devices known to those skilled in the art. The communication channel area also has the headend transmitter 50 which is provided by a multiple service provider.

The reception area 14 has a digital receiver 54 such as B. a digital set-top box, the signals for display on the television display device 58 decoded. At the hardware of the digital receiver 54 it is any known to the expert hardware for digital receiver 54 ,

In operation and also regarding 2 and 2A loads a designer video data 22 from a video source 20 in the authoring tool 24 , The video data will also be from the video source 20 to the video encoder 36 sent, for example, to be encoded using the MPEG standard. With the authoring tool 24 The designer selects portions of a video image to associate with screen annotations. For example, the designer could wear a shirt 2 Assigned on the video image by an actor, and assign annotation data to the maker of the shirt 2 , its purchase price and the name of a local dealer. On the other hand, annotation data may also contain additional text information about the object. For example, annotation data in a documentation could include biographical information about the person on the screen. The note data 5 ( 1D ) along with information about the shape of the shirt 2 and the position of the shirt 2 The image that is the mask image, as described below, will be called data structures 25 . 25 ' in a database 28 saved.

Once a designer has authored a particular program, the authoring tool determines the area over which objects appear and data structures are used in the annotated program. This information is used by the present inventive system to ensure that the data enabling viewer interactions with an object is sent before the object is presented to the viewer. This information is also used by the present inventive system to determine when data from a program is no longer needed and from the memory discussed below 128 can be deleted.

As described above, this annotation data also becomes for conversion into a coded data packet stream 27 to the data packet stream generator 40 cleverly.

Timestamp data in the transport stream 29 ' from the video encoder 36 are also an input to the data packet stream generator 40 and are used to synchronize the mask and annotation data with the image data. The data packet stream generator 40 accomplishes the synchronization by processing a program step by step and associating the time information of each video frame with the corresponding mask. Time information can be any type of information that allows synchronization of video and masking information. For example, the time information may be time stamp information as generated by an MPEG encoder, time code information as provided by the SMPTE time code standard for video, frame numbering information such as frame number information. For example, a unique identifier for a frame or a sequential number for a frame, the global time, and the like. In the present illustrative representation of the invention, time stamp information is used as an exemplary embodiment.

The encoded video data from the video encoder 36 be in a multiplexer 44 with the encoded data packet stream 27 from the data packet stream generator 40 combined, and the resulting extended transport stream 46 is an input to a multiplexer system 48 , In this illustrative embodiment, the multiplexer system is 48 able to handle additional transport 29 ' and extended transport 46 ' To receive streams. The transport 29 ' and extended transport 46 ' Streams include digitally encoded video, audio and data streams generated by the system or by other methods known to those skilled in the art. The output from the multiplexer system 48 is used to store and / or transmit to the communication channel 12 cleverly. The broadcast signal is sent to the digital receiver 54 sent and received by this. The digital receiver 54 sends the coded video portion of the multiplexed signal to the TV for display 58 , The digital receiver 54 Also takes instructions from a viewer who uses a portable remote to display any notes accompanying the video images. In one execution Form is the digital receiver 54 also directly in communication with an alternative network connection 56 ( 2 ).

In an alternative embodiment, information is from the object database 28 in a second database 30 in 2 for access via a network 31 such as As the Internet or directly into a database 33 transfer. In this embodiment, the head end engages 50 at the request of the viewer to the one in the second database 30 saved annotation object data. This arrangement is useful in cases where, for example, the viewer has recorded the program for later viewing and the recording medium can not record the annotation data, or when the data can not be sent in-band during the program. Thus, the digital receiver 54 when the recorded image through the digital receiver 54 is played and the viewer requests annotation data, the headend 50 Instruct the data over the network 31 retrieve. In addition, the headend would be 50 able to on the order of the digital receiver 54 Data in a database 33 in the network or in a headend database 52 to write. Such from the headend 50 Data written may be marketing data indicating which objects have been viewed, or it may be order information required by the viewer to order the displayed article over the network. A third embodiment combines attributes of the previous embodiments in that a portion of the information is included in the original transmission and a portion is retrieved in response to viewer requests.

The coded data packet stream 27 concerning and with reference to 2 B . 2C and 2D is in detail the data packet stream generator 40 is configured to generate a constant data rate stream despite variations in the size of the mask data and annotation data corresponding to the video frame. The data packet stream generator 40 achieves this through a process in three steps. First, the data packet stream generator determines 40 an acceptable range of packet rates included in the multiplexer 44 can be entered. Next, the data packet stream generator determines 40 the number of packets filled by the largest mask in the program being encoded. This defines the number of packets in each set of masking packets 39 ie the number of packets allocated for the transport of each mask. In the in 2 B . 2C and 2D example shown are in any amount of masking packets 39 eight packages. Using this number, the data packet stream generator generates 40 an initial version of the encoded data packet stream 27 ' where a fixed number of packets are allocated for each mask. If the number of packets required to accommodate a particular mask is less than the fixed number, the data packet stream generator buffers 40 the initial coded data packet stream 27 ' with zero packets. The number of null packets depends on the number of packets left after the mask data has been written. In 2C fill the mask data 42 for frame 1000, four packets, leaving four packets to be filled with null packets. Likewise are for the data 42 ' . 42 '' . 42 ''' , Mask 999, Mask 998, and Mask 997 require three, five, or two packages, respectively. Thus, five, three, or six packets remain to be filled with null packets.

Lastly, the data packet stream generator generates 40 the finished encoded packet stream 27 '' by adding the object data. The data packet stream generator 40 accomplishes this by determining the first occurrence of a particular object in a program from the authoring tool 24 provided information. Data corresponding to this object is then included in the initial coded data packet stream 27 ' inserted, starting at a point before the first occurrence of this object. The data packet stream generator 40 goes the initial coded data packet stream 27 ' backward step by step and replace null packets with object data as needed. For example, in 2D determines that the object 98 appears in frame 1001. This means that all data associated with object 98 must arrive before frame 1001. The data 43 for object 98, five packets, O98A, O98B, O98C, O98D and O98E, are dropped and added to the sets of packets associated with mask data 1000 and mask data 999. The data 43 ' for object 97, two packets, O97A and O97B, fill and have been added to the set of packets associated with mask data 998.

To simplify the process of extracting data from the transport stream, in one embodiment, the multiplexer associates 44 the mask data and the object data with different packet identifiers (PIDs), as used in the MPEG2 standard for the identification of elementary streams. In this way, the digital receiver 54 Route mask and object data to different compute threads based on their PIDs only, thereby eliminating the need to perform an initial analysis of the contents of the packets. When reassembling the masks and object data, the digital receiver is 54 being able to extract the right number of packets from the stream because of this information through the data packet stream generator 40 provided as part of the coding process. For example, with reference to 2D the data packet stream generator 40 set the mask 1000 to four packages 42 has filled and that the data 43 for the object 98 filled five packages. These data are in a header - 38 . 38 ' . 38 '' . 38 ''' - Contain area of the packet occupying the first 16 bytes of the first packet of each masking packet set.

As in 2D in an enlarged view of the mask header 38 3, the header packet contains information regarding the number of packets carrying masking information, coding information, timestamp information, visibility word information, and the object identifier table unique identifier (UID) associated with the respective mask. UIDs and object mapping tables are referred to below 5A - 5B described in more detail. Likewise, the first packet for each object begins with a 16-byte header 45 . 45 ' containing information enabling the digital receiver 54 the data in the object packages 43 . 43 ' extracted, stored and manipulated. Also included, as in 2D in an enlarged view of the object data header 45 shown, the object data header information, the number of packets carrying data for each object, the data type of the object, the UID of the object and time stamp related information such. For example, the last occurrence of using the object data in the program. The type of data structures used by the system and the use of timestamps by the system will be discussed below with reference to FIG 5A - 5B . 6 and 7 explained in more detail.

In terms of 3 is the multiplexer system 48 ' an extended version of the in 2A shown multiplexer system. The multiplexer system 48 ' is able to handle multiple transport 29 ' and extended transport streams 46 . 46 '' as inputs to produce a single signal which is forwarded to the broadcast medium. The illustrative multiplexer system 48 ' has three transport stream multiplexers 60 . 60 ' . 60 '' , three modulators 68 . 68 ' . 68 '' , three boosters 72 . 72 ' . 72 '' and a mixer 78 on. The multiplexer system 48 ' has three duplicate subsystems for converting multiple sets of transport streams to inputs to a mixer 78 on. Each subsystem has a multiplexer 60 . 60 ' . 60 '' to combine a set of transport streams (TS1 to TSN), (TS1 'to TSN'), (TS1 '' to TSN '') into a single transport stream (TS, TS ', TS'') called the Input signal is used in a digital modulator (for example, a quadrature amplitude modulator (QAM) in the case of a North American digital cable system or an 8VSB modulator in the case of terrestrial transmission) 68 . 68 ' . 68 '' , In one embodiment, each of the transport streams, for example TS1 to TSN, represents a television program. The output signal of the modulator 68 . 68 ' . 68 '' is an intermediate frequency input to a boost converter 72 . 72 ' . 72 '' , which is the output of the modulator 68 . 68 ' . 68 '' converted to the correct channel frequency for transmission. These converted channel frequencies are the input frequencies to a frequency mixer 78 placing the combined signals on the broadcast medium.

In terms of 4 points the digital receiver 54 a tuning device 100 for selecting the broadcast channel of interest from the input broadcast stream and generating an intermediate frequency signal (IF) containing the video and annotation data for the channel. The IF signal is an input to a demodulator 104 which demodulates the IF signal and extracts the information into a transport stream (TS). The transport stream is the input to a video decoder 108 such as B. an MPEG decoder. The video decoder 108 buffers the received video frames in a frame buffer 112 , The decoded video 114 and audio 118 Output signals from the decoder 108 are input signals to the television display device 58 ,

The annotation data will be sent by the video decoder 108 separated and sent to a CPU for processing 124 Posted. The data will be in the memory 128 saved. The memory also stores a computer program for processing annotation data and instructions from a viewer. If the digital receiver 54 from which viewer receives instructions to display the annotation material, the annotation data is rendered as computer graphics images that include a portion or the entire frame buffer 112 overlap. Then the decoder transmits 108 the corresponding video signal 114 to the television display device 58 ,

For transmissions carried by media that can bidirectionally carry signals, such as cable or fiber, a connection may be from the digital receiver 54 to the headend 50 of the broadcasting system. In an alternative embodiment for transmissions carried by unidirectional media, such as conventional television broadcasting or satellite television broadcasts, a connection may be from the digital receiver 54 to the alternative network connection 56 which communicates with a broadcaster or other entity without using the broadcast medium. Communication channels for communication with a broadcaster or any other entity that is not part of the broadcasting medium may be a telephone, an Internet or similar computer connection, and the like. It should be noted that such non-broadcast communication channels can be used even when bidirectional broadcast media is available. Such communication links, the messages sent from the viewer's location to the broadcaster or to another entity, such as a mobile device. B. carry an advertiser are collectively referred to as return channels.

The Back-channel communications can for a variety of purposes are used, including collecting of information for the broadcaster or for could be of value to advertisers, as well as enabling a communication of the viewer with the broadcaster, the Advertisers or others.

In one embodiment, the digital receiver generates 54 Reports related to the viewer's interaction with annotation information via the remote control. The reports sent to the broadcaster via the return channel may include reports regarding the operation of the remote control, e.g. For example, bug reports containing information regarding incorrect choices regarding the viewer's choices of incorrect use of the remote control, such as an attempt to perform an "illegal" or undefined action, or an activity report containing actions performed by the viewer that are tagged to show the timestamp of the material currently displayed on the TV screen The information that can be detected and sent includes a report of a viewer's actions when material provided by advertisers is available, such as: Actions of the viewer to access such material, as well as actions taken by the viewer to terminate such access to the material, such as detecting the point at which a viewer cancels an access attempt In some embodiments, the channel may be a cache as a.

The Information that can be detected and sent further includes information regarding one Transaction that a viewer wants to make, such as giving up an order for an article advertised in a shipment (eg a shirt) including the Order quantity, the size, the Color, the credit information and / or personal identification number (PIN) of the viewer as well as shipping information. In addition, the information includes which can be recognized and sent, also information regarding a Service request, for example, a request to show a Pay-per-view shipment including identification the service, time and place of its provision, payment information and similar. Furthermore include the information that can be detected and sent, too Information regarding not business Information such as B. political information, public information Broadcasting, as z. B. provided by the National Public Radio be, and requirements for Access to data archives such. The patent and trademark databases the US Patent and Trademark Office and the like.

Of the backchannel can also be used for interactive communication, where z. B. a potential buyer select an article which is currently not available, and a series of messages in terms of possibility follows, to make another choice, or whether and how long the spectator ready to wait for a re-delivery of the item. Other illustrative examples of Interactive communication is the display of a current one Price, availability certain goods or services (such as the location of available seats in one Stadium at a particular sporting event, for example the third game of the 2000 World Series), and a confirmation of one Purchase.

When a viewer begins to interact with the annotation system, the recipient may 54 set a flag that protects the data required to perform the interaction with the viewer as long as the viewer continues the interaction, regardless of the program material that may be displayed on the video display device, and regardless of a point in time at which the data would be discarded in the absence of viewer interaction. In one embodiment, the recipient sets 54 an "in-use bit" for each value or data structure that appears in a data structure that provides information to the viewer.A set "in-use bit" prevents the receiver 54 discards the value or data structure. When the viewer ends the interaction, the "in-use bit" is reset to zero, and the value or data structure may be discarded when its expiration expires, as well as present in the data structures of the system, but in 5A - 5B not shown, is an expiration timestamp for each data structure with which the system discards this data structure as soon as the time of the program has exceeded the expiration timestamp. This process of discarding is done by a garbage collector 532 controlled.

In the course of interacting with the annotation system, a viewer can view a catalog set and modify. The catalog may contain articles that the viewer may decide to purchase, as well as descriptions of information that the viewer wishes to receive. The viewer can make a selection from one or more programs for inclusion in the catalog. The viewer may modify the contents of the catalog and may initiate a business transaction as soon as an article is added to the catalog or at a later time.

Of the Catalog can provide entry information about a program that the viewer has viewed, and the number of articles included in the catalog added have been. At a highest level the viewer can connect to the system by using a device such as B. a remote interact to the article of interest, the Order details of interest, such as: Eg quantity, price, model, Size, color and the like, and to identify the status of an order, where, for example, the order is placed immediately or the article of only a list of articles of interest added to the catalog.

On At a further level of detail, the viewer can view the entry for the Select program and can the individual entries in the catalog list, including the status of the entry, for example "saved" or "ordered". The entry "saved" means that the Item was added to the list but was not ordered (ie the data related to the article has been blocked), while "ordered," as the name implies, implies that an actual Ordering the item on the list has been abandoned via the return channel. The viewer can turn the list on to another level of detailing to see the details of an article (eg brand, Model, description, price, order quantity, color etc.). These it is the article not a commercial product, but to get information for the spectators are of interest, such as biographical information about an actor, which occurs in a scene, so by a query on the lowest level information is displayed. In one embodiment Navigate through the catalog with the remote control.

Of the Viewers can use an account for use in the settlement of transactions set up as described above. In one embodiment can the viewer information such. B. his name, a delivery address and financial information such as a credit card or Enter debit card number. This allows the viewer, of every receiver, the according to the system works, for example, a recipient in the house of a friend or in a hotel room to place an order. In another embodiment the viewer can an identifier such. B. a subscription account number and use a password, such as the subscriber account number, the provision of the service by the broadcaster is associated. In such a situation, the broadcaster already has the Address and other delivery information for the viewer as well as over one open financial account with the viewer. In such a case there the viewer simply places an order and confirms his or her wishes by using the password. In yet another embodiment the viewer can set up a personalized catalog. When example for Such a situation can be a personal one for the members of a family Catalog can be given, and they can provide goods and services up to spending limits and according to rules order, with the or the financially responsible in the Family are agreed.

ever It may be the location of the viewer and the broadcasting system in the format of over transmitted the return channel Information about one of the formats QPSK modulation (as used in the US), DVB modulation (as used in Europe) or other formats. Depending on the security requirements for sending, the messages sent via the return channel can be used Completely or partially encrypted be any encryption method can be used. The over the backchannel can communicate information Information includes the authentication of the sender relate (for example, a unique identifier or a digital signature), the integrity of the communication (eg an error correction procedure or system such. CRC), information regarding non-refoulement a transaction, systems and procedures for preventing a Denial of service as well as other similar Information regarding Privacy, authenticity and legal validity of the communication.

Depending on the type of information communicated, the information may be directed to the broadcaster, for example, information regarding viewer responses to transmitted material and requests for pay-per-view material; Information may be directed to an advertiser, such as an order of a shirt; and information may be directed to third parties, such as an access request for a third party managed database. 5A - 5B show data structures used in the invention for storing annotation data information. The data structures store information about position and / or shape video-identified objects and information that enable viewer interactions with identified objects.

In particular, show 5A - 5B a video frame 200 A picture of a shirt 205 as the first object, a picture of a cap 206 as a second object and a picture of a pair of pants 207 contains as a third object. To represent the shape and / or position of these objects, the authoring tool generates 24 a mask 210 which is a two-dimensional pixel array, each pixel having an associated integer value independent of the color or intensity value of the pixel. The mask represents the position information in various ways, including by bordering or highlighting the object (or area of the display device), by altering or enhancing a visual effect with which the object (or area) is displayed by placing a graphic in solid Reference to the object or by placing a number in fixed relation to the object. In this illustrative embodiment, the system generates a single mask for each frame or video image 210 , A collection of video elements with common elements and a common camera perspective is defined as a setting. In the illustrative mask 210 There are four identified areas: a background area identified by the integer 0 212 , a shirt area identified by the integer 1 213 , a cap area identified by the integer 2 214 and a trouser area identified by the integer 3 215 , Those skilled in the art will recognize that alternative forms of representation of objects could equally well be selected, such as mathematical descriptions of an outline of the image. With the mask 210 are a unique identifier (UID) 216 , a timestamp 218 and a visibility word 219 associated. The UID 216 refers to an object assignment table associated with the respective mask 217 , The timestamp 218 comes from the video encoder 36 and is used by the system to synchronize the masks with the video frames. The synchronization process will be described below with reference to FIG 6 described in more detail. The visibility word 219 is used by the system to identify objects in a particular setting that are visible in a particular video frame. Although this in 5A - 5B not shown, all other data structures of the system also include an in-use bit as described above.

In the 5A - 5B The illustrated illustrative set of data structures that enable viewer interactions with identified objects include: Object Map Table 217 ; Object property tables 220 . 220 ' , Primary dialog table 230 ; dialogue tables 250 . 250 ' . 250 '' , Selectors 290 . 290 ' . 290 '' , Action identifiers 257 . 257 ' . 257 '' , Stylesheet 240 as well as strings 235 . 235 ' . 235 '' . 235 ''' . 256 . 256 ' . 256 '' . 259 . 259 ' . 259 '' . 259 ''' . 259 '''' . 292 . 292 ' . 292 '' . 292 ''' ,

The object mapping table 217 has an area number for each of the identified areas 212 . 213 . 214 . 215 in the mask 210 and a corresponding UID for each area of interest. For example, in the object mapping table 217 the shirt area 213 is stored as an integer value "one" and assigned the UID 01234. UID 01234 points to the object property table 220 ,

Also in the object mapping table 217 is the cap area 214 stored as an integer value of two, and it is associated with the UID 10324. UID 10234 points to the object property table 220 ' , The object mapping table starts with the integer one because the default value for the background is zero.

Generally, object property tables store references to the information about a particular object that the system uses to enable viewer interactions with that object. For example, the object property table assigns 220 a title field 221 with the UID 5678, a price field 222 with the UID 910112 and a primary dialog box 223 with the UID 13141516 on. The second object property table 220 ' has a title field 221 ' with the UID 232323, a primary dialog box 223 ' with the same UID as the primary dialog box 223 as well as a price field 222 ' with the UID 910113 on. The UIDs of the title field 221 and the price field 222 the object property table 220 show each on strings 235 . 235 ' containing information about the name of the shirt, "crew polo shirt", and its price, "$ 14.95". The title field 221 ' the object property table 220 ' points to the string 235 '' containing information about the cap's name, "Sport Cap." The price field of the Object Properties table 220 ' points to the string 235 ''' , One skilled in the art will readily recognize that for a given range of authoring-processed video, there are many object property tables that are generated by the authoring tool 24 correspond to identified objects.

The UID of the primary dialog box 223 the object property table 220 points to a primary dialog table 230 , The dialog table 230 is also identified by the UID of the primary dialog box 223 ' the second object property table 220 ' referenced. One skilled in the art will readily recognize that the second object property table 220 ' corresponds to another object within the video frame 200 containing program is identified. General structure dialog tables 230 the text and graphics fields used by the system in interacting with the viewer. dialogue tables 230 serve as a data model in a programming paradigm "Model View Control." The view seen by a viewer is represented by a stylesheet table 240 described with the UID 13579, and the control component is software on the digital receiver 54 provided. The illustrative primary dialog table 230 is used to initiate interaction with a viewer. Other examples of dialog tables include fields for displaying the colors available for a particular article 250 or sizes 250 ' , the number of items he or she wants to buy to confirm a purchase 250 '' and to thank a viewer for his or her purchase. One skilled in the art will be aware that this list is not exhaustive and that the scope of possible dialogue tables is wide.

The look and feel of a particular dialog displayed on the viewer's screen is controlled by a style sheet. The style sheet controls the view parameters in the programming paradigm "Model View Control," a software development paradigm known to those skilled in the art 232 the dialog table 230 Contains a UID 13579 that appears on the stylesheet table 240 shows. The stylesheet table 240 contains a font field 241 , a Gestalt field 242 and the graphics box 243 , In operation, each of these fields has a UID pointing to the corresponding data structure resource. The font field 241 points to a font object, the gestalt field 242 points to an integer, and the graphics box 243 points to an image object, which is explained below. By providing different stylesheets, the present system is easily able to adapt the presentation of information to a particular program. For example, a retailer who wants to advertise a shirt in two programs aimed at different demographic audiences would only need to enter the product information once. The viewer interactions supported by these programs would reference the same data using only different stylesheets.

Of the Structure of many data structures of the present embodiment as a name UID pair provides compatibility benefits to the system. In particular, you can through the use of name UID pairs instead of fixed fields, the data types and protocols have no influence on older ones Digital Receiver Software be extended, and it is possible to annotate the same provided television program several times.

The flexibility the present inventive system is increased by the requirement of the system that the UIDs globally unique have to be. In the illustrative embodiment The UIDs are defined as numbers, with the first set of bits represents a particular database license and the second set of bits represents a particular data structure element. One skilled in the art will recognize that this is a particular embodiment and that there are several ways to ensure that the UIDs are global are unique.

The global uniqueness of the UIDs has the advantage that, for example, two broadcasters transmitting on the same cable system can be sure that the items identified in their programs can be distinguished. It also means that the headend receiver 50 Data from databases 30 . 33 over the network 31 for a particular object because that object has an identifier that is unique across all components of the system. The global nature of the UIDs means that the system can ensure the distinctness of various objects, but it also means that users of the present inventive system can choose not to distinguish articles if such operation is more efficient. For example, a retailer who sells the same shirt in multiple programs only needs to enter the relevant object data once and, in addition, the merchant may use the UID and referenced data with their supplier, eliminating the cost of additional data entry.

In the present embodiment of the current system, there are four defined UID classes: null UIDs, resource UIDs, non-resource UIDs, and extended UIDs. The null UID is a specific value that the system uses to indicate that the UID is not pointing to a resource. Resource UIDs can identify nine different resource types: object mapping tables, object property tables, dialog tables, selectors, stylesheets, images, fonts, strings, and vectors. Selector data structures and vector resources are explained below. Image resources reference graphics used by the system. The non-resource UIDs have four types of values: color values, action identifiers, integer values, and symbols. The action identifiers include Save / Bookmark, Cancel, Next Item, Previous Item, Submit Order, and End, among other actions performed by the viewer. Symbols can represent names in a Name UID pair; the system searches for the name in the stack and replaces it with the associated UID. Non-resource UIDs contain an actual value. Extended UIDs provide a mechanism by which the system is able to increase the size of a UID. An advanced UID indicates to the system that the current UID is the prefix of a longer UID.

When the system requires input from the viewer, it uses a selector 290 data structure. The selector 290 Data structure is a table of pairs of UIDs, where a first column contains UIDs of articles to be displayed to the viewer, and a second column contains UIDs of actions associated with each article. If software in the digital receiver 54 on a selector 290 Data structure, it will display on the viewer's screen all the articles in the first column, generally choices for the viewer. These articles could be strings, images, or any combination of non-resource UIDs. Once on screen, the viewer can scroll up and down through the articles. If the viewer selects one of the articles, the software will run in the digital receiver 54 the action associated with this article. These actions include playing a dialog box, running a non-resource action identifier, or playing another selector 290 ' Data structure. The selectors are through the menu fields 253 . 253 ' . 253 '' . 253 ''' referenced.

In operation, when a viewer selects an object and navigates through a series of data structures, the system places each of the successive data structures used to display information for a viewer on a stack in the memory 128 , As an example, the following viewer interaction called by the in 5A - 5B supported data structures is supported. First, a spectator chooses the cap 214 which causes the system to go through the object allocation table 217 the object property table 220 ' localized and the object property table 220 ' puts on the pile. The following discussion implicitly implies that every viewer-referenced data structure is placed on the stack.

Next, the system displays a primary dialog table containing the title 235 '' and the price 235 '' the cap contains, and also the format of the information presented to the viewer through the stylesheet 240 is controlled. In addition, the initial ad for the viewer includes a number of choices based on that in the selector 290 contained information. Based on the selector 290 the system presents to the viewer the choices made by the strings "end" 256 , "To buy" 265 ' and "save" 256 '' are represented and each referenced with the UIDs 9999, 8888 or 7777. The action identifiers end 257 ' and saving 257 are referenced to the system by UIDs 1012 and 1010, respectively.

When the viewer sees the "buy" string 256 ' selects, the system uses the dialog table 250 , UID 1011 to show the viewer the color options. In particular, the selector brings 290 ' the system to give the viewer the strings "red" 292 , "Blue" 292 ' , "Green" 292 '' or "yellow" 292 ''' , UIDs 1111, 222, 3333 and 4444, respectively. The title for the dialog table 250 is determined by the system by the variable symbol1 266 ' localized. As the object property table 220 ' was placed on the stack, the symbol became 1 266 associated with UID 2001. When the system is on the symbol1 266 ' Therefore, it will track up through the stack until it becomes the symbol1 266 ' localized, which in turn causes the system via the UID 2001 to select the string "select color" 259 ' display.

When the viewer sees the "blue" 2222 string 292 ' selects, the system executes the action identifier associated with UID 5555 and displays a dialog table that uses the "Select Size" string 259 denoted by Symbol2, UID 2002. Based on the selector 290 '' the system locates the string "Large", which is located by the UID 2003 259 '' , UID 1122, as the only available size again. If the viewer had chosen a different color, he would have been directed to the same dialog table, UID 5555, because the cap is in "Large" only. 259 '' the system presents the viewer the dialog table 250 '' , UID 6666 to confirm the purchase. The dialog table 250 '' uses the selector 290 '' , UID 2003, to present the strings "yes" and "no" to the viewer, UIDs 1113 and 1114 respectively. After the viewer enters the "yes" string 259 ''' has selected, the system sends the transaction as specified by the action identifier Send order 257 '' , UID 1013. If the viewer had the "no" string in response to the confirmation prompt 259 ''' the system would have terminated the respective viewer interaction by executing the action identifier 257 ' leave. As part of the exit operation, the system would have the object property table 220 ' and any subsequent data structures that have been placed on the stack based on that particular viewer interaction with the system are removed from the stack. Likewise, after the purchase request has been made by the system, it would have removed the data structures from the stack.

If an action requires more than one step, the system uses a vector resource that is an ordered set of UIDs. For example, if a viewer wants to store a reference to an article he or she found The system must perform two operations: first, it must perform the actual storage operation indicated by the non-resource UID store, and second, it must present a dialog box to the viewer indicating that the item has been saved. Therefore, the vector UID that can store a reference would include the non-resource UID store and a dialog table UID pointing to a dialog table that references the corresponding text.

A particular advantage of the present inventive system is that the data structures are designed to be functionally efficient and flexible. For example, the distributed property of the data structures means that only a minimal amount of data needs to be sent. Multiple data structure elements, such as the object property tables 220 . 220 ' , can point to the same data structure element, such as the dialog table 230 , and this data structure element needs to be sent only once. The stack operation described works together with the distributed property of the data structure such that, for example, the cap 214 no own selector 290 has, the selector 290 but still when displaying for the cap 214 can be specified. The distributed nature of the data structures also has the advantage that individual data units can be modified independently of one another without affecting the information stored in the other data structures.

Another aspect of the present inventive system offering flexibility is additional use of symbols of a variable data type. An icon, in addition to having a value provided by a reference on the stack, may also reference a resource that may be delivered at some time after the initial authoring process. For example, a symbol may cause the DTV broadcasting infrastructure 12 provides a price at the transfer time. This makes it possible, for example, that a dealer distinguishes an object depending on the cable system sending each with different prices.

One another aspect of flexibility, those through the distributed data structure of the present invention is that they have interaction paradigms with multiple Supports viewers. For example, the far-reaching variation in the dialog tables means and the arrangement of their association that the structure of the viewer interaction flexible and easily controlled by the author.

One another example of the variation of the viewer's impression supported by the system is its ability between several video streams switch. This feature uses the structure of an MPEG2 transport stream, from several program streams where each program stream consists of video, audio and data information can exist. In a MPEG2 transport stream, a single transmission on a certain frequency several digital television programs result in parallel. As the skilled person will be aware, is this by associating a program allocation table called PMT is called, with every program in the stream reached. The PMT identifies the packet identifiers (PIDs) of the packets in the current, which correspond to the respective program. These packages contain the video, audio and data packages for each program.

With reference to 5C becomes an object property table 220 '' shown a Linktypfeld 270 with a corresponding link type entry in the UID field and a stream_num field 227 ["Stromnummer"] with a corresponding PID 228 contains. To enable switching between video streams, the authoring tool chooses 24 the PID 228 , which is the PID of a PMT 229 corresponds to a specific program stream. When the object is selected, the object property table 220 '' corresponds, the digital receiver uses 54 the video link entry 271 of the link field 270 to determine that the object is a video link object. Then the digital receiver replaces 54 the PID of the current PMT through the PID 228 the PMT 229 , Then the digital receiver uses 54 the PMT 229 to extract data that matches the new program. In particular, this is due to the PMT 229 designated program a video stream 260 identified by PID17, two audio streams 261 . 262 identified by PID18 and PID19, and a private data stream 263 which is identified by PID20. In this way, the viewer is able to switch between different program streams by selecting the objects associated with those program streams.

6 is a flowchart 500 of data flow and flow control according to an embodiment of the in 2 shown system. 6 shows sequences of steps that take place when a viewer interacts with the hardware and software of the system. In 6 becomes a stream 502 from data for masks on a mask decoder 504 decoded. The decoded masking information 506 be as masks 508 . 508 ' . 508 '' put into a buffer or a snake. Parallel to the masking information 506 becomes a stream 510 of events that can be thought of as interruptions, as events 512 . 512 ' . 512 '' placed in an event queue, where in an one-to-one correspondence an event 512 a mask 508 equivalent. A mask 514 called Thread operates on the masks 508 , The mask 514 Thread locates the mask header, assembles one or more buffers, and queues the masking information to generate mask overlays.

To display masking information decodes and expands a decompression 528 said thread, a mask, for example, held in a resolution of (320 by 240) pixels, to the appropriate size for display on a video screen 530 for example, using a resolution of (640 by 480) pixels. The decompression thread 528 synchronizes the display of mask overlays with the video by examining a timestamp encoded in the masking information and comparing it with the timestamp of the current video frame. If the mask overlay frame is ahead of the video frame, the decompression thread rests for a calculated amount of time representing the difference between the timestamps of the video and the mask. This mechanism keeps the masks in close synchronization with the video so that masks seem to overlay video objects.

A second stream 516 of annotations data is used for a second, objects 518 provided software thread. The object data stream 516 is determined by the objects 518 Thread analyzes and decodes, which decodes each object and inserts it into the object hierarchy. The output of the object 518 Threads is a stream of objects 519 with changing properties such as shape, size and the like.

A "model" called thread 520 combines the masks 508 and the objects 519 to form a model of the system. The masking information includes a unique ID for each object represented in the mask overlay. The unique IDs correspond to objects stored in the model. The model 520 -Thread uses these unique IDs to synchronize or tune the appropriate information.

The model 520 Thread contains organizational structures such as A stack, a hash table, and a queue, all of which are well known to those of ordinary skill in the software arts. For example, the stack may be used to store in a memory a temporary indication of a state that may be recovered, or a memory location that may be recalled. A hash table can be used to store data of a particular type or pointer to that data. A queue may be used to store a sequence of bits, bytes, data or the like. The model 520 interacts with a view 526 called thread, which is on a screen 530 such as B. controls a television screen displayed information. The opinion- 526 Thread uses the information contained in the model 520 Thread, which tracks the information needed to display a particular image with or without interactive content. The opinion- 526 Thread also interacts with the mask 514 Thread to ensure that for the display screen 530 the right information will be provided at the right time.

A soft called thread 522 controls the functions of a state 524 named state machine. The details of state 524 be re 7 explained in more detail. Status 524 interacts with the view thread 536 ,

It is a garbage collector 532 is provided to collect and discard data and other information that is becoming obsolete, such as data with a timestamp for the most recent use, which corresponds to an earlier date. The garbage collector 532 can periodically clean the system's memory to remove such extraneous data and information and free up space to store new data and information. Such garbage collector software is known in the software arts.

7 Represents the interactions between and among states within the state machine state 524 The state machine state 524 contains a provision 602 that when enabled the state machine state 524 placed in a renewed starting position, with all values such. Memory contents, stack pointers, and the like may be set to default values which may be stored in ROM, SDRAM, magnetic memories, on CD-ROM or in a protected memory area for use as needed. Once the system has been reset, the state of the state machine goes to state 524 into an interactive content icon 604 status above, as indicated by the arrow 620 displayed. Please refer 7 ,

Inter content icon 604 is a state in which a visual image, similar to a logo, in a defined area of the television display device 58 appears. The visual image is referred to as an "icon", hence the name interactive content icon for a visual image that is active.The icon may change its appearance or alter or enhance a visual effect with which the icon is displayed, such as by changing the image Color, change in transparency, change in brightness, flash or blink, or by apparent motion or the like when interactive In formations are available.

The viewer of the system may respond to an indication that information is available by pressing a button on a portable device. For example, in one embodiment, pressing a right arrow or left arrow (analogous to the arrows on a computer keyboard or the volume buttons on a portable TV remote) causes the state of the state machine to be state 524 of interactive content icon 604 on mask highlighting (MH) 606 replaced. The state change of interactive content icon 604 to MH 606 is by the arrow 622 displayed.

In the state MH 606 For example, when one or more areas of an image correspond to material ready for presentation to the viewer, such area is highlighted, for example, by providing a borderline that outlines the area appearing on the video display device (see FIG 1D ) or by changing the appearance of the area or part thereof. In one embodiment, when a shirt worn by a man is the highlighted object, a visually distinct outline of the shirt appears in response to the keystroke, or the shirt changes appearance in response to the keystroke. Repeating the keypress or pressing the other arrow key will cause another object, such as a wine bottle on a table, to be similarly highlighted. In general, the highlightable objects are highlighted successively by successive keystrokes.

If the viewer does not perform any action for a predetermined period of time, such as ten seconds, the state of the state machine will go to state 524 in the state of the interactive content icon 604 back, as by the arrow 624 displayed. Alternatively, if the viewer activates a button other than an arrow pointing to the side, for example, the "select" button, often seen in the middle of the navigation arrows on remote controls, will go to state MH 606 in a state over, as an info box 608 referred to as. About box 608 is a status in which information appears in a pop-up box (ie an information box). The state change of MH 606 to info field 608 is by the arrow 626 displayed. The information that appears is set by an advertiser or provider of the information and may, for example, For example, the brand name, model, price, local reseller, and highlighted object specifications. For example, in the case of the man's shirt, the information could include the brand of the shirt, the price, the range of sizes available, examples of available colors, information about one or more dealers, information about special offers, information about telephone numbers or e-mail addresses, about which a contact to order is possible, and the like more. One potential problem that can occur with the system is that in two consecutive settings, the same object (ie an area with the same OPT) may appear, but may have different indexes within the mask frames and OMTs. For example, two people can edit different sections of a video cast and assign different indices to the same object as it appears in the section of the video being edited. The possibility that a single object in different settings may be labeled with different indexes is a problem if, ideally, the system should continue to display a highlight over the area after the setting changes. To counteract this potential problem, the system uses a reverse lookup algorithm before playing an object overlay. When an object is first highlighted by the viewer, the OMT for that setting is referenced and the UID for the OPT of the highlighted object is stored in memory, then the highlighting is displayed. Prior to rendering the highlight for each of the consecutive frames, the mask frame is first checked to find the current OMT. Then, a search is performed using the OPT UID of the current object to search the OMT. If the OPT UID is found, the area within the mask frame is highlighted with a reference corresponding to that associated with the OPT UID in the OMT. This reverse search continues until a time limit is exceeded or a new object is selected.

There are many possible responses from a viewer, and these responses lead back to the state interactive content icon in several ways 604 , as generally by the arrow 628 displayed. These responses may include, for example, the viewer expressing his interest in the information provided, for example, by purchasing the described article, requesting further information, or waiving such purchase.

While the system is in the interactive content icon 604 State, the viewer can press a fast-forward button which will burst [burst] 610 activated state, creating a transition 630 of interactive content icon 604 to fast passage 610 be is acting. In the high-speed 610 State, the video display device automatically highlights sequentially all objects that currently have associated information that can be presented to a viewer. The highlighting period of each individual object is short, of the order of 0.03 to 5 seconds, so that the viewer can quickly determine which objects may have associated information for presentation. A preferred emphasis period is in the range of 0.1 to 0.5 seconds. The high-speed 610 State is analogous to a search state for radio receivers in the search, in which signals that can be received with a suitable signal strength are set in succession for short periods of time.

The high-speed 610 Status automatically returns to the interactive content icon 604 State back 632 as soon as the various objects having associated information have been highlighted. As soon as the system reaches the interactive content icon 604 State has returned, the viewer has the option of activating an object of interest having associated information as described above.

In another embodiment, the high speed pass may be 610 State caused by an instruction embedded in a message. In yet another embodiment, the high speed pass may be 610 Condition to inform a viewer of the areas that may be active, or the rapid transit 610 State can be invoked when a new setting begins that contains areas with set visibility bits.

The Interactive content icon can be used to a viewer to give visual cues. In one embodiment, the interactive content icon appears only if there is material to display for the viewer with one or more areas of an image.

In one embodiment, the interactive content icon is active when the high-speed content icon is active. 610 Condition is caused. The interactive content icon may take on a shape that signals that the high-speed 610 State begins, for example, by displaying the interactive content icon itself with an enhanced visual effect similar in appearance to the enhanced visual effect that each visible region assumes. In various embodiments, an enhanced visual effect may be a change in color, a change in brightness, a change in the icon itself, a blinking or flashing of a portion of a display device, or the like.

In one embodiment, the interactive content icon is augmented by additional areas shaped like pointers to points on the compass or like buttons on the remote control of the digital receiver. The expansion areas are displayed, either simultaneously or sequentially, with an enhanced visual effect. An illustrative example of various embodiments is shown schematically in FIG 8A to 8G shown. 8A represents an inactive interactive content icon. 8B represents an active interactive content icon that is visually enhanced. 8C represents an interactive content icon entering the high-speed state in which four arrowheads are added pointing north (N), east (O), south (S) and west (W) directions. For example, the extension areas may be presented in shapes reminiscent of the shapes of the buttons on a portable device. In one embodiment, the north (N) and south (S) arrowheads may correspond to buttons used to change channels on a portable video remote, and the east (O) and west (W) arrowheads may correspond to buttons On a video remote control, the volume is changed to remind the viewer that a press of these buttons in response causes a fast-pass condition.

8D FIG. 12 illustrates an interactive content icon in the active high-speed state, in which the interactive content icon itself and the northern (N) pointing northward arrowhead are displayed with enhanced visual effects. 8E FIG. 12 illustrates an interactive content icon in the active high-speed state in which the interactive content icon itself and the east-east pointing arrowhead are displayed with enhanced visual effects. 8F FIG. 12 illustrates an interactive content icon in the active high-speed state in which the interactive content icon itself and the south-facing (S) arrowhead are displayed with enhanced visual effects. 8G FIG. 12 illustrates an interactive content icon in the active fast-pass state in which the interactive content icon itself and the west-waving arrowhead are displayed with enhanced visual effects.

As explained earlier, the information displayed on the video display device 58 appear, including the television program and all comment information that can be provided, from a headend 50 to the digital receiver 54 Posted. Video images generally contain much information. In modern high-definition television formats, a single video frame may be used contain more than 1000 lines. Each line can span more than 1000 pixels. In some formats, a 24-bit integer is required to represent each pixel. Sending such large amounts of information is cumbersome. Compression methods that can reduce the amount of data to be transmitted play a useful role in television communication technology. The compression of files in general is known in the field of computer engineering. However, the invention uses new forms of file compression which are of particular use in the field of image compression.

One conventional Compression method is called run-length compression. In this process, each pixel or group becomes identical Pixels that appear consecutively in a video line as ordered Pair encodes a first number that indicates how many identical Pixels are to be played, and a second number that looks the same defines each such identical pixel indicates. If it is long Runs identical Pixels, such a coding method can be the total number reduce bits that need to be sent. However, in pathological cases for example, if every pixel from the previous pixel and If the following pixel is different, the coding scheme will actually be more Bits require as the number of bits to represent the pixel sequence itself is required.

In an embodiment will improve runlength encoding, referred to as "section runlength coding", scored when two or more consecutive lines are so categorized can be that they have the same sequence of runlengths with the same sequence appearance or color. The two or more lines are treated as a section of the video image. An example of one such section is a person in front of a plain background you can see. A sender encodes the section by providing a single sequence of colors, covering all the lines in the section is valid, and then encodes the numbers of pixels per line, each consecutive Have color. This procedure repeats the sending of bypassed redundant color information, which is a long per color Bit pattern requires.

9A makes a picture 700 one person 705 that is in front of a monochrome background 710 , for example a blue background. 9A illustrates several embodiments of video image compression techniques. In 9A the person has a skin color that is in the area 720 appears. The person is wearing a purple shirt 730 and a green pants 740 , Different colors or appearances may be encoded as numbers having low values when the encoder and decoder use a look-up table to translate the encoded numbers into complete (eg, 24-bit) display values. As one embodiment, a background color for the purposes of a mask may be defined as a zero or a transparent visual effect, allowing display of the original visual appearance of the image without modification.

In this embodiment of the "section run-length encoding," the encoder samples each row 752 . 754 . 762 . 764 and records the color value and the length of each run. If the number of runs and the sequence of colors of the first row 752 one video frame not to that of the next row 754 fits, becomes the first row 752 is coded as a section of length l, and the subsequent row 754 is compared to the next succeeding row. When two or more rows actually contain the same sequence of colors, the portion is encoded as a number of rows having the same sequence of colors, followed by a series of ordered pairs representing the colors and run lengths for the first row of the portion. As in 9B for an example with three rows, the first row (s) contains values of pairs of colors and run lengths. The remaining two rows are coded as just run lengths, and the colors used in the first row of the section are used by a decoder to recover the information for displaying the later rows of the section. In one embodiment, the portion may be defined to be less than the entire extent of a video line.

As an example, referring to 9A are called the illustrative series 752 and 754 Video scan lines representing the background 710 , a section of the skin 720 the person and further background 710 contain. Both illustrative series 752 and 754 include runs of blue pixels, skin colored pixels, and other blue pixels. Thus, the rows would 752 and 754 and other adjacent rows intersecting the skin-colored head or neck area of the subject encoded as follows: a number indicating exactly how many the rows 752 . 754 similar series are in a section defined by the blue background color - skin color - blue background color pattern; a first series code comprising the value indicating the blue background color and an associated pixel number, the skin color value 720 and an associated pixel count, as well as the value indicating the blue background color and another associated pixel count. The remaining rows in the section would be encoded as a number representing a number of pixels in blue Background color represents a number representing a number of pixels in skin color 720 and a number representing the remaining pixels in blue background color.

In another embodiment includes a method that reduces the information to be coded, labeled "X-run-length encoding", encoding only the information within objects that have been identified are. In this embodiment the coded pixels are only those that are defined within the Object or within an outline of the object appear. An encoder in a transmitter represents the pixels as ordered Triplet comprising a value, a run length and an offset, which is the Defined starting position of the run with respect to a known pixel, for example, the beginning of the line. In a receiver puts a decoder restores the coded information by placing the ordered Triple reads and the pixels according to the coded information reproduces.

Again with reference to 9A is each of the illustrative lines 752 and 754 in the X run length coding method represents an ordered triad of numbers comprising a number representing the skin color 720 indicates a number representing how many pixels in skin color 720 and a number representing the distance from an edge 712 of the picture 700 indicating in which the skin color 720 reproduced pixels are to be positioned. An illustrative example will be in 9C given.

In again another embodiment a method called "X-section run-length encoding" uses the features of the section run length and X run length coding methods combined. The X-section run-length encoding method uses Color values and run lengths as encoding parameter, but ignores the coding of background. Each entry in this coding scheme is an ordered triad of color, yardage and offset values as in X run length coding.

The illustrative lines 762 and 764 are part of a section of consecutive lines that can be described as follows: The illustrative lines 762 . 764 contain, in this order, segments of blue background 710 , one arm of the violet shirt 730 , blue background 710 , the main part of the violet shirt 730 , blue background 710 , the other arm of the violet shirt 730 and a last segment of blue background. The illustrative lines 762 . 764 and the other adjacent lines having the same color pattern are encoded as follows: an integer defining the number of lines in the section; the first line is coded as three sets of three numbers indicating a color, a run length and an offset; and the remaining lines in the section are encoded as three ordered groups of two numbers indicating a run length and an offset. The color values are decoded from the sets of triplets and are then used for the remaining lines of the section. Pixels that are not defined by the ordered groups of two or triplets are rendered in the background color. An illustrative example will be in 9D shown using three rows.

In turn another embodiment includes a method referred to as "super runlength coding". In this embodiment decompose a video image into a variety of areas by a CPU, may include the sections. The CPU applies the decompression methods described above the different areas and determines section by section a coding of the most efficient compression method.

Then, the CPU encodes the picture section by section as a mixture of the most efficient methods, with the addition of a leading integer or prefix symbol indicating the method by which each section has been coded. An illustrative example of this super run-length encoding is the encoding of the image 700 using a combination of runlength coding for a few lines of the image 700 X-run length coding for other lines (eg. 752 . 754 ) of the picture 700 , X-section run-length encoding for other lines (eg. 762 . 764 ) of the picture 700 and so on.

Other embodiments of coding schemes may be used. One useful embodiment involves calculating an offset of the pixels of a line from the previous line, for example shifting a subsequent line, such as one near the neck of the one in FIG 9A Person represented by a small number of pixels, and filling all undefined pixels at both ends of the shifted line with pixels that represent the background. This approach can be applied to both run lengths and row position information. This embodiment offers the advantage that an offset of seven or fewer pixels can be represented as a signed four-bit value, with great savings in the amount of information that must be sent to define the line thus encoded. Many images of objects include offsets between the lines, which are relatively small, and such coding can provide a significant reduction in the data to be transmitted.

A another embodiment includes the coding of running values within the limit by an outline as ordered pairs, starting at an edge of the outline. Other combinations of such coding schemes will be apparent to those skilled in the art be seen in the field of data compression.

to fulfillment The object of the invention is in addition to the efficient representation the content of pictures an ability to carry out An analysis of the content of images makes sense. TV pictures, the include a variety of pixels, can be analyzed to the presence or absence of persons, objects and features to determine, so that selected Persons, objects and features can be assigned annotations. The Movements of people, objects and features can also to be analyzed. Before carrying out such an analysis becomes an assignment of pixels in an image or a frame too one or more persons, objects and / or features.

The analysis is useful in manipulating images to produce a smooth image, or one that is pleasing to the viewer, rather than a jagged or rough image. The analysis can also be used to define an area of the image that is enclosed by an outline of defined thickness in pixels. In addition, the ability to define an area using mathematical relationships allows the visual modification of such area by using a visibility bit that indicates whether the area is visible or invisible, and by using techniques that allow the rendering of all pixels in an area with one specific color or a specific visual effect. An image is examined for areas that define matter of interest. For example, in 9A a shirt area 730 , a head area 720 and a trouser area 740 identified.

In an embodiment The pixels in an image or frame are considered to be an area belonging classified. The classification may be based on the observations of a Spectator based, which can interact with an image in digital form on a digital display device such. B. presented to the monitor of a computer becomes. In one embodiment The author / annotator can use areas of an image an input device such. A mouse or other computer pointing device, a touch-sensitive Screen, a light pen or the like. In another embodiment can the areas by a computing device such. B. a digital Computer or a digital signal processor in conjunction with Software to be determined. In both cases, there may be pixels that difficult to classify as part of an area, For example, when a plurality of areas adjacent to each other.

In an embodiment can be a pixel that is difficult to classify or whose Classification is ambiguous, to be classified by a method which includes several steps. First, the classification of the pixel is eliminated or aborted. This declassified Pixel is used as the origin of a classification figure that extends to a variety of pixels (i.e., a neighborhood) near of the declassified pixel. The thus covered pixels become examined in terms of their classification, and the ambiguous Pixel is assigned to the class, the largest representation in the neighborhood Has. In one embodiment The neighborhood includes neighbors next to each other of the ambiguous pixel. In one embodiment, a rule for Make an assignment in case of ties with regard to representation applied. In one embodiment The rule may be to classify the pixel of a pixel assign specific position relative to the pixel, for example the class of the closest to the upper left corner of the image nearest neighbor, the one to the strongest represented class belongs.

In a further embodiment can be a pixel that is difficult to classify or its classification is ambiguous, to be classified by a procedure that has a novel implementation of principles of mathematical morphology includes. The mathematical morphology represents the pixels of an image in mathematical expressions and allows the algorithmic calculation of properties and transformations of images, for example using a digital computer or digital signal processor and appropriate software. The principles The mathematical morphology can be used to create various Use image processing applications. It will be a very short one here explanation submitted to the principles.

Especially the methods known as dilatation and erosion are described and explains. Generally speaking Dilatation and erosion are used to shape, size and some characteristics to change from areas. In addition will be some illustrative examples of Applications of the principles of mathematical morphology to the Image processing described.

Dilation and erosion are fundamental mathematical operations based on sets of pixels Act. As an exemplary description regarding an image in two-dimensional space, the set of points of a region R and a two-dimensional morphological mask M are presented. The illustrative explanation presented in terms of binary mathematical morphology will be discussed with reference to FIG 10A and 10B given. In 10A For example, the morphological mask M has a shape, for example, a five-pixel array in the form of a plus sign. Depending on the effect to be achieved, morphological masks of different shapes can be selected. The region R can have any shape; for illustration purposes, area R will be the one in 10A taken irregular shape shown.

The morphological mask M moves over the image in 10A , and the result of the operation is recorded in an array that can be visually represented as a frame that is in 10B will be shown. For the illustrative morphological mask, the pixel located at the intersection of the vertical and horizontal lines of the plus sign is selected as a "test" pixel, or as the pixel "turned on" according to the result of the applied operation (e.g. B. is set to 1) or "switched off" (eg set to 0).

For binary erosion, the mathematical rule, expressed in terms of set theory, may be that the intersection of one or more pixels of the morphological mask M with the region R defines the condition of the pixel to be stored in an array or at the position in 10B graphically representing the location of the test pixel in 10A equivalent. This rule means that when moving the morphological mask by one pixel at a time, when all designated pixels of the morphological mask M overlap with pixels of the region R, the test pixel is turned on and the corresponding pixel in 10B is left in an on state. For example, the scan of the mask may be from left to right over each row of the image, beginning in the top row and moving down. Other scan paths covering the entire image (or at least the regions of interest) may be used, as will be apparent to one of ordinary skill in the art of mathematical morphology. This operation tends to smooth a region, and depending on the size and shape of the morphological mask, it may have a tendency to eliminate pointed protrusions along the contours of a region. In addition, depending on the size and shape of the morphological mask, the size of an image can be reduced.

In terms of set theory, the binary dilation can have the mathematical rule that the union of the morphological mask M with the region R defines the condition of the pixel that is at the position in 10B graphically representing the location of the test pixel in 10A equivalent. For a given location of the morphological mask M, the pixels of R and the pixels of M are examined, and if a pixel turned on in M corresponds to a pixel turned on in R, the test pixel is turned on. This rule is also applied by scanning the morphological mask over the image as described above, for example from left to right over each row of the image, again from the top row down. This operation may tend to cause a region to expand and fill small holes. Dilation and erosion operations are not commutative, which means that erosion and subsequent dilatation are generally performed differently than dilatation and subsequent erosion.

The operations of erosion and dilation, as well as other operations based on these basic operations, can be applied to sets of pixels defined in space, as found in a two-dimensional image, as just explained. The same operations can equally well be applied to sets of pixels in a time sequence of pictures, as in FIG 11A and 11B shown. In 11A Time can be viewed as a third dimension that is orthogonal to the two dimensions that define each image or frame. 11A FIG. 11 shows three pictures or frames labeled N-1, N and N + 1, with frame N-1 displayed first, frame N next, and finally frame N + 1. Each frame is conceivable to have an x-axis and a y-axis. In an illustrative example, each frame includes 480 horizontal rows of 640 pixels or columns each. By convention, rows are numbered from top to bottom, and columns are progressively numbered from left to right. The upper left corner is Row 0, Column 0 or (0, 0). The x-axis determines the series, the x-value increases as one moves down the left edge of the frame, and the y-axis determines the number of columns per row, the y-value increasing as one moves moves to the right along the top edge of the frame. The time axis along which time increases is then considered to be in 11A horizontally from left to right.

The operations of erosion and dilatation in two-dimensional space used a morphological mask such. For example, the five-pixel plus sign in the plane of the image or frame is aligned. An operation in the time dimension using the two-dimensional plus-five-plus sign as a morphological mask can be thought of as in the following discussion, it being understood that one dimension of the plus sign lies along the time axis and the other lies along a space axis , In other embodiments, one could use a one-dimensional morphological mask along only the time axis or a three-dimensional morphological mask having dimensions in two non-collinear spatial directions and one dimension along the time axis.

The "test" pixel of the five-dimensional two-dimensional morphological plus-sign mask is located at row r, column c, or position (r, c) of frame N in FIG 11A , The pixels in the vertical line of the plus sign are at column c of row r - 1 (the row over row r) of the frame N and column c of row r + 1 (the row under r) of the frame N. The pixel "left" of the "test" pixel is located at row r, column c of frame N-1 11A (the frame before the frame N), and the pixel "right" of the "test" pixel is located at row r, column c of the frame N + 1 11A (the frame following frame N). The result of an operation using this morphological mask is thus visually recorded at row r, column c of a frame corresponding to frame N, and the result can be recorded in an array at the corresponding position. However, in this example, the calculation is dependent on three pixels that are in the frame N, one pixel that is in frame N-1, and one that is in frame N + 1. 11A schematically illustrates the use of the five-pixel "plus" mask on three images or frames representing successive images in time, and 11B represents the result of the calculation in a frame corresponding to the frame N.

In this system according to the invention a novel form of erosion and dilatation is applied all areas are eroded and dilated in one go, instead of processing only a single area at a time (where the area is denoted by "1" and the non-area by "0"), and the process is repeated several times if multiple areas too treat. If, in the case of erosion, the input image R ranges contains whose pixels are each denoted by 1, 2, ... r, then the For example, test pixels labeled '3' when and only if all pixels are below the set pixels in the morphological Mask are denoted by 3. Otherwise, the test pixel is assigned 0 or unclassified in the case of dilatation, the input image R contains regions whose Pixels are respectively denoted by 1, 2, ... r, then the test pixel is for example, labeled '3' if and only if the area with the highest number of pixels is the one with the designation is 3. Otherwise, the test pixel becomes 0 or "not classified ".

Two-Dimensional Floodfill is a technique known in the art, the change of a Characteristic of a two-dimensional surface to a defined characteristic causes. For example, two-dimensional floodfill can be used become the visual effect of a connected area of an image to change in a defined way, For example, all pixels of the area to be colored red. Three-dimensional floodfill can be used to make all elements of a volume into one to change the defined characteristic. For example, can a volume can be used to represent an area which appears in a series of sequential two-dimensional images, those in the sequence number or the display time as the third dimension differ from each other.

It is an efficient, novel algorithm for filling one connected three-dimensional volume developed by Floodfill starting with an image containing a region which Is part of the volume. Overview the method allows the selection of an element on a two-dimensional surface within the volume and leads a two-dimensional flood fill on the area containing the selected element. The Method selects a direction along the third dimension determines whether a subsequent surface contains an element within the volume, and if so is, leads it made a two-dimensional floodfill of the area, one such Contains element. The process repeats the process until no more items are found and returns to the first flood fill filled area back and repeats the process while it moves along the third dimension in the opposite direction emotional.

It is an algorithmic image processing technique using of a three-dimensional floodfill operator the author classified a point within a group of incorrectly Selects points. The selected Point can be determined using a classification method already described be reclassified. The entire group of Pixels that match the selected ones Point adjoin, then becomes the classification of the selected point reclassified. Adjacent pixels of the reclassified pixels in Previous and subsequent frames can also be reclassified become.

In an embodiment, the three-dimensional volume to be reclassified comprises two dimensions representing the image plane, and a third dimension representing time. In this embodiment, for each pixel (r, c) in the frame N is off 11A which changes from A to B due to the two-dimensional flood-fill operation in the frame N, when the pixel (r, c) in the frame N + 1 turns off 11A currently assigned the color A, the two-dimensional Floodfill performed starting at pixel (r, c) in frame N + 1 11A , whereby all adjoining pixels are changed in frame N + 1, which are assigned color A. Again with respect to 11A For example, it is also possible to start such a process at frame N and proceed backwards in the time dimension to frame N-1. In one embodiment, the three-dimensional floodfill process is terminated on a frame in which no pixel has a designation that requires modification as a result of the floodfill operation. In one embodiment, once the three-dimensional flood fill is completed in one direction of time, the process continues by starting at the initial frame N and continuing in the opposite direction in time until the process ends again.

11C is a flowchart 1150 , which illustrates an illustrative process for achieving three-dimensional flood fill according to one embodiment of the invention. The process begins at the circle 1152 The process operating entity, such as an operator of an authoring tool, or alternatively, a computer that analyzes images to locate within images one or more regions that correspond to objects, selects multiple sequential two-dimensional ones Portions that delineate the volume to be filled in the three-dimensional floodfill process, as in step 1154 displayed. In one embodiment, the three-dimensional volume comprises two-dimensional portions arranged orthogonal to a third dimension, each two-dimensional portion including positions identified by a first coordinate and a second coordinate. For example, in one embodiment, the two-dimensional portions may be frames, and the third dimension may represent the time or a frame number that identifies successive frames. In one embodiment, the first and second coordinates may represent row and column positions that define the position of a pixel within an image frame on a display device.

In step 1156 The process operator defines multiple areas in at least one of the two-dimensional sections, each area including at least one location. From this point on the process, the process is performed using a facility such as As a computer that can execute a series of instructions that may be encoded in software. The computer may record information according to the definitions for later use, such as in a machine-readable memory. For example, an operator in an image may define a background and an object of interest, e.g. B. a shirt 2 ,

In step 1158 For example, the computer selects a first area in one of the two-dimensional sections, the area being contained within the volume to be filled with a selected icon. In one embodiment, the symbol may be a visual effect in rendering on a display device, such as a color, a highlight, a change in brightness, or the like, or may include a character such as a color. Example, an alphanumeric character or other such symbol that can be displayed on a display device.

In step 1160 The computer running the display fills the first area with the selected icon. There are many different known graphics routines for filling a two-dimensional area with a symbol, for example, changing a defined area of a screen to a defined color. Any such known two-dimensional graphics routine may be implemented to perform the step of two-dimensional filling.

In step 1162 The computer moves in a first direction along the third dimension to the subsequent two-dimensional section. In one embodiment, the process operator moves to the image immediately before or after the first selected image, thereby defining a direction in time or in the image sequence.

In step 1164 The computer determines whether a position in the subsequent two-dimensional portion corresponding to a filled position in the two-dimensional portion of the previous two-dimensional portion belongs to the volume. The process operator looks for information that is listed in the definitions of the two-dimensional areas that are in step 1156 were achieved.

The computer takes on the result of the step in step 1164 a choice. Is a positive result of the determination step 1164 before, the computer fills an area containing the position in the subsequent two-dimensional section with the selected icon, as in step 1168 displayed. As with step 1170 appears, the computer repeats, starting with the newly filled area in the subsequent two-dimensional section, the movement step 1162 , the determination step 1164 and the filling step 1168 (ie, the immediately preceding steps) until the determination step results in a negative result.

For a negative result of a previously mentioned determination step 1164 the computer returns to the first in step 1158 identified area (which has already been filled) and repeats, moving along the third dimension in a direction opposite to the first direction, the steps of the movement (eg a step similar to Step 1162 but in the opposite direction), determination (eg a step like step 1164 ) and stuffing (eg a step like step 1168 ), as stated above, until a determination step has a negative result. This sequence is summarized at step 1172 played. At step 1174 the process ends with a negative result of a determination step.

Another application involves creating outlines of areas, for example, to allow an area to be either completely highlighted or highlighted by changing the visual effect associated with the outline of the area, or a combination of the two effects. In one embodiment, a method of constructing outlines from designated areas, as in FIG 12A - 12B represented, implemented. An area to be bordered 1210 with an outline 1215 in the input picture 1218 is in 12A shown. A square morphological mask 1220 with an odd number of pixels whose size is proportional to the desired outline thickness is over the range 1210 guided. At every position in the input area 1210 The pixels within the morphological mask are checked to see if they are all the same. If so, the test pixel will be in the output image 1230 out 12B assigned a '0'. If one pixel differs from another pixel within the morphological mask, the label falling below the middle pixels of the morphological mask becomes the test pixel in the output image 1230 assigned. While the morphological mask 1220 over the area 1210 moves, is in an output image 1230 a result outline 1215 ' generated. In other embodiments, even-numbered square morphological masks may be used, morphological masks having shapes other than squared and quadratic morphological masks having odd pixel numbers, in which one selects a particular pixel within the morphological mask as the pixel corresponding to the test pixel 1222 in the output image 1230 equivalent.

It It is understood that the average person skilled in the use the principles of mathematical morphology the preceding ones Example applications using alternative morphological Masks and alternative rules can build and many more, similar Recognize applications based on such principles.

In a number of related Images or a setting as previously described, for example a sequence of images that one sitting on a park bench Person can show one or more of the selected Objects over one Number of frames persist. In other situations, such as an abrupt one change in the picture, for example when the scene changes to the view, which the person on the park bench perceives, some or all of the in the first scene or setting identified areas in the second Scene or shot is missing. The system and method of the invention can both notice that the scene has changed (eg starts a new setting) and that one or more in the first scene existing areas in the second scene are not present.

In one embodiment, the system and method determines that the scene or setting has changed by calculating a histogram of pixels that have changed from one image to a subsequent image and comparing the slope of the subsequent instances (or time history) of the histogram with a given slope value. 13 Figure 3 shows three illustrative examples of the evolution of histograms over successive frames (or over time). The topmost curve 1310 has a small variation of zero slope and represents moderate speed movement. The middle curve 1320 shows a slightly greater variation of tilt and represents sudden movement. The lowest curve 1330 shows a large variation of the inclination and represents a change of setting or scene change in frame F. When the slope of the histogram development curve exceeds a predetermined value as in the lowest curve 1330 , the system detects that a setting change has occurred.

A preferred embodiment of this system has been described in terms of digital television broadcasting. It should be understood that an alternative embodiment of this system may also be similarly integrated with old analog television broadcasting systems. As used herein, the term metadata is understood to include masks, objects, annotation information, data structures, timestamps, or other time information, and generally all additional information and data used in conjunction with the original, unaltered television broadcasting signals. In the analog embodiment, the metadata may be associated with an analog transmission signal by one or more of the following steps: inserting metadata into the "vertical blanking interval" (VBI) as performed by closed captioning systems and transmitting metadata via a secondary communication channel such as, but not limited to, a subcarrier signal, another channel frequency within the electromagnetic transmission spectrum, or a connection to a computer network, such as the Internet.

In this embodiment support the video source, the modulator, the tuner and the receiver analogue signals, while the metadata can be digital. The analog receiver can collecting the metadata from the VBI can store metadata in memory and can process them as described herein.

Claims (19)

A hyperlink annotation data system comprising: a voting device ( 100 ); a demodulator ( 104 ) in conjunction with the tuner ( 100 ); a video decoder ( 108 ) in conjunction with the demodulator ( 104 ); a display device ( 58 . 530 ) in conjunction with the demodulator ( 104 ); a memory ( 128 ) which stores annotation data to be associated with at least one object displayed in a video frame; and a central processing unit ( 124 ) in conjunction with the decoder ( 108 ) and the memory ( 128 ); the memory ( 128 ) stores a computer program which, when stored by the central processing unit ( 124 ), the display of annotation data ( 5 ) on the display device ( 58 . 530 ), the annotation data being provided with an expiration time stamp used by the computer program, the annotation data ( 5 ) after a predetermined time. The system of claim 1, further comprising: a viewer interaction device, the computer program storing the annotation data ( 5 ) on the display device ( 58 ) in response to activation of the viewer interaction device by a viewer. The system of claim 1, wherein the memory ( 128 ) Masking information ( 210 . 506 ) stores; the masking information ( 210 . 506 ) is used by the computer program to display areas on the display device ( 58 . 530 ) to identify. The system according to claim 3, wherein said masking information ( 210 . 506 ) a masking timestamp ( 218 ) having. The system of claim 4, wherein the masking timestamp ( 218 ) is used by the computer program to extract the masking information ( 506 ) in temporal relation to a video frame. The system of claim 3, wherein the memory ( 128 ) Stores object information; wherein the object information is used by the computer program, objects in a video frame on the display device ( 58 . 530 ). The system according to claim 6, wherein the object information comprises the expiration time stamp. The system of claims 6 or 7, wherein the object information comprises: an object mapping table ( 217 containing an object number for an object in a video frame and a corresponding first identifier; and an object property table ( 220 ) referenced by the first identifier, the object property table containing a first group of annotation data. The system according to claim 8, wherein the first group of annotation data is an annotation data field and a second identifier containing a second group referenced by annotation data. The system of claim 9, wherein the annotation data field is a title field ( 221 ) and the second identifier references a string containing a title of the object. The system of claim 9, wherein the annotation data field is a menu field ( 253 ) and the second identifier references a selector containing a set of display identifiers and a corresponding set of action identifiers. The system according to a the claims 9-11, wherein the first and second identifiers in the system are unique. The system according to a the claims 9 to 12, wherein the identifier is a variable value. A method of using broadcast information that includes hyperlinked annotation data, comprising the steps of: using a receiver ( 54 ) adapted to receive, decode, store and manipulate broadcast information, the receiver having a central processing unit ( 124 ) and at least one storage device ( 128 ) having; Receiving a stream of broadcast information; Decoding the broadcast information to obtain masking information, wherein a masking timestamp and an expiration timestamp are associated with each mask; Storing the masking information in a first data queue in the at least one memory device ( 128 ); Comparing the masking timestamp with a timestamp of a displayed video frame; Displaying the mask based on a relationship between the mask timestamp and the timestamp of the displayed video; and clearing the mask after a specific time calculated based on the expiration timestamp. The method of claim 14, further comprising the steps of: decoding the broadcast information to obtain object information, each object associated with an object timestamp; and storing the object information in a second data queue in the at least one memory device ( 128 ). The method of claim 14, wherein said decoding is performed by a first active thread (16). 514 ), which applies operations to the first data queue, and wherein the comparing is performed by a second active thread applying operations to the first data queue. The method according to any one of claims 14 to 16, wherein the step of comparing the masking timestamp with a timestamp of a displayed video frame with the comparison one earliest Masking timestamp with a timestamp of an immediately last displayed video frame begins. The method of claim 16, wherein the first thread ( 514 ) adjusts a display of masking data and the second thread adjusts a display of object data. The method according to any one of claims 14 to 18, further comprising the steps of: in the event that the masking timestamp a frame that has not yet been displayed corresponds to: (A) Decoding the associated mask into a frame buffer; (B) Check the masking timestamp to see if the decoded mask should be displayed immediately; (c) if the decoded Mask is to be displayed immediately, displaying the decoded Mask; and (d) if the decoded mask at a later time to be displayed, resting for a time calculated to end at the time at which the decoded mask should be displayed and, immediately when waking up, displaying the decoded mask.