TWI491218B - Media relay video communication - Google Patents

Description

Media relay video communication

The present invention relates to video communications and, more particularly, to the field of multipoint video conferencing.

This application is a continuation of U.S. Patent Application Serial No. 12/542,450, entitled "METHOD AND SYSTEM FOR CONDUCTING CONTINUOUS PRESENCE CONFERENCE", filed on August 17, 2009, which claims the title of the application dated January 30, 2009 </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; The present application also claims priority to U.S. Provisional Application Serial No. 61/522,474, entitled "METHOD AND SYSTEM FOR SWITCHING BETWEEN STREAMS IN A CONTINUOUS PRESENCE CONFERENCE", filed on August 11, 2011, the entire contents of which is incorporated by reference. The manner is incorporated herein.

As the traffic on the Internet Protocol (IP) network continues to grow rapidly with the growth of various videoconferencing devices, more and more people use videoconferencing as their communication tool. A multipoint conference between three or more participants requires a multipoint control unit (MCU). An MCU is a conference control entity that is typically located in one of the nodes of a network or receives one of several channels from a number of endpoints. According to certain criteria, the MCU processes the audio and visual signals and distributes them to a set of connected channels. MCU instance Contains MGC-100, RMX® 2000, both of which are available from Polycom Corporation (RMX 2000 is a registered trademark of Polycom Corporation). A terminal (which may be referred to as an endpoint) is an entity on the network that is capable of providing instant two-way audio and/or audio visual communication with other terminals or with the MCU. One of the endpoints and one of the MCUs can be found in the International Telecommunication Union ("ITU") standards (such as the H.320, H.324 and H.323 standards found on the ITU website: www.itu.int). definition.

An MCU can include a plurality of audio and video decoders, encoders, and bridges. The MCU can use a large amount of processing power to handle audio and video communications between a variable number of participants (endpoints). The communication can be based on a variety of communication protocols and compression standards and can be received from different endpoints. The MCU may need to (respectively) synthesize a plurality of input audio or video streams into at least one single output stream of audio or video, and the output stream may be sent to at least one participant (endpoint) to which the output stream is sent The nature of the product is compatible. The compressed audio stream received from the endpoints is decoded and can be analyzed to determine which audio stream to select for mixing into a single audio stream of the participant. For the purposes of this disclosure, the terms decoding and decompression are used interchangeably.

A conference can have one or more video output streams, with each output stream being associated with a layout. A layout configuration defines the occurrence of a conference on a display that receives one of the streams or one of the participants. A layout configuration can be divided into one or more segments, where each segment can be associated with a video input stream sent by a participant (endpoint). Each output stream can be composed of several input streams, resulting in a continuous presentation (CP) conference. In a CP conference, one of the remote terminals is used Several other participants in the conference can be observed at the same time. Each participant can be displayed in a segment of the layout configuration, where each segment can be the same size or size. The selections displayed and associated with the segments of the layout configuration may vary among different participants participating in the same agenda.

An MCU may need to: decode each input video stream into a full frame of uncompressed video; manage a plurality of uncompressed video streams associated with the conference; and synthesize and/or manage a plurality of output streams, each output string The stream may be associated with a participant or a specific layout configuration. The output stream can be generated by one of the MCU's video outputs. A video output port can include a layout configuration builder and an encoder. The layout configuration builder can collect different uncompressed video frames from selected participants and scale the different uncompressed video frames to their final size and place them in the layout configuration, etc. In the fragment. Thereafter, the video of the composite video frame is encoded by the encoder and sent to the appropriate endpoint. Therefore, processing and managing a plurality of video conferencing requires enormous and expensive computing resources, and thus an MCU is often an expensive and non-complex product. An MCU is disclosed in a number of patents and patent applications (e.g., U.S. Patent Nos. 6,300,973, 6, 496, 216, 5,600, 646, or 5, 838, 664), the contents of each of which are incorporated herein by reference. These patents disclose the operation of one of the video units in one of the MCUs of the video output stream that can be used to generate a CP conference.

The growing trend of using video conferencing has led to the need for low-cost MCUs that will implement multiple conference agendas with composite CP video images.

There is less than one conventional MCU resource to synthesize the compressed video stream into one The existing technology of CP video images. Some techniques disclose the use of an image processing device that synthesizes a plurality of quarter-universal intermediate format (QCIF) encoded images into a CIF image. These techniques do not require decoding of such images when compressing a plurality of encoded images using the H.261 standard. QCIF is a video conferencing format that specifies a video frame that contains 144 lines and each line has 176 pixels, which is one-quarter of the resolution of the Common Intermediate Format (CIF). Some International Telecommunications Union (ITU) video conferencing standards require QCIF support.

Other techniques to overcome the QCIF limitations of size and layout use a sub-coding approach. Such sub-coding methods are disclosed in U.S. Patent No. 7,139,015, the disclosure of which is incorporated herein in its entirety for all purposes.

Other video conferencing systems use Media Relay Conference (MRC). In the MRC, a media relay MCU (MRM) receives one or more streams from each participating media relay endpoint (MRE) (referred to herein as a relay RTP compressed video stream or a relay stream) . The MRM relays a plurality of video streams received from other endpoints in the conference to the participating endpoints, which may be referred to herein as relay RTP compressed video streams or relays. Streaming. Each receiving endpoint uses the plurality of streams to generate a CP video image in accordance with a layout configuration. The CP video image is presented to the user of the MRE. An MRE can be one of the participants in the agenda, having the ability to receive relay media from an MRM and deliver the compressed media in accordance with instructions from an MRM. MRM is described in more detail in U.S. Patent Application Serial No. 2010/0194847, the disclosure of which is incorporated herein in its entirety for all purposes. For the purposes of this disclosure, the terms endpoint and MRE are used interchangeably.

In some MRC systems, one transmission MRE is in two or more streams The video image is sent in, and each stream is associated with a different quality level. This system can use multiple streams to provide different window sizes in the layout configuration, different resolutions used by each receiving endpoint, and the like. In addition, the plurality of streams can be used to overcome packet loss. These qualities may differ in frame rate, resolution, and/or signal to noise ratio (SNR).

Video streaming is becoming more and more popular. In addition, video streaming and more and more sources of video conferencing systems deliver multiple streams in parallel, wherein the streams differ from each other in the quality of the compressed video. The number of available domains (such as time domain (eg, number of frames per second), spatial domain (eg, high definition (HD) or CIF)) and/or quality (eg, sharpness) expressive quality . Video compression standards for video streaming and multi-quality streaming include H.264 AVC, H.264 annex G, MPEG-4, and the like. More information on compression standards such as H.264 can be found on the ITU website at www.itu.int or at www.mpeg.org.

Sometimes, during an agenda, an MRE is required to receive an Intra frame from one of several MREs. The in-frame is attributable to a lost packet, a change in the layout configuration presented in the receiving MRE, joining a participant in an ongoing video conferencing agenda, and the like. In some cases, the in-frame is requested only by one of the receiving MREs (and not by other MREs participating in the agenda and obtaining the same quality level stream). An in-frame is a video frame compressed with respect to information contained only in the same frame and not related to any other frame in the video sequence. An inter frame is information about the information contained in the same frame and also compressed by one or more other frames (reference frames) in the video sequence. A video frame. An inter-frame may include a prediction frame (a P-frame) and/or a bi-predictive frame (a B-frame). In a video conference, the B frame is usually not used because of its introduction delay. In the following description, the term P-frame is used as a representative term for an inter-frame.

Video streaming may involve lost packets, skipping forward while playing video or switching between streams of different qualities. To support these capabilities, video compression standards provide special frame types that are periodically placed along such streams. The special type of the first type is a switching P frame (SP). An SP frame is similar to a P frame (using similar macroblock mode and motion compensated prediction). However, the SP frame allows the same frames to be reconstructed even when different reference frames are used to predict the same frame. The second special frame type is called the primary SP frame (SSP). This SSP frame uses a special encoding. Regardless of which reference frame, macroblock, or motion vector is used to encode the SSP frame, the decoding will always reconstruct the same picture. The special type of the third type is the frame inside the switch box (SI). The SI frame can be considered to reconstruct the frame within one of the SP frames in the same way. In the present disclosure, the terms encoding and compression are used interchangeably.

SP frames, SSP frames, and SI frames and the use of such frames for switching between different streams or loss of packets are well known in the art of video streaming and will not be discussed further. I hope to learn more about these frames and the use of such frames. Readers are asked to read the H.264 AVC standard and Michael Walter's thesis (November 26, 2004) "Advanced Bitstream Switching for Wireless Video Streaming" "." Another paper is written by Marta Karczewics et al. and published in IEEE Book 13, No. 7 (July 2003). The SP- and SI-frames Design for H.264/AVC".

In response to a request from the demand MRE to an intra-frame request (Intra request) to an associated MRE, an MRM relays the request to the associated MRE. In response, the related MRE may send an intra-frame to the MRE (including the MRE that does not require and does not require an intra-frame) to be relayed to the MRE currently received from the associated MRE. MRE. The coding efficiency of the frame in the frame is lower than the coding efficiency of the frame between the frames, so that the same quality requires a higher frequency width. In addition, the encoding/decoding of a frame within a frame takes longer than the encoding/decoding of a frame between frames and requires more computing power. Therefore, sending these intra-frames to all MREs creates an unnecessary load on the communication link and also increases the computational load in the receiving MRE and the transmitting MRE. Therefore, in order to maintain the bandwidth limitation, the in-frame frame can be encoded with lower quality. Alternatively, the frame rate may temporarily decrease during the transition period. Therefore, in a nutshell, the in-frame frame degrades the experience of the participants in the time period.

A novel method and system for reducing the impact on resources associated with an MRC agenda and enhancing the user experience when one of several MREs requires an in-frame frame is disclosed.

According to various embodiments, when an MRE (a demand MRE) request is used for an intra frame of one of the video streams received from another MRE (a presentation MRE), an MRM may request the presentation MRE The response is sent in response to a temporary stream of one of the demand MRE relays while concurrently transmitting a normal stream to be relayed to the rest of the plurality of MREs. Revealing novel techniques in the detailed description Additional embodiments of the technique.

The accompanying drawings, which are incorporated in the specification of the claims

The deficiencies of the media relay conference described above are not intended to limit the scope of the inventive concepts of the present disclosure in any way. These defects are presented only for the purpose of illustration.

In the following description, for the purposes of illustration However, it will be apparent to those skilled in the art that the present invention may be practiced without the specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the invention. Reference to a number without a subscript or a suffix is to be understood as referring to all instances corresponding to the subscript and suffix of the reference number. In addition, the language used in the present disclosure has been selected primarily for readability and indication purposes, and is not selected to depict or limit the subject matter of the present invention. The subject matter. References to "an embodiment" or "an embodiment" in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in the at least one embodiment of the invention, and The embodiment or the "an embodiment" should not be construed as necessarily all referring to the same embodiment.

Although some of the following descriptions are explicitly written with respect to software or firmware, several embodiments may implement the features and functionality described herein in the desired software, firmware, or hardware, including software, firmware, and hardware. Any combination. In the following description, the vocabulary "unit", "component", "module" and "logic mode" Groups are used interchangeably. Any device designated as a unit or module can be a single unit or a specialized or integrated module. A unit or module may be modular or have a modularized form that allows it to be easily removed or replaced with another similar unit or module. Each unit or module can be any one or any combination of software, hardware, and/or firmware, which ultimately results in the programming of one or more processors that are functional to the unit or module. In addition, multiple modules of the same or different types may be implemented by a single processor. The software of a logic module can be embodied on a computer readable medium such as a read/write hard disk, CDROM, flash memory, ROM or other memory or storage device. To perform a specific task, a software program can be loaded to an appropriate processor as needed. In the present disclosure, the terms tasks, methods, and procedures are used interchangeably.

FIG. 1 illustrates a novel multimedia conferencing system 100 in accordance with one embodiment of the present disclosure. System 100 can include a network 110, one or more media relay MCUs (MRMs) 120, and a plurality of media relay endpoints (MREs) 130. Network 110 can be any desired network, including a packet switched network, a circuit switched network, an IP network, or any combination thereof. Multimedia communications over the network may be based on communication protocols such as H.320, H.323, and SIP, and may use media compression standards (such as audio compression standards G.711 and G.719) and/or for video streaming And multi-quality streaming video compression standards, such as H.264 AVC, H.264 annex G, MPEG-4 and so on.

Upon establishing a connection with an endpoint, the MRM 120 can instruct the endpoint to deliver compressed audio. The MRM 120 can determine the audio energy of each audio stream, and accordingly select one or more audio streams to be relayed to one or more endpoints, where One or more audio streams can be decoded and mixed and sent to the endpoint speaker.

In a similar manner, the MRM 120 can instruct the endpoint to deliver the video image to match a particular size of a segment size in a layout configuration in which a compressed video image will be presented. For example, the size (height and width) can be defined by the number of pixels in each direction. In addition, MRM 120 can delegate one of the several endpoints to the current speaker endpoint and can request the current speaker endpoint delivery to fit a larger image size of one of the speaker segments in the relevant layout configuration. Video image. In some embodiments, the endpoints can be adapted to deliver compressed video images in two or more different sizes, where each video image size can fit a different segment size (resolution). In this embodiment, a previous speaker endpoint can deliver two sizes of its compressed video image: (i) one of the regular sizes to be displayed on a segment layout of a regular participant; and (ii) staying A speaker size (a larger image) that is assigned to the layout profile of the current speaker.

In other embodiments, an endpoint may transmit its video image towards MRM 120 in a plurality of streams during an entire MRC agenda. Each stream carries compressed video images of different qualities. These qualities may differ in spatial resolution (image size), frame rate, bit rate, sharpness, and the like. In addition, MRM 120 can determine which stream to relay to one or more other endpoints.

For example, one of the current speakers of a meeting agenda can be the participant with the highest audio energy. In an alternate embodiment, a speaker may be the most active attendee during a given time period. For example, the most active Participants can be defined as one of the participants who have an audio energy up to a certain percentage of the period (such as 60% or more).

Each MRE 130 is capable of providing instant two-way audio and/or visual communication to another MRE 130 or MRM 120. An MRE may be a terminal of one of the participants in the agenda, capable of receiving relay compressed media from an MRM and relaying compressed audio and video data chunks according to instructions from the MRM. Give the MRM. The compressed media can be sent as chunks of Real Time Agreement (RTP) data. Information on RTP can be found at the Internet Engineering Task Force (IETF) website at www.ietf.org. Each MRE 130 can transmit a relay RTP compressed video data chunk at (several) appropriate desired bit rate and desired compression criteria. Similarly, each MRE 130 can transmit a relay RTP compressed video data chunk at (several) appropriate desired size, bit rate or frame rate and desired compression criteria. In an embodiment, each MRE 130 can be adapted to transmit by embedding one of its audio energy indications in one of the fields in the header or in one of the extended RTP compressed audio data chunks. Audio energy indication. The term data chunks and packages are used interchangeably herein.

At the beginning of an MRC agenda, the MRM 120 can handle the needs of the conference, including the number of participants, the number of layouts, the number of participants presenting in each layout, and the size of the images of different participants. A presentation participant is a participant who presents one of his video images in a CP video image of at least one other participant and one of the participants. A participant who presents to one of the plurality of participants may receive the participant for one of the other participants. Based on an agenda The MRM 120 can negotiate with each of a number of endpoints to construct a connection for each of the plurality of streams that the endpoint can send or receive during the agenda.

Each MRE 130 can be associated with an identifier (ID) that can be carried in an RTP header of one of the media data relay RTP compressed chunks to identify the source of a received compressed audio or video packet. In an embodiment, the ID may be randomly selected by an MRE 130 and possibly confirmed by the MRM 120 after verifying its uniqueness. In another embodiment, the ID may be assigned by the MRM 120 and shipped to the associated MRE 130. The MRE 130 can write the ID into the Synchronization Source (SSRC) field in the RTP header of each of the relay compressed media data chunks. In another embodiment, the ID can be written into a contribution source (CSRS) field of the RTP header. In an alternate embodiment, the ID can be written into an extension header of each relay RTP compressed media data chunk. This ID allows the MRM 120 to identify the source of the received relay RTP compressed audio and/or video packets. Each compressed audio or video stream can be associated with its own ID. In some embodiments, the relay RTP compressed audio data chunks and the relay RTP compressed video data chunks of the same MRE 130 may have the same ID. In an alternate embodiment, the relay RTP compressed audio data chunks and the relay RTP compressed video data chunks of the same MRE 130 may have different IDs. In some embodiments, if an MRE 130 transmits its video images in a plurality of streams of different qualities to the MRM 120, each stream can be assigned a different ID. In some embodiments, each segment in a display layout configuration can be associated with an ID, and the MRM 120 can be responsible for distributing the ID of the segment to each of the agendas based on different parameters, such as, for example, audio energy. MRE 130. In another embodiment, each receiving MRE 130 can determine one of the segments in its layout configuration. The segment IDs are transmitted to the MRM 120 with the segment IDs and associated information. The information of each segment ID may include the video parameters desired by the segment, such as resolution, frame rate, and the like. As used herein, the term RTP header may include a common RTP header and also includes an extension header added to the RTP header.

In some embodiments, the ID is a number; in other embodiments, the ID can be any other value that can provide an exclusive identification of the MRE 130 or a particular stream. In other embodiments, the IP address and IP address of the relay RTP compressed audio and/or video data chunks received on the MRM 120 can be used to identify rather than as an ID number.

In one embodiment, based on the received audio energy of each participant (MRE 130), MRM 120 may determine which attendee will be presented in a CP image during a particular period of the agenda. For example, the MRE 130 with the highest audio energy can be selected, presented, and learned for a given time period of the future. The MRM 120 can further determine which of the participating conference participants will be displayed in the speaker segment in the layout configuration. In an alternate embodiment, each MRE 130 may determine which attendees will be presented in the layout configuration in which they are displayed and which attendee (MRE 130) will be displayed in the speaker segment. In such embodiments, the MRE 130 user may use one of the pick and view options disclosed in U.S. Patent Publication No. 20030174202, the disclosure of which is incorporated herein by reference. The MRM 120 may also deliver the appropriate received stream to the appropriate MRE 130.

Based on the nature of the agenda, each endpoint can build its own CP video image. According to the layout of the agenda, an endpoint can organize the received payload of compressed video in two or more compressed video segment memories (CVSMs). A packet in which each CVSM is associated with one of the segments in the layout configuration. Each CVSM can be associated with one of the compressed video stream streams that will be presented in the segment. Storing the received compressed video data in an appropriate CVSM may be based on an ID number embedded in the RTP header of the packet carrying the received compressed video. The association between a CVSM and the ID number can be dynamically changed based on the activities occurring in the agenda. For example, the association may be altered by a command from one of the MRMs 120 based on a change in one of the agendas (such as an endpoint joining or leaving the agenda) or a change in one of the speakers. An endpoint may have a cross-reference list that associates an endpoint ID with a segment of a layout configuration. The cross-reference list can be updated during the agenda to reflect the dynamics of the agenda. For example, the data in the organization-CVSM can be based on the serial number of the packet or the timestamp of the frame embedded in the RTP header.

In an embodiment, the endpoint may be adapted to transmit the audio energy indication by embedding one of its audio energy indications in one of the RTP headers or the extension header of the RTP packet. In this embodiment, the MRM 120 may parse the header of the RTP carrying the audio data to determine the audio energy of each endpoint and select the speaker to attend the conference and/or present the conference attendees accordingly. In an alternate embodiment, MRM 120 may instruct the endpoints to send an indication of the audio energy of the endpoints on a signaling or control connection, or alternatively MRM 120 may decode the received audio stream and determine the The energy of the audio stream.

In other embodiments in which audio compression complies with, for example, compression standard G.7221.C or G.719, the endpoint's audio codec can be configured to add one of the audio energy indications to the audio header. In this embodiment, the MRM 120 can be adjusted It is suitable to search the header of the audio payload and capture the field of the audio energy. In other embodiments, the indication of the audio energy can be sent from the endpoints to the MRM 120 via a dedicated out-of-band status and control connection.

In addition, an endpoint can be adapted to decode the stored compressed video obtained from each of the plurality of CVSMs. The decoded video can be stored in a segment frame memory (FM) module. A segment FM stores the decoded video data to be presented in the CP at the relevant segment. In some embodiments, a scaler can be added between the decoder and its associated segment FM and can be used to adjust the size of the received image to the associated segment. In still other embodiments, the decoded video may be stored in a section of one of the CP images associated with one of the associated CVSMs.

In an embodiment, a CVSM can be associated with a specific ID for an entire agenda. In this embodiment, the MRM 120 can dynamically associate one of the IDs representing a CVSM (a segment of a layout) with the compressed video material to be displayed in the associated segment. In this embodiment, there is no need to notify a receiving endpoint of a change in the layout configuration. The MRM 120 may manage the changes by associating a segment's associated ID with associated compressed video material sent from a participant and presenting the relevant segment. In some embodiments, the ID representing one of the segments in a layout configuration may be added to the source ID or the source ID may be replaced by the segment ID.

An MRM 120 may determine that one of the plurality of receiving MREs 130 (referred to as the demanding MRE 130) requires an in-frame from one of the plurality of rendering MREs 130. The decision may be based on an in-frame request received from the demand MRE 130. For example, you can respond to a packet loss or demand MRE 130 attendance meeting The user desires to add the video image of the MRE 130 to the required MRE 130 layout configuration to send the request. In some cases, MRM 120 may determine that an in-frame frame from one of the present participants is to be sent to a demand MRE 130. For example, when the demand MRE 130 joins an ongoing MRC agenda, the MRE 130 then needs to obtain an in-frame from each of the present participants. In conventional video conferencing systems, in such cases, the in-frame is sent to all endpoints of the received video stream, even if the endpoint does not require the in-frame. The disclosed technique avoids transmitting the in-frame frame to the MRE 130 that does not require the in-frame frame.

In an embodiment, an MRM 120 may further verify whether the decoder of the demand MRE 130 has been processed by the decoder from the presentation MRE 130 when determining that a demand MRE 130 requires an in-frame from one of the presentation MREs 130. One or more reference frames are decompressed when the video frame was previously received. As is well known in the art, H.264, MPEG-4, and similar encoders and decoders can aggregate and use a plurality of reference frames, such as up to 16 frames. The encoder determines which of the reference frames is used to encode the current video image when processing a current video image. In addition, the encoder will specify which reference frame is used to encode the header of the current compressed video frame indicating the addition to the current compressed video image. The indication is used by the decoder receiving the MRE to synchronize the received decoding of the currently compressed video image with the encoding process.

At the beginning of the MRC video agenda, an MRM 120 may instruct the participating MREs 130 to store a number of reference frames to be used in the encoding/decoding process. The reference frames can be the last two frames, the last four frames, and the last eight Frames, etc. During the ongoing agenda, the MRM 120 may need to know which of the reference frames from the presentation MRE 130 the decoder of the demanding MRE 130 has. In an embodiment, the demand MRE 130 may signal one of the in-frame frames with the frame number corresponding to the last stored reference frame to replace one of the requests.

In other embodiments, the MRM 120 may store the frame numbers of the last few frames sent from each of the plurality of MREs 130 in a table or other conventional data structure. Based on this table, the timing of receiving a request for an in-frame from a demand MRE 130 and the event of receiving a frame from the presentation MRE 130 at the MRM 120 is due to the fact that the frame is not received and decoded at the demand MRE 130. The MRM 120 can estimate which reference frame is present in the demand MRE 130, by obtaining an estimated delay between events within a frame request.

If the MRM 120 determines that the decoder of the demand MRE 130 has a previous reference frame (this frame may be referred to as an existing previous reference frame), the MRM 120 may indicate that the encoder of the associated stream of the endpoints of the presentation is based on the There is a previous reference frame to compress the next video frame into an inter-frame and delete its previously stored reference frame. In this case, all receiving MREs 130 obtain an inter-frame that is compressed based on an older previous frame and cause all of the receiving MREs 130 to be synchronized with the rendering MRE again.

If the MRM 120 determines that the decoder requesting the MRE 130 requires an in-frame from the presentation MRE 130, the MRM 120 may begin to synchronize all of the reference frames of the decoder that receives the MRE 130 (including the required MRE 130), Instead of sending an in-frame to all receiving MREs 130. In the case where a presentation MRE 130 sends one or more streams, the term is presented The normal stream of MRE 10 may refer to one of the other MREs 130 that are sent to the same quality level that receives the demand MRE 130.

In some embodiments, multiple consecutive frames may be involved in the synchronization process. One reason for extending the program across multiple frames is to avoid the presence of computing resources in the MRE 130 and the network resources between the MRE 130 and the MRM 120 and the network between the MRM 120 and the demand MRE 130. A jump occurs in the consumption of resources. At some point in time, when the presentation MRE 130 transmits, for example, the frame number M-1, the MRM 120 may instruct the presentation MRE 130 to send an additional stream (a temporary stream) in parallel with the normal stream. The temporary stream generates the temporary stream by encoding a reference stream of a normal stream, rather than generating the temporary stream from the video image received by the camera. The first coded frame of the temporary stream is an in-frame frame. For example, the in-frame frame has the same spatial resolution as the normal stream but is compressed with lower quality, smaller image sharpness, and lower frame rate.

When the MRM 120 command is received by the presentation MRE 130, for example, the frame M from the camera presenting the MRE 130 may be compressed into an inter-frame by one or more quality levels and sent to the MRM 120. In addition, a temporary stream is initiated by compressing the reference frame of the normal stream for frame M into a low quality inter-frame (TIm) and sent to the MRM 120, which in turn will be in the temporary stream. Continue to demand MRE 130.

The MRE 130 is required to decode the TIm and the low quality decoded image is placed in the CP image and presented to the participant. The decoder produces in parallel one of the references to the frame M of the normal stream presenting the MRE 130.

The encoder presenting the MRE 130 continues to encode the temporary stream by encoding the reference memory for the normal stream of M+1, M+2, etc. (up to frame N) as an interframe. The number of frames may be in the range between zero frames and up to a predetermined number of frames (eg, 10 frames). These frames can be referred to as quality-enhanced frames, because the compression quality of the frames between the frames is higher than the previous ones. The frame N received by the video camera is compressed into an SP frame. The SP frame is sent to other receiving endpoints as a P-frame in normal streaming. This frame can be called SPn. The reference frame generated at the encoder presenting the MRE 130 generated when encoding the SPn frame is compressed into an SSPn. The SSPn frame is sent to the demand MRE 130 as the last frame of the temporary stream.

The demand MRE 130 decodes the received SSPn, thereby generating a reference frame for one of the frames N. The decoded image is placed in the CP video image and presented as frame N. At this point in time, the reference frame generated due to decoding SSPn is identical to the reference frame memory generated at other receiving MREs 130 having the same quality level and the reference memory presenting the normal stream in MRE 130. The next frame N+1 and above is encoded once and sent as a normal stream to the demand MRE 130 and other receiving MREs 130 having the same quality level.

In another embodiment, a temporary stream may be generated from a video image received from the camera, rather than a temporary stream from the reference frame of the encoder of the normal stream in the above example. In this embodiment, the SP frame and the SSP frame are generated from the same video image received from the camera. At some point in time, such as when the presentation MRE 130 transmits the frame number M-1, the MRM 120 can instruct the presentation MRE 130 to send an additional in parallel with the normal stream. Streaming (a temporary stream). The temporary stream is generated by encoding a video frame received from the video camera when encoding the normal stream. The first coded frame of the temporary stream is an in-frame frame (TIm) of the frame M. The frame in the frame has the same spatial resolution as the normal stream but is compressed with lower quality, smaller image sharpness, lower frame rate, and the like.

The MRE 130 is required to decode the TIm and the low quality decoded image is placed in the CP image and presented to the conference participant. The decoder produces a reference frame in one of the frames M from the presentation MRE 130 in parallel. The encoder presenting the MRE 130 continues to encode the temporary stream by encoding the next video frame received from the video camera into an inter-frame. The inter-frames are related to frames M+1, M+2, etc. (up to frame N). The number of frames may be in the range between zero frames and up to a predetermined number of frames (eg, 10 frames). These frames can be referred to as quality-enhanced frames, because the compression quality of the frames between the frames is higher than the previous ones.

The frame N of the normal stream and the frame between the other frames are compressed differently. Frame N is compressed into an SP frame. At this point in time, the demand MRE 130 has a quality similar to that of at least one other receiving MRE 130. However, the reference frame memory of the demand MRE 130 may not be similar to other receiving MREs 130.

Thus, MRM 120 can synchronize the reference frame of demand MRE 130 with other receiving MREs 130 having the same quality stream. In order to synchronize the reference frame of the demand MRE 130 with the other receiving MREs 130, the MRM 120 may instruct the encoder presenting the MRE 130 to encode the frame N of the temporary stream into an SSPn frame, and the SSPn frame serves as the temporary string. The last frame of the stream is sent to the demand MRE 130.

The demand MRE 130 decodes the received SSPn, thereby generating a reference frame for one of the frames N. The decoded image is placed in the CP video image and presented as frame N. At this point, the reference frame memory generated by the self-decoding SSPn is identical to the reference frame memory generated by the other receiving MREs 130 having the same quality level and the reference memory presenting the normal stream in the MRE 130. The next frame N+1 and above is encoded once and sent as a normal stream to the demand MRE 130 and other receiving MREs 130 having the same quality level.

In another embodiment using a plurality of previously encoded reference frames, a lossless compression can be used, for example, when the presentation MRE 130 transmits the frame number M-1, transmitting one of the desired parameter sets to the previous reference frame toward the demand MRE. 130 and then encode the next frame based on the previous reference frame. The MRM 120 may instruct the presentation MRE 130 to send an additional stream (a temporary stream) in parallel with one or more normal streams. The temporary stream is generated by lossless compression of the reference frame generated when encoding the frame M-1 of the normal stream. The lossless compressed reference frame (LLCRF) of frame M-1 may be transmitted out of band on the IP connection from the presentation MRE 130 to the demand MRE 130 via the MRM 120. The temporary stream can also carry the desired parameter set, such as "picture parameter set", "sequence parameter set" and the like. Information about the desired parameter set can be found in a compression standard such as H.264.

In another embodiment, the presentation MRE 130 can directly transmit the temporary stream towards the demand MRE 130. The sending of the LLCRF and the desired parameter set can be done in a few packets and can take one or more frame intervals. The lossless compression method can be any lossless method such as ZIP, Lempel-Ziv=Welch (LZW), lossless JPEG 2000 and so on.

In still another embodiment, a demand MRE 130 may have received a stream of relay RTP compressed video data chunks from a presentation MRE 130 at a first resolution, while other MREs 130 may have a second resolution. The presentation MRE 130 receives a stream of relay RTP compressed video data chunks. The stream at this second resolution may be referred to as a normal stream. At a particular time, the demand MRE 130 may need to change the received stream resolution from the first resolution to the second resolution. In this embodiment, the presentation MRE 130 can allocate a temporary encoder. The temporary encoder can be loaded with one of the first resolution transition reference frames. Then, the resolution of the load transition reference frame can be changed from the first resolution to the second resolution. The resolution of one of the reference MREs 130, which is equivalent to the transition reference frame at the presentation MRE 130, may be changed in parallel from the first resolution to the one in a manner similar to that done on the presentation MRE 130. Second resolution. Thus, at this point, the temporary encoder at the MRE 130 and the decoder at the demand MRE 130 have the same reference frame.

The immediately following one or more video images received by the video camera presenting the MRE 130 may be compressed by the temporary encoder at a second resolution and transmitted to the demand MRE 130 in parallel with the normal stream transmitted to the other receiving endpoints. . After one or more video frames, the conventional encoder can compress a video image received by the video camera into an SP frame and transmit it to the one or more receiving MREs 130 on the normal stream. The temporary encoder can encode the same video image as an SSP frame and transmit it to the demand MRE 130. Presenting the MRE after transmitting the SP frame and the SSP frame The temporary encoder may be released and the demand MRE 130 may receive the normal stream as other one or more receiving MREs 130. In still another embodiment, the transition reference frame can be used to generate the SP frame and the SSP frame and send the SP frame and the SSP frame.

The demand MRE 130 uses a lossless algorithm to decompress the obtained LLCRF and the desired parameter set. Next, the decoder of the MRE 130 is loaded with a decompressed reference frame and the desired parameter set. Therefore, the decoder has the same reference frame as the reference frame M-1 and the same desired parameter set of the other receiving MREs 130. The encoder presenting the MRE 130 may encode the next camera frame based on the reference frame M-1 and send the resulting inter-frame to a number of receiving MREs 130 (including the demand MRE 130). In addition, an encoder that requires MRE 130 can delete all previous reference frames.

Another embodiment can be combined using one of the techniques disclosed above. This embodiment may use a temporal stream that is the same resolution as a normal stream but starts with a lower quality frame within a frame, followed by a few quality enhanced frames. In a typical video conferencing agenda, the changes between consecutive frames are minimal, so after a few quality-enhanced frames, the uncompressed frame below the input of the temporary streamed encoder is quite similar to the encoding. The reference frame produced after the previous frame. In this embodiment, the presentation MRE 130 may use a lossless compression technique to compress the difference between the reference frame of the normal stream and the reference frame of the temporary stream and send the difference to the demand MRE 130. The demand MRE 130 can decompress the equal difference and correct its decoder reference frame accordingly.

An encoder using both normal and temporary streams uses a lossless compression technique Compressing the same frame will cause the same reference frame to be generated in the encoder and decoder of the two streams, so the next frame can be compressed by the normal encoder, and the normal stream can be transmitted to the demand MRE 130 and others. Receive MRE 130.

In an embodiment, as disclosed above, the MRM 120 controls the activity associated with the temporary stream by indicating the presentation of the MRE 130. In other embodiments, MRM 120 may initiate the program merely by assigning a channel for carrying a temporary stream between presentation MRE 130 and MRM 120. From this point on, the MRE 130 is configured to automatically execute the entire program starting at a low quality in-frame frame and ending at SSPn, then closing the additional channels for the temporary stream. In an alternate embodiment, MRM 120 may begin the process but present MRE 130 and demand MRE 130 may negotiate when the temporary stream moves from one program to another.

In some embodiments, the encoder allocated to generate the temporary stream may be one of the encoders separate from the encoder of the normal stream. The separate encoder can access the reference memory of the normal stream encoder. Moreover, the separate encoder can direct the normal encoder during the transition period and indicate when the normal encoder performs special encoding between the actions of the encoder in a modified order. In other embodiments, a single encoder can be configured to generate both normal and temporary streams.

The MRE 130 can decode the received relayed RTP compressed video stream from the data chunks of the video images selected by the participating conference participants and display the images in the appropriate layout configuration segments. MRE 130 can decode the received relayed RTP compressed audio stream of audio data chunks, and mix different decoded audio strings Stream and transmit the mixed audio to the MRE 130 speaker. In other aspects, the MRE 130 can deliver relay RTP compressed audio and video data chunks in accordance with instructions from the MRM 120.

In addition to the above functionality of an MRE 130, one of the compressed video to be relayed to two or more receiving MREs 130 is normally streamed to one of the MRMs 120. The presentation of the MRE 130 may be in a novel manner in response to a A request for a frame in the MRE 130 is required. In an embodiment where the encoder and decoder store and use a plurality of previous reference frames, the presentation MRE 130 can respond by providing an inter-frame to all involved MREs 130 instead of encoding a frame. The inner frame is sent to the MRM 120 as one of the normal frames to be relayed to the normal stream of two or more MREs 130 involved. The provided inter-frame is encoded based on one of the same reference frames (IRFs) present in all of the MREs 130 that are involved in the streaming of the compressed video. The coded interframe is sent towards the MRE 130 involved and then the encoder releases all previously stored reference frames. The decoder MRE 130 and other decoders that receive the MRE 130 may decode the provided inter-frame coding frame based on the previously stored copy of the IRF.

In an embodiment where the encoder and decoder use one of the plurality of previous reference frames MRE 130 for encoding/decoding a current frame, the encoder presenting the MRE 130 can be notified in the decoder of the demand MRE 130. There is a previous reference frame. The encoder can use this reference frame as an IRF, which is the same as one of the reference frames in other receiving MREs 130. A video image received from the camera presenting the MRE 130 can be compressed into an inter-frame based on the IRF, and the presentation MRE 130 can be released. The rest of the previous reference frame of the encoder.

In some embodiments, a presentation MRE 130 can establish a temporary connection with the MRM 120 for carrying a novel temporary stream of compressed video in parallel with normal streaming. This temporary stream can be relayed by MRM 120 to demand MRE 130. The temporary stream is used to cause the decoder of the required MRE to prepare to receive and decode the future interframe frame belonging to the normal stream and also to one of the other receiving MREs 130. The temporary stream and the normal stream have the same resolution. However, the temporary stream can begin with a low quality in-frame frame and the quality can be improved from the inter-frame to the next inter-frame.

In order to switch from the temporary stream to the normal stream, the presentation MRE 130 can encode one of the next video images received from the camera into one SP frame of the normal stream, and encode the corresponding frame of the temporary stream into one. SSP frame. The result of decoding the SP frame and the SSP frame may generate an IRF in the decoders of the other receiving MRE 130 and the demand MRE 130, respectively. The reference frame of the normal encoder is also the same as the IRF. Thus, a video image received from the camera can be encoded based on the IRF and sent to all involved MREs 130 on the normal stream, including the demand MRE 130. In an embodiment, the frame of the temporary stream may be generated by self-encoding the reference frame of the normal stream. In yet another embodiment, both normal and temporary streams can be generated from video images received from a video camera that presents the MRE.

Yet another embodiment of the disclosed system can be responsive to one of the in-frames by constructing a temporary connection for carrying one of the temporary streams in parallel with one or more normal streams. The temporary stream can be generated by lossless compression of one of the reference frames when encoding a recent frame of the normal stream. For example, the temporary connection can be opened out of band on the IP connection from the presentation MRE 130 to the demand MRE 130 via the MRM 120. The Transmit Lossless Compression Reference Frame (LLCRF) can be done with a few packets and can take one or more frame intervals. The lossless compression method may be ZIP, Lempel-Ziv-Welch (LZW), lossless JPEG 2000, or the like. In addition to transmitting the LLCRF, the presentation MRE 130 may also send the desired parameters for decoding, such as one of the "picture parameter sets" or a "sequence parameter set" as disclosed in the H.264 compression standard.

An embodiment can be combined using one of the methods disclosed above. In this embodiment, the presentation of the MRE 130 allows the temporary stream to start at the same resolution as the normal stream but starts with an in-frame within a lower quality, followed by some quality-enhanced frames. After some quality-enhanced frames, the reference frame at the encoder of the temporary stream is similar to the reference frame at the encoder of the normal stream. At this point in time, the difference between the reference frames of the two encoders can be calculated and compressed by a lossless compression method. This compression difference can be transmitted to the demand MRE 130. At the demand MRE 130, the lossless compression difference can be decompressed without loss and added to the reference frame at the decoder of the demand MRE 130 to generate an IRF at the decoder of the demand MRE 130 and other decoders that receive the MRE 130. . At this point in time, the temporary stream can be terminated and the normal stream can also be transmitted towards the demand MRE 130.

Another embodiment can be combined using one of the methods disclosed above. In this embodiment, the presentation MRE 130 enables a temporary stream to begin at the same resolution as a normal stream but starts with a lower quality frame, followed by a few quality enhanced frames. After a few quality-enhanced frames, at the moment The input of the streamed encoder is similar to that of the uncompressed frame received by the camera presenting the MRE 130, similar to the reference frame generated after encoding the previous frame. In this embodiment, the normal stream encoder and the temporary stream encoder can compress the difference between the next frame and the reference frame in a lossless compression method.

Compressing the same frame in a lossless compression method by an encoder presenting both normal and temporary streams at MRE 130 will produce the same reference frame in the encoder and decoder of the two streams, and thus may be rendered The normal encoder of the MRE 130 compresses the next frame and can transmit a single stream (the normal stream) towards the MRM 120 that spreads it to the demand MRE 130 and other receiving MREs 130. The normal stream and the lossless compression of the temporary stream can be accomplished by a quantizer module in both the bypass encoder and the decoder. More detailed information on the MRE 130 is disclosed below in conjunction with FIG. 11B.

In another embodiment, the novel MRM 120 can be configured to initiate a process for requesting a request from an in-frame of an MRE 130 by a demand MRE 130, and can be managed by the presentation MRE 130 and the demand MRE 130 The rest of the activities.

In yet another embodiment in which multicast is used, as long as the temporary stream is used, the MRM 120 can instruct a demand MRE 130 to listen to a new multicast address and then listen to the multicast address associated with the normal stream. The MRM 120 can instruct the presentation MRE 130 to start generating the temporary stream in parallel and send the temporary stream to the new multicast address in parallel to the normal stream to the multicast address associated with the normal stream. . More detailed information of the MRM 120 is disclosed below in conjunction with FIGS. 2, 3, 4, 8A, and 8B.

2 is a block diagram of one of the components of the MRM 120 in accordance with an embodiment. For example, the MRM 120 can include a network interface module (NI) 220, one or more agenda compressed audio RTP processors (SCARP) 230, a signaling and control module (SCM) 240, and one or more agenda compressions. Video RTP Processor (SCVRP) 250. In an alternate embodiment, MRM 120 may include one or more SCMs 240 per agenda.

A network interface module (NI) 220 can communicate with a plurality of video conferencing devices, such as MRE 130, via network 110. The network interface 220 can parse the communication in accordance with one or more communication standards, such as H.323 and SIP. In addition, the network interface module (NI) 220 can process the physical layer, data link layer, network layer, and transport layer (UDP/TCP layer) of the received communication. The NI 220 can receive and transmit control and data information to and from the MRM 120 and MRE 130 or other nodes (not shown in the drawings).

NI 220 multiplex/demultiplexing delivers different signals and streams between the MRE 130 and the MRM 120. RTP slabs for compressed audio can be transmitted to and from the MRE 120 and SCARP 230 via the NI 220. Each SCARP 230 can be associated with a particular agenda. The NI 220 can determine which agenda an MRE 130 participates based on the source and/or destination IP address and/or IP address and/or ID of the MRE 130 packet, thereby enabling the NI 220 to determine which SCARP 230 should be received from a particular MRE 130. RTP chunks of compressed audio received. In other aspects, the RTP chunks of compressed audio received from SCARP 230 can be converted to IP packets by NI 220 and transmitted towards the appropriate MRE 130 or other node.

RTP compressed chunks for transmitting video data to and from MRE 130 and SCVRP 250 via NI 220. Each SCVRP 250 can be associated with a video conferencing agenda Association. The NI 220 can determine which agenda an MRE 130 participates based on the source and/or destination IP address and/or source ID of the MRE 130 packet, thereby enabling the NI 220 to determine which SCVRP 250 should be received from a particular MRE 130. The received packet. In other aspects, the RTP chunks of compressed video received from SCVRP 250 can be converted to IP packets by NI 220 and transmitted towards the appropriate MRE 130 or other node.

The NI 220 can also transmit and receive signaling and control data to and from the SCM 240 and MRE 130. The NI 220 may also handle signaling and control in response to a need to send a frame from a presentation MRE 130 to an in-frame MRE 130.

For each meeting agenda disposed by MRM 120, a SCARP 230 may be allocated for handling agenda audio. A SCARP 230 can receive the relay RTP compressed chunks (header and payload) of the audio material from the MRE 130 participating in the agenda via the NI 220. SCARP 230 can manage a plurality of MRE sequential memories, and an MRE sequential memory is used to participate in each MRE 130 of the agenda. The SCARP 230 may parse the RTP header from the received relay RTP compressed audio chunk of one of the MREs 130 and store the chunk in the appropriate MRE sequential memory. The SCARP 230 may also determine an appropriate order based on a sequence number or a time stamp embedded in the relay RTP header to store the relay RTP compressed audio chunks in the MRE sequential memory.

The SCARP 230 collects information on the audio energy of each MRE 130. In one embodiment, the audio energy can be obtained by parsing the appropriate fields in the relay RTP header of each of the received relay RTP compressed audio chunks. In another In one embodiment, the audio energy can be obtained by sampling the audio energy level of the received relay RTP compressed audio chunk.

The SCARP 230 may periodically (typically each tens of milliseconds) select a group of relay RTP compressed streams to be learned and thus transmitted to the audio chunks of the MRE 130. The selection can be based on comparing the audio energy associated with the received streams. The number of selected relay RTP compressed streams depends on the audio mixing capabilities of the MRE 130. The SCARP 230 may also select which MRE 130 is the primary speaker (e.g., the speaker to be displayed in the largest layout profile) and forward the signaling and control information to the SCM 240 accordingly. The primary speaker may be a speaker having a maximum percentage of audio energy for a certain percentage of the learned-streaming-selection interval over a period of time.

In yet another alternative embodiment, SCARP 230 may forward information of the audio energy of MRE 130 to SCM 240. The SCM 240 will select the primary spokesperson of the MRE 130 and the group of RTP compressed streams that will be informed of the (mixed) audio data, and will send signaling and control data to the appropriate SCARP 230 and SCVRP 250. In some embodiments, information of selected groups and/or primary speakers of the attendees may also be transmitted to the MRE 130. Based on the signaling and control data sent from the SCM 240, the SCARP 230 can configure the selected group of relay RTP compressed audio chunks and relay them to the appropriate compressed audio data chunks via the NI 220. MRE 130. More information about SCARP 230 is disclosed below in conjunction with FIG.

An SCVRP 250 is assigned to each video conferencing agenda dispositioned by the MRM 120. An SCVRP 250 can receive a stream of relay RTP compressed video chunks (header and payload) from the MRE 130 participating in the agenda via the NI 220. The SCVRP 250 can manage a plurality of MRE sequential memories, and an MRE sequential memory is used to participate in each MRE 130 of the agenda. The SCVRP 250 may parse the header of the received relay RTP compressed video chunk and store the header in the appropriate MRE sequential memory based on the frame number or time stamp of the header. Sometimes, depending on the frame rate used by the agenda, the SCVRP 250 can access one or more MRE sequential memory groups and pass through the NI 220 based on the signals and control information received from the SCM 240. The data for the group following the MRE 130 is selected to the appropriate MRE 130.

During a transition period, the SCVRP 250 may be assigned an additional sequential memory in addition to its normal operation. The additional sequential memory can be used to store the temporary stream received from a presentation MRE 130 and the target is directed to a demand MRE 130 in parallel with the conventional one or more streams received from the presentation MRE 130. In addition, this temporary memory can be drawn toward the demand MRE 130 instead of one of several conventional streams having similar quality levels. More information about SCVRP 250 is disclosed below in conjunction with FIG.

The SCM 240 controls the operation of an agenda. The SCM 240 can initiate a meeting agenda (reserved or impromptu), set up a connection with an endpoint, determine the nature of the conference, and set the EP to transmit and receive media accordingly. The SCM 240 can further allocate resources, assign IDs, indicate that the encoder of the endpoint uses a plurality of reference frames, and the like. Occasionally, the SCM may select a new speaker based on the audio energy and video source to be sent to each endpoint as directed by one of the speakers attending the meeting. This new selection can be transmitted to SCARP 230 and SCVRP 250 accordingly. The SCM 240 can instruct the EP to send appropriate video parameters based on changes in the layout configuration.

The SCM 240 can determine which RTP compressed stream of video data is relayed to which MRE 130 and in which layout segment the video image is presented. Based on the instructions received from the SCM 240, the SCVRP 250 relays the appropriate streams to the MRE 130 and can instruct the MRE 130 in which of the layout segments to present each of the video images. In one embodiment, SCVRP 250 may notify MRE 130 of the changes that occurred in presenting the present participants by changing the ID in the RTP header of the relay RTP compressed video data chunk that it sent. The change ID in the header can be used as an indication for the receiving MRE 130 to specify which of the CP images should display an indication of the relay RTP compressed video data chunks. In an alternate embodiment, SCM 240 may notify MRE 130 of such changes via NI 220 by signaling and controlling the data. In still another embodiment, SCVRP 250 may notify MRE 130 of the changes by setting a predefined one of the RTP headers of the relay RTP compressed video data chunks it sends. The predefined fields in the RTP header can be used to indicate in which segment the MRE 130 should display the relay RTP compressed video data chunks.

In addition to the above activities, the SCM 240 can also handle transitions and switching cycles. The SCM 240 may respond to an in-frame request from one of the plurality of receiving MREs 130. The SCM 240 can determine the reason for the request, such as whether the request for a frame within the frame is due to a change in the layout configuration or to a new participant joining the meeting. The SCM 240 manages the temporary connections used to carry the unique stream from the presentation MRE 130. Some embodiments of an MRM 120 may respond to an in-frame request as follows: the SCM only initiates the transition period, and the MRE 130 and the demand MRE 130 manage the number of temporary low quality frames and when to switch back to the conventional Streaming. Hereinafter, in conjunction with FIG. 3 and FIG. 4, Figures 8 and 11 reveal more information about the MRM 120.

3 illustrates a simplified block diagram of related components having SCARP 230, one of the techniques and elements of implementing the various embodiments. The SCARP 230 can include an RTP audio input buffer 310, an audio RTP header parser and organizer 320, a plurality of MRE sequential audio memories 330, an internal bus or a shared memory bus 340, A plurality of RTP compressed audio stream builders 350 and an audio energy processor 360. The SCARP 230 can be controlled by the SCM 240 via, for example, one of the internal bus bars or one of the shared memory controls the bus 365. The input and output of SCARP 230 can be coupled to NI 220 via a compressed RTP audio data interface (CRADI) 305 for receiving and transmitting compressed audio data chunks. The CRADI 305 can be an internal bus or a shared memory.

Each SCARP 230 can be assigned to handle the audio of one of the meeting agendas handled by the MRM 120. A SCARP 230 RTP audio input buffer 310 can obtain relay RTP compressed audio data chunks from CRADI 305. The MRE 130 participating in the agenda receives the chunks of the relay RTP compressed audio data. In one embodiment, the RTP audio input buffer 310 can determine which relay RTP compressed audio data chunks are obtained from the CRADI 305 by using the ID number in the relay RTP header. In an alternate embodiment, the RTP audio input buffer 310 can receive the relay RTP compressed audio data chunk from the NI 220 based on the source and/or destination IP address and port number of the received associated packet.

An audio RTP header parser and organizer 320 can retrieve the relay RTP compressed audio data chunk from the RTP audio input buffer 320, and parse the header of the relay RTP compressed audio data chunk for use in capturing Related information, Such as: the ID, serial number and / or time stamp of the thick blocks, and the audio energy (if it exists). In some embodiments, the audio header can be parsed to capture audio energy information. For example, based on the ID, the audio RTP header parser and organizer 320 can transmit the parsed relay RTP compressed audio data chunk to the appropriate MRE sequential audio memory and transmit the audio energy to the audio energy processor 360. .

Each MRE sequential audio memory 330 can be associated with an MRE 130 (FIG. 1). The received relay RTP compressed audio data chunks from the associated MRE may be stored in the MRE sequential audio memory 330 based on their serial number and/or time stamp. Each MRE sequential audio memory 330 can be accessed via the bus 340. The bus 340 is connected to one or more of the MRE sequential audio memory 330 and the plurality of RTP compressed audio stream builders 350.

Each RTP compressed audio stream builder 350 can be assigned to one or more MREs 130. An RTP compressed audio stream builder 350 can include an MRE multiplexer sequencer 352 and an MRE RTP audio output buffer 354. The RTP compression streamizer 350 can select one or a group of sources of compressed audio relay data chunks by accessing one or more MRE sequential audio memory 330. The group selection may be based on different parameters, such as control signals received from the audio energy processor 360, user specifications independent of one of its specific sources, or audio mixing capabilities of the destination MRE 130. Typically, groups of such selected sources do not include audio streams received from destination MRE 130. In an alternate embodiment, RTP compression streamizer 350 may receive control signals from destination MRE 10 regarding which MREs 130 are selected. In addition, sometimes the RTP compressed audio stream builder 350 can be changed according to the instant change in the conference. And change its input selection.

An MRE multiplexer sequencer 352 can select one or more input relay RTP compressed audio data chunk streams from the bus 340. The selected relay RTP compressed audio data thick block stream can be multiplexed into a relay RTP compressed audio data thick block stream, and the relay RTP compressed audio data thick block stream is sent to the MRE RTP audio output FIFO 354 and The relay RTP compressed audio data chunk stream is transmitted from the MRE RTP audio output FIFO 354 via the CRADI 305 and NI 220 towards the appropriate MRE 130.

An alternate embodiment of the RTP compressed audio stream builder 350 (not shown in the drawings) may include a group of selectors. Each selector is coupled to bus 340 and may select the output of one of a plurality of MRE sequential audio memories 330. The other ports of the selector can be connected to the CRADI 305 via the FIFO. In this embodiment, the selected audio stream is transmitted towards the MRE 130 as a plurality of relay RTP compressed audio data chunk streams.

In an alternate embodiment, an RTP compressed audio stream builder 350 can be used to servo a group of participants participating in a conference agenda, wherein all associated MREs 130 will receive the same relay RTP compressed audio data chunks. Streaming.

An audio energy processor 360 can receive audio energy associated with each of the relay RTP compressed audio data chunks, and the audio energy processor 360 determines based on this information which MRE 130 will be selected for mixing in the next time period and the selected transmission The MRE multiplexer sequencer 352 is compressed to the appropriate RTP compressed audio streamer 350. Additionally, the audio energy processor 360 can determine which endpoint should be presented as a primary speaker, as described above. Specify a new active hair This information can be delivered to the SCM 240 via the control bus 365.

4 is a simplified block diagram of one of the related elements of SCVRP 250 having one of the techniques and elements of implementing the various embodiments. An SCVRP 250 can include an RTP video input buffer 410, a video RTP header parser and organizer 420, one or more MRE sequential video memories 430, such as an internal bus or a shared memory. Bus 440 and one or more RTP compressed video stream builders 450. The SCVRP 250 can receive control from an SCM 240 via, for example, one of the internal bus bars or one of the shared memory controls the bus 465. The compressed input and output video data chunks can be communicated between the SCVRP 250 and the NI 220 via a compressed RTP video data interface (CRVDI) 405 (eg, which can be an internal bus or a shared memory).

Each SCVRP 250 manages the video of a conference agenda. An RTP video input buffer 410 can obtain, from the NI 220, a relay RTP compressed video data chunk received from the MRE 130 participating in the agenda via the CRVDI 405. In one embodiment, for example, the RTP video input buffer 410 can determine which relay RTP compressed video data chunk is handled by the ID number embedded in the RTP header. In an alternate embodiment, RTP video input buffer 410 receives video chunks from NI 220 based on the source and/or destination IP address and port number associated with the associated packet.

The video RTP header parser and organizer 420 can retrieve the relay RTP compressed video data chunks from the RTP video input buffer 410 and parse the headers of the relay RTP compressed video data chunks for capturing relevant information. For example, an ID, a serial number, and/or a timestamp and/or a frame number having a first macroblock address (MBA) associated with each video chunk. Based on the selected The information, video RTP header parser and organizer 420 can store the associated relay RTP compressed video data chunks in the appropriate MRE sequential video memory 430.

Each MRE sequential video memory 430 is associated with one of the streams received from one of the participating agendas MRE 130. Each stream received from an MRE 130 can carry a compressed video of a different quality level and/or resolution. Each output of the MRE sequential video memory 430 is linked to a bus 440 that connects one or more of all MRE sequential video memory 430 and a plurality of RTP compressed video stream builders 450.

During a transition period, an SCM 240 may assign an additional temporary MRE sequential video memory 430 that will be assigned to one of the temporary streams to be transmitted from the presentation MRE 130. In addition, the video RTP header parser and organizer 420, the RTP video input buffer 410, and the NI 220 can be notified of the ID of the temporary stream to deliver the ID to the temporary MRE sequential video memory 430.

Each RTP compressed video stream builder 450 can be assigned to one or more receiving MREs 130 for selecting a group of suitable relay compressed video streams to be relayed to the MRE 130. An RTP compressed video stream builder 450 can include an MRE multiplexer sequencer 452 and an MRE RTP video output buffer 454. Each RTP compressed video stream builder 450 may select one of a group of relay RTP compressed video data chunks or one of a plurality of sources (MRE 130) and one or more groups of one or more MRE sequential video memories 430. This selection may be based on control signals received from SCM 240 via control bus 465 and may be altered due to changes in the agenda. In an alternate embodiment, the RTP compressed video stream builder 450 can refer to itself via the SCM 240 and the control bus 465. The MRE 130 itself receives a control signal as to which MRE 130 the assigned MRE 130 will see.

An MRE video multiplexer sequencer 452 can obtain a selected stream of input relay RTP compressed video data chunks from the bus 440, and multiplex the stream group to one of the relay RTP compressed video data chunks. In streaming, the stream is stored in the MRE RTP video output FIFO 454 and transmitted from the MRE RTP video output FIFO 454 via the CRVDI 405 and NI 220 to the assigned receiving MRE 130. In some meeting agendas, an RTP compressed video stream builder 450 can be used for all MREs 130 of the agenda, so all MREs 130 will receive the same relay RTP compressed video data chunk stream.

An alternate embodiment of the RTP compressed video stream builder 450 (not shown in the figures) may include a group of selectors. Each selector is coupled to bus 440 and may select the output of one of a plurality of MRE sequential video memories 430. The other ports of the selector can be connected to the CRVDI 405 via the FIFO. In this embodiment, the selected video streams are transmitted toward the MRE as a plurality of relay RTP compressed video data thick block streams.

In yet another embodiment, the exemplary MRE sequential video memory 430 may not organize the received relay RTP compressed video data chunks based on the received serial number of the relay RTP compressed video data chunks. Rather, the received relay RTP compressed video data chunks are organized in accordance with the order in which the received relay RTP compressed video data chunks are received.

In an embodiment in which an ID number is assigned to each of the segments in a layout configuration, the MRE RTP video output FIFO module 454 can be adapted to add the appropriate segment ID to each processed compressed video data chunk. In this embodiment The association between a segment ID and one of the associated IDs of the source MRE 130 may be maintained via a control signal received by the bus bar 465. The segment ID may replace the source ID associated with the chunk or the segment ID may be stored in another field in the RTP header.

During a transition period, an SCM 240 may instruct the RTP compressed video stream builder 450 assigned to the demanding MRE 130 to obtain compressed video data chunks from the temporary MRE sequential video memory 430 assigned to the temporary stream, rather than from The MRE sequential video memory 430 assigned to the stream sent from the presentation MRE 130 obtains a compressed video data chunk. The RTP compressed video stream builder 450 assigned to the remainder of the plurality of receiving MREs can continue to retrieve the MRE sequential video memory 430 assigned to the stream transmitted from the rendering MRE.

FIG. 5 illustrates a simplified block diagram of a related component having an embodiment of an MRE 130. The MRE 130 can include an MRE network interface module (MRENI) 520, an MRE audio module (MREAM) 530, an MRE control module (MRECM) 540, and an MRE video module (MREVM) 550.

An MRE 130 can communicate with the MRM 120 via the MRENI 520 or with another MRE 130 via the network 110. An MRENI 520 can process the communication in accordance with one or more communication standards (such as H.323, SIP, or similar standards) and compression standards (such as H.264, MPEG, etc.). In addition, the network MRENI 520 can perform communication to and from the MRE 130 of the physical layer, the data link layer, the network layer, and the transport layer (UDP/TCP layer).

The MRENI 520 can multiplex and/or demultiplex the signals and controls and media streams that are communicated between the MRE 130 and the MRM 120. Can be transferred to and from MRM 120 and MREAM 530 and MREVM 550 via MRENI 520, respectively RTP compressed data chunks (header and payload) for audio and video. The MRENI 520 can also transmit and receive signaling and control between the MRECM 540 and the MRM 120.

A MREAM 530 can receive a group of a plurality of relay RTP compressed audio data chunks (headers and payloads) from the MRM 120 via the MRENI 520, and parse the RTP headers of the relay RTP compressed audio data chunks To determine parameters such as source ID, time stamp, and serial number. The MREAM 530 may also configure the received relay RTP compressed audio data chunks according to the ID, time stamp and/or serial number of the received relay RTP compressed audio data chunks, and then decode, mix and Zoom in on the thick chunks of the audio data. The MREAM 530 can transmit the mixed audio to one or more speakers of the MRE 130.

In other aspects, the MREAM 530 can collect audio signals from the microphones of the MRE 130 and convert the signals from analog signals into digital signals, calculate audio energy, and encode/compress the audio into RTP compressed audio data according to appropriate compression standards. Thick block. The compression standard used may include G.711, G.719 or G.722.1C.

The MREAM 530 may embed the calculated audio energy, the ID of the audio stream assigned to the MRE 130 by the MRM 120, the time stamp, and the serial number in the appropriate fields of the RTP header of the compressed audio data chunk. In another embodiment, the MREAM 530 can transmit an indication of the audio signal energy via an MRECM 540. More information about MREAM 530 is disclosed below in conjunction with FIG.

An MREVM 550 can be received from the MRM 120 via the MRENI 520 Following RTP compression of a group of video data chunks (header and payload), and parsing the RTP headers of the received relay RTP compressed video data chunks to determine such as source ID, fragment ID, time stamp And the parameters of the serial number. During a transition period, an ID number can be assigned to a temporary stream. The MREVM 550 may configure the received relay RTP compressed video data chunks according to the received time stamp and/or serial number of the relay RTP compressed video data chunks, and decode the relay RTP compressed video data. Thick blocks are organized into an appropriate segment FM (frame memory) based on the ID number. For each segment in the layout configuration, there may be a segment FM and each segment and/or source ID may be associated with a particular segment FM in the layout configuration. In an alternate embodiment, the source and/or destination IP address and port number may be associated with one of the segments in the layout configuration. Depending on the frame rate used by the MRE 130, the MREVM 550 can combine the different segments FM into a composite FM (CP FM) and transmit the complete CP FM to be displayed on one or more displays of the MRE 130.

In an alternative embodiment, the MREVM 550 can configure the received relay RTP compressed video data chunks according to the time stamp and/or serial number of the received relay RTP compressed video data chunks, and decode the video. The data is chunked and organized into a CP FM covering one of the entire layout configurations. In another embodiment, the MREVM 550 may also receive information from the MRECM 540 regarding changes to the primary speaker of the meeting, changes in the number of participants to be presented, changes to some of the participants, and the like.

In some embodiments, the decoder of the MREVM 550 that decodes the received relay RTP compressed video data chunks can be configured to be during the transition period Dispose of the decoding program. In an embodiment, the decoder can store and use a plurality of previous reference frames. In this embodiment, the decoder may send an Intra replacement request with one of the indications of the last referenced previous reference frame. In other embodiments, the decoder can process unique frames, such as SP frames and SSP frames. In still other embodiments, the decoder can be adapted to decode a lossless compressed frame to load the lossless compressed frame as a reference frame or the like for the next interframe frame.

In other aspects, the MREVM 550 can collect video images from the camera of the MRE 130, scale the video images to one or more desired sizes/qualityes, and encode/compress the video images into RTP according to appropriate compression standards. Compress video data chunks. The compression standard may include H.264, MPEG 4, and the like. Information about the desired size and compression criteria can be received from MRM 120 via an MRECM 540. The MREVM 550 can embed different parameters (such as source ID, timestamp, serial number, frame number, etc.) in the appropriate fields in the RTP header. Depending on the frame rate, the relay RTP compressed video data chunks may be transmitted to an MRM 120 via the MRENI 520.

In some embodiments, the encoder of the MREVM 550 that encodes/compresses the relay RTP compressed video data chunks can be configured to handle the encoding process during the transition period. In one embodiment, a decoder that receives the MRE 130 can store and use a plurality of previous reference frames. In this embodiment, the encoder can receive an in-frame replacement request from a demanding decoder that replaces the request with an indication of one of the previous reference frames that the decoder has last stored. The encoder can then be stored based on the indicated last The previous video image from the camera was previously compressed into an inter-frame frame in response to the frame. In other embodiments, the encoder can encode SP frames and SSP frames. In still other embodiments, the encoder can be adapted to encode its reference frame into a lossless compressed frame according to a lossless compression method, to be loaded as a reference frame for one of the next interframe frames. Requires the decoder of the MRE 130.

An MRE Control Module (MRECM) 540 can control the operation of the MRE 130. The MRECM 540 can be connected to the MRM 120 and communicate parameters regarding the number of participants, image size, compression criteria, primary speaker, ID information, etc., to be displayed in the MRE 130 in the layout configuration. The ID information may include ID information of different audio or video data chunks sent from the MRE 130, communication regarding a transition period, and the like. The information about the transition period may include starting and terminating one of the transition periods, one of the temporary streams, the signal and control regarding the transition period, and the like.

The MRECM 540 can allocate audio and video resources based on the number of selected participants in the agenda, the desired layout, the number of FMs required, and the like. The MRECM 540 can instruct the MREVM 550 how to construct a layout configuration to be displayed on one or more of the MREs 130. The MRECM 540 also enables the MREAM 530 to update the number of participants to participate in the conference, and the like. In some embodiments, an out-of-band connection between MRE 130 and MRM 120 that will cause MRE 130 and MRM 120 to communicate dynamic changes in the agenda can be established.

In some embodiments of the MRE 130, the MRECM 540 and MREVM 550 can be adapted to add information to the displayed CP image. The information can refer to Shows the current speaker and/or the name of one of the present participants in each of several segments. In this embodiment, the MRECM 540 can be adapted to receive the information from the MRM 120. This information can be passed to the MREVM 550, which contains a text and graphics generator for converting status information to be displayed at the endpoint. In other aspects, MREVM 550 and MRECM 540 can be adapted to display a menu from the endpoint, where the menu can be used to control MRM 120.

During the transition period, the MRECM 540 can be adapted to allocate video and network resources (respectively) in the MREVM 550 and the MRENI 520 for handling the temporary stream, in some embodiments, presenting the MRE 130 and the demand MRE 130 The MRECM 540 can be configured to manage the process of handling the transition period. The MRM 120 may initiate the transition period but after the transition period is initiated, the MRE 130 and/or the MRECM 540 presenting the MRE 130 may manage the continuation and termination of the transition period. More information on MRE 130 is disclosed below in conjunction with Figures 6, 7, 9A, and 9B.

6 is a simplified block diagram of one of the components of a portion of MREVM 550 in accordance with an embodiment. The MREVM 550 can have two main segments: (i) an input segment that can handle the group of received relay RTP compressed video data chunks; and (ii) an output segment that can be disposed of by one Video data captured by the MRE 130 camera. For example, an input section according to an embodiment may include the following modules, such as an RTP video input buffer 610, a video RTP header parser and organizer 620, and one or more compressed video segment memories (CVSM). 630 (each segment in the layout has a compressed video segment memory (CVSM) 630), one or more MRE video Decoder 640 (each segment in the layout has an MRE video decoder 640), one or more segments FM 650 (each segment in the layout has a segment FM 650), and an MRE CP image builder 660 , an MRE CP frame memory module 670 and a background FM 655. For example, the output section can include one or more ratio adjusters and FM 680, one or more video encoders 685, a temporary encoder 687, and an MRE video RTP processor (MREVRTP) 690. Encoder 685 and decoder 640 can use compression standards such as H.264 AVC, H264 annex G, MPEG-4, and the like.

Each CVSM 630 is associated with one of the video streams to be presented in the segment. In an embodiment, the association with one of the ID CVSMs may be changed during an agenda. In other embodiments, the CVSM is tied to the ID for a whole agenda.

One of the input sections of the MREVM 550, the RTP video input buffer 610, can obtain a relay RTP compressed video data chunk from the MRENI 520. The video RTP header parser and organizer 620 can access the input buffer 610 and parse the RTP header for determining different parameters of the received relay RTP compressed video material. Such parameters may include, but are not limited to, a serial number, a frame number, a source ID, and/or a segment ID, a time stamp, and the like. For example, the RTP header parser and organizer 620 can also have an index table that associates the source ID with the segment in the displayed layout configuration. In one embodiment where the segment ID is not associated with a received data chunk, for example, each CVSM 630 can be associated with a particular segment of the displayed layout configuration. Thus, for example, the RTP header parser and organizer 620 can stream a suitable relay RTP compressed video data chunk to a CVSM 630 based on the source ID. RTP header parser The organizer 620 can also organize the RTP compressed video data chunks in the CVSM 630 based on the serial number or time stamp or the frame number and the address of the first MB of the received data chunk.

The output of each CVSM 630 can be associated with an MRE video decoder 640, and the output of each MRE video decoder 640 can be associated with a segment FM 650. Thus, the MRE video decoder 640 can access the appropriate CVSM 630, decode the RTP compressed video data chunks, and store the decoded video in the appropriate segment FM 650. In an alternate embodiment, a scaler can be added between a decoder 640 and a segment FM 650. The MRE CP image builder 660 can transfer the contents of the different segments FM 650 to the MRE CP frame memory module 670 to construct a CP image. A complete frame of one of the CP images to be displayed on the MRE display unit can be transmitted from the MRE CP frame memory module 670. Sometimes, the background FM 655 can be loaded based on different parameters such as background color, boundary lines between different segments, colors of border lines, patterns, and names of attendees. The background FM 655 can be generated at the beginning of an agenda, but the background FM 655 can be changed at any time during the agenda. In one embodiment of an MRE 130, the background can be generated by the MRECM 540. When constructing a CP image, the MRE CP Image Builder 660 can collect data from the background FM 650 module as if it were collecting data from the segment FM 650.

During a transition period, the input section of one of the MREVMs 550, one of the demand MREs 130, can function differently than in normal operation. During the transition period, the RTP video input buffer 610 can also be configured to obtain a relay RTP compressed video data chunk from the MRENI 520 regarding the temporary stream received by the autocorrelation presentation MRE 130. Video RTP header parser and organizer 620 The relay RTP compressed video data chunks that can be configured to obtain the temporary stream are organized in the CP image to be displayed on the display of the demand MRE 130 in the CVSM 630 associated with the segment assigned to the presentation MRE 130. The relay RTP compresses the video data chunks.

In an embodiment, the MRE video decoder 640 associated with the segment assigned to the presentation MRE 130 can be configured to use a plurality of stored previous reference frames. In this embodiment, the decoder can be configured to send an in-frame frame in place of the serial number of the previous reference frame requesting the same store. In response, the associated encoder presenting the MRE 130 can send an inter-frame frame compressed based on the previous frame of the indicated storage. The interframe can be sent on a normal stream.

In another embodiment in which an SP frame is used on a normal stream and an SSP frame is used on a temporary stream, the decoder of the associated stream 640 can be configured to handle the SP frame and/or the SSP. A frame is used to receive an MRE 130 or a demand MRE 130, respectively.

Also, in an alternate embodiment in which a reference frame is transmitted over the temporary stream using a lossless compression algorithm by the encoder presenting the MRE 130, the correlation decoder 640 may be capable of decoding the reference frame for encoding. The lossless compression algorithm loads the lossless compression algorithm into its reference frame to be able to decode the next frame of the normal stream to be encoded based on the reference frame.

The management transition period can be accomplished by the associated decoder 640 and the encoder presenting the MRE 130. In other embodiments, however, the management can be handled by the SCM 240. In other embodiments, the transition week can be initiated by the SCM 240. The transition period of the decoder/encoder is initiated at the beginning of time, and then the decoder 640 and the encoder presenting the MRE 130 spontaneously continue the transition period activity.

The output section of the MREVM 550 receives video images from the MRE 130 camera and stores the images in one or more scalers and FM 680. For example, each of the scalers and the FM 680 can scale the video image to a different size (resolution) and store the video image. The scale adjusters and the output sections of the FM 680 are associated with a video encoder 685, which can cause one of the compressed video data chunks to encode data at a different rate and/or quality. The video encoder 685 then transmits the compressed video data chunks to an MREVRTP 690, which may embed a source ID, time stamp, serial number or other parameter in the header of the RTP compressed video data chunk. Next, the MREVRTP 690 outputs the relay RTP compressed video data chunks to the MRENI 520.

During a transition period, an output section of one of the MREs 550 that presents one of the MREs 130 can function differently in normal operation. During this transition period, one embodiment of video encoder 685 associated with a scaler that is scaled to the size associated with demand MRE 130 can be configured to use a plurality of stored previously used reference frames . In this embodiment, when the encoder receives an in-frame frame from a demand MRE 130 with a sequence number of a stored previous reference frame, the encoder can be referenced based on the previous use. The frame responds by encoding the received camera frame from the associated scaler and FM 680 as an inter-frame. Sending on normal streaming via MREVRTP 690 based on this first The frame between the frames in the previous frame is compressed. After sending the interframe, the encoder can delete all other previously stored reference frames.

In another embodiment in which an SP frame is used on a normal stream and an SSP frame is used on the temporary stream, a temporary encoder 687 can be allocated for generating the temporary stream. In addition, MREVRTP 690 can be adapted to handle the compressed chunks of the temporary stream. The temporary encoder 687 can obtain the same scaled frame of the encoder 685 assigned to the normal stream from the correlation scaler and the FM 680. The first compressed frame of the temporary stream can be encoded into an in-frame frame of the same size but lower than the quality of the normal stream. The quality of the temporary stream can be improved from the frame to the other frame. After some inter-frames (eg, 3 frames to 15 frames), the temporary encoder 687 can transmit an SSP frame while the normal encoder 685 can transmit an SP frame. The next camera frame will be encoded by the associated video encoder 685 as a normal frame and sent to the demand MRE 130 on a normal stream as sent to all other receiving MREs 130.

In an alternate embodiment in which a reference frame is transmitted over a temporary stream using a lossless compression algorithm by an encoder 685 that presents the MRE 130, the temporary encoder 687 can be derived from the encoder 685 of the normal stream. Refer to the frame. The obtained reference frame can be compressed by the temporary encoder 687 using a lossless compression algorithm. The lossless compression frame may be sent towards the demand MRE 130 via a MREVRTP 690 on a temporary stream or over an out-of-band IP connection. In one embodiment using an out-of-band connection, the lossless compression reference frame can be transmitted directly from the temporary encoder 687 to the MRENI 520. A lossless algorithm can be ZIP, Lempel-Ziv-Welch (LZW), lossless JPEG 2000 and so on.

7 is a simplified block diagram of one of the related elements of an embodiment of MREAM 530. For example, the MREAM 530 can have two segments: (i) a MREAM input segment that can handle the group of received input relay RTP compressed audio data chunks of the MREAM 530; and (ii) a MREAM output segment, It can handle audio data that will be output from MRE 130 towards MRM 120. A MREAM input section can include the following modules, such as an RTP audio input buffer 710, an audio RTP header parser and organizer 720, one or more MRE sequential audio memories (MRESAM) 730, one or more MRE audio decoder 740 and an audio mixer 750. For example, the MREAM output section can include an encoder 760 and an MRE audio RTP processor 770. In some embodiments in which the two receiving MREs 130 use different audio compression standards, for example, the output section may include two or more sets of one encoder 760 and one MRE audio RTP processor 770, each group being The compressed audio stream is associated based on one of the compression standards different from the other groups.

An RTP audio input buffer 710 can obtain a group of relay RTP compressed audio data chunks from the MRM 120 via the MRENI 520. The audio RTP header parser and organizer 720 can access the input buffer 710 and parse the RTP header for determining parameters such as sequence number, source ID, timestamp, and the like. For example, each MRESAM 730 can be associated with participating in an agenda and being selected to learn about a particular source MRE 130. For example, the RTP header parser and organizer 720 can properly stream one of the relay RTP compressed audio data chunks to the MRESAM 730 based on the data chunk ID. In an alternate embodiment, for example, the RTP header parser and organizer 720 can be based on the source IP address and One of the relay RTP compressed audio data chunks is properly streamed to the MRESAM 730. The RTP header parser and organizer 720 can also organize the chunks of data based on the serial number or time stamp of the RTP compressed audio data chunks of each of the plurality of MRESAM 730. Each MRESAM 730 output is associated with an MRE audio decoder 740 that accesses MRESAM 730 and decodes the RTP compressed audio data chunks. The compression standard used by the decoder may be, for example, but not limited to, G.719, G.7221.C, and the like. The audio mixer 750 can receive the outputs of all of the MRE audio decoders 740, mix the outputs, and output the mixed audio to the speakers of the MRE 130.

One of the output sections of the MREAM 530 can receive audio from the microphone of the MRE 130. The encoder 760 can collect the received input audio, determine the input audio energy, and encode the input audio into a compressed audio data chunk. This compression can be based on compression standards such as G.719, G.7221.C, and the like. The compressed audio data chunks are then transferred to the MRE audio RTP processor 770. The indication of the audio energy can also be transmitted to the MRE audio RTP processor 770, which can embed different parameters in the header of the RTP audio data chunk. The parameters may include a serial number, a source ID, a time stamp, an audio energy indication, and the like. Next, the MRE audio RTP processor 770 outputs the relay RTP compressed audio data chunks to the MRENI 520 and outputs the MRENI 520 to the MRM 120. In an alternate embodiment, the audio energy indication can be stored in the header of the compressed audio data chunk. In yet another embodiment, the audio energy can be transmitted toward the MRM 120 over a signaling and control connection via the MRECM 540 and the MRENI 520.

FIG. 8A is a flow diagram illustrating one of the procedures associated with one of the conference setup methods 800 implemented by one embodiment of an SCM 240. In act 810, method 800 is initiated by the MRM control module conference setup program. At act 815, meeting parameters are collected, such as the number of participants (MRE 130), bit rate, audio and video compression standards, and the number of previous reference frames available for video encoding/decoding, CP layout, and the like. Next, at act 820, method 800 allocates various conferencing resources, such as NI 220 source, an associated SCARP 230, an associated SCVRP 250, bandwidth, and the like. Next, a loop between action 830 and action 840 is performed for each MRE 130 participating in the conference. At act 832, one or more possible output image sizes (quality) from the MRE 130 and associated IDs of the MRE 130 are defined. In an embodiment, method 800 can use the IP address and 埠 of each stream from MRE 130 as the ID. Act 836 associates MRE 130 with internal modules of MRM 120, such as SCARP 230 and SCVRP 250.

At act 836, the MRE 130 is linked to one of the plurality of MRE sequential audio memories 330 in the SCARP 230 of the MRM 120, linked to one of the plurality of RTP compressed audio stream builders 350, linked to the MRM The plurality of MREs in the SCVRP 250 are one of the sequential video memories 430, and are linked to one of the plurality of RTP compressed video stream builders 450, and the like. Next, various connection parameters (external and internal) of the resources associated with the associated MRE 130 are set in act 838. The external connection parameters may include one of the MRE 130 ID, a remote IP address and port, and a local IP address and port. The internal connection parameters may include internal connections of modules in the MRM 120 itself, such as internal connections of modules within the SCARP 230 and SCVRP 250.

Next, a determination is made in act 840 as to whether there are more MREs 130 to be processed. If so, the method 800 returns to act 830 to manage the settings of the next MRE 130. If no, method 800 proceeds to act 845, which involves defining a starting CP layout for each MRE 130 in the conference. The method 800 proceeds to act 850 where another loop is initiated for each MRE 130 in the conference. At act 851, method 800 can cause the associated MRE 130 to transmit one or more image sizes, the MRE 130 needs to transmit one or more compressed audio streams, and the MRE 130 will need to add it to the audio of each stream and One or more IDs or the like in the video RTP header are loaded to the MRE 130. At act 852, the parameters assigned to the CP layout of the MRE 130 can be calculated and loaded into the MREVM 550. The parameters may include the number of segments in the layout configuration, the ID or IP address associated with each segment and/or the MRE 130 to be presented, the current speaker, and the like. In other embodiments, each MRE 130 may define its own layout configuration to be displayed on its own display unit and notify the MRM 120 of such parameters.

At act 852, the MREVM 550 of the MRE 130 may also be instructed to define parameters of the segment frame memory 650 and the CP frame memory 670. The internal modules of the MREAM 530 can be set up in a similar manner, such as, for example, an audio RTP header parser and organizer module 720 and an audio mixer 750. One result of this action may be to map different modules of the MRE 130 to one of the associated stream IDs.

Next, method 800 sets the relevant parameters in the internal modules of MRM 120, such as SCARP 230 and SCVRP 250. At act 854, different parameters and connections of the internal modules of the SCVRP 250 are set. Parameters and connections The MRE video memory 430 is associated with one of the MRE 130 ID addresses and/or an IP address and port and the MRE video multiplexer sequencer 452 for selecting the video of the MRE 130. At act 856, different parameters and connections of the internal modules of SCARP 230 are set. The parameters and connections may include the association of the MRE sequential audio memory with one of the MRE 130 ID addresses and/or an IP address and port and the associated MRE multiplexer sequencer 352 for selecting the MRE 130 audio.

At act 858, the MRM 120 requests an in-frame from one of the associated MRMs 130 and proceeds to act 860. At act 860, a determination is made as to whether more MREs 130 need to be set. If yes, method 800 returns to act 850. If no, method 800 ends. One method similar to method 800 can be performed whenever a primary speaker changes or joins a new attendee or a new attendee leaves the agenda.

FIG. 8B is a flow diagram illustrating one of the related actions of transitioning task 8000 by one of SRMs 240 of one of MRMs 120, in accordance with an embodiment. Task 8000 may be initiated in act 8010 when a demanding MRE 130 requires an in-frame frame from one of the present MREs 130 while the other received MREs receiving the same video image on the normal stream does not require an in-frame. This situation may occur when the demand MRE 130 is intended to join a new participant in an ongoing agenda; when one of the layout configurations used by the demand MRE 130 occurs (not in the other receiving MRE 130) When the change occurs); or when one or more packets transmitted to the demand MRE 130 are lost, and the like.

After the start 8010, in block 820, the SCM 240 can verify the requirements. Whether the MRE 130 has a reference frame for use in relation to the pre-light from the associated stream (in relational size) that presents the MRE 130. This verification can be accomplished by signaling the incoming MRE 130 associated with the indication in the box. Alternatively, SCM 240 may manage a history table that stores a sequence number of some frames (eg, the last 16 frames) of each of the newly transmitted MREs 130 to a number of MREs 130 on a normal stream. Based on the request and the timing of the history table, in act 820, SCM 240 can infer whether a previous reference frame exists in the decoder of demand MRE 130. If the SCM 240 changes the layout configuration of the demand MRE 130, the SCM 240 can determine whether the previous reference frame matches the new layout configuration, and the like.

If there is a previously used reference frame in the decoder of the demand MRE 130 in act 8020, then in act 8026, the SCM 240 may indicate that the associated stream at the MRE 130 is present based on the existing previous test frame. The encoder encodes the next frame as an interframe and sends the interframe to the demand MRE 130 and the remainder of the plurality of receiving MREs 130 on a normal stream. In addition, the encoder can be instructed to reset all other previous reference frames. The SCM 240 can reset the history table of the associated stream in parallel. Method 8000 can then terminate. In other embodiments, the in-frame frame received from the demand MRE 130 replaces the request to an encoder that presents the associated stream in the MRE 130, the encoder being configured to rely on the reference based on the previous usage. The frame responds by encoding the next frame as an interframe.

If there is no previously used reference frame in the decoder of the demand MRE 130 in act 8020, then in action 8022 the SCM 240 may allocate one of the communication channels for temporary streaming and the SCVRP 250 and the present MRE 130. Additional resources in the output section of MREVM 550, and the input section of MREVM 550 of demand MRE 130 is adjusted to handle the temporary stream. The allocated resources in SCVRP 250 may include additional MRE sequential video memory 430 and additional RTP compressed video stream builder 450. The allocated resources in the MREVM 550 that present the MRE 130 may include a temporary encoder 687.

After allocating resources and bandwidth, in act 8022, the presentation MRE 130 can be instructed to begin generating a temporary stream in parallel with a normal stream of the same size (resolution) according to a defined sequence. The definition sequence of the temporary stream can start with an intra-frame with low quality but with the same resolution, and then achieve a similar quality in the two streams (the normal stream and the temporary stream) at most. A few of the enhanced interframe frames. At the end of the sequence, each of the plurality of encoders presenting the MRE 130 encodes and transmits a unique frame. The normal stream encoder 685 encodes an SP frame and transmits the SP frame towards the other receiving MREs on the normal stream, while the temporary encoder 687 can encode an SSP frame and transmit the SSP frame on the temporary stream. SSP frame.

After instructing the presentation MRE 130 to begin transmitting the temporary stream, the task 8000 may wait in action 8024 to receive a unique frame (SP and SSP) from the presentation MRE 130. When the unique frames are received in action 8030, in action 8032, the SP frame and the SSP frame are relayed to the other receiving MRE 130 and the demand MRE 130, respectively. The temporary stream can then be terminated, and the MRM 120, the MRE 130, and the allocated resources in the demand MRE can be released in action 8032, and the task 8000 can be terminated.

9A is a flow chart illustrating one of the related actions of an MRE conference setup method 900 implemented by one embodiment of an MRECM 540. Can move The initial method 900 is performed at 910. At act 912, the build is connected to one of the MRMs 120 and the MRECM 540 can receive the setup instructions. The setup instructions can include: one or more IDs to be associated with the media stream sent from the MRE 130; the size of the image that the MRE 130 will need to transmit; the number of participants to be learned and displayed on the MER 130 ; and layout information. Method 900 can then allocate resources in act 914 accordingly, including: MRENI 520, MREVM 550, resources in MREAM 530; bandwidth, and the like.

Next, method 900 proceeds to act 920 where different internal modules of MREVM 550 are set. The settings may include: instructing the video RTP header parser and organizer 620 based on the ID received at action 912; establishing which of the CVSMs 630 the relay compressed video RTP data chunks are to be stored in; according to the layout of the agenda Setting parameters of each segment FM 650 and MRE CP FM module 670; indicating how the CP builder 660 constructs the CP from the segment FM 650; updating the background FM 655; setting the proportional adjuster and FM 680 to deliver the correct size; indicating MREVRTP 690 adds the ID to the RTP header. In act 920, the CP Builder 660 is instructed how to implement the CP image. The indication may define an MBA (macroblock address) of the first and last MBs in each column of the CP image, a first MBA of each segment in the CP image, and a last MBA of the segment, etc. .

In some embodiments in which MRECM 540 and MREVM 550 are adapted to add information to the displayed MRE 130 of the CP image, act 920 can be adapted to deliver the information. The information may indicate the current speaker, the name of the participant present in each of the several segments, and the like. In such embodiments, the CP Builder 660 can be adapted to convert the information into video data and Add it to the CP image to be displayed on the endpoint.

Next, at action 922, different internal modules of the MREAM 530 are set. The setting may include: setting an audio encoder 760 according to a compression standard; instructing the MRE audio RTP to add an ID and an audio energy indication to the RTP header; indicating which ID the audio RTP parser 720 stores in which MRESAM 730; setting the MRE audio Decoder 740, etc. After act 922, the method is complete and the associated MRE 130 is ready to send an in-frame. One method similar to method 900 can be performed whenever a primary speaker changes or a new attendee joins or leaves.

FIG. 9B is a flow chart illustrating one of the related actions of transitioning task 9000 by one of MREs 540, one of MREs 130. The illustrated method 9000 includes actions to be handled by a presentation MRE 130 and actions to be handled by a demand MRE 130. When a demanding MRE 130 requires an in-frame frame from one of the present MREs 130 while receiving the same quality/size video image on a normal stream without the need for an in-frame, it may be in action 9010. Start task 9000. This may occur when the demand MRE is for a new participant wishing to join an ongoing agenda; or when the demand MRE 130 requires one of the layout changes, while the other receiving MRE 130 does not require the change; Or when the demand MRE 130 wishes to switch from one quality/size stream to another quality/size stream, and the like. In addition, when one or more packets in the stream transmitted to the demand MRE 130 are lost, etc., the demand MRE may require an in-frame and the other receiving MREs 130 do not need the in-frame.

Compressed stream that may be received for decoding from the presentation MRE 130 The required MRE decoder 640 (Fig. 6) completes the beginning 9010 of task 9000. Task 9000 may be initiated when the decoder requires an in-frame after receiving and decoding a plurality of previous inter-frames from the presentation MRE 130. For example, the need for a frame within this box can be attributed to the loss of the packet. Alternatively, the SCM 240 of the MRM 120 may initiate the task when determining an intra-frame of a particular stream, and the normal stream sent from the presentation MRE 130 needs to be sent to the demand MRE 130.

Method 9000 can be implemented in an embodiment where the encoder/decoder is configured to store a plurality of previous reference frames and replace the request (rather than an in-frame request) with an in-frame. The in-frame frame replaces the request for one of the inter-frames based on one of the previous reference frames. A request indicating one of a number of stored previous reference frames is sent from the demand MRE 130 to the presentation MRE 130. In this embodiment, task 9000 may be initiated by demand MRE decoder 640 in act 9010, which determines (9012) which of the plurality of previous reference frames owned by the decoder is available for decoding from A received code frame below the MRE 130 is presented. The decision can be based on the timestamps of the reference frames. Next, in act 9110, an in-frame replacement request along with a sequence number or timestamp of the selected previous reference frame may be sent from the decoder of the demand MRE 130 to the SCM 240 of the MRM 120 via the MRECM 540.

In act 9110, the SCM 240 of the MRM 120 then transmits the in-frame replacement request along with the serial number or timestamp of the previous reference frame to the MRECM 540 presenting the MRE 130. At the MRECM 540 presenting the MRE 130, the in-frame replacement request is obtained from the SCM 240 in act 9110. The in-frame replacement request is transmitted to the video encoder 685 presenting the MRE 130, which is assigned to transmit the associated stream (quality/size) to the demand MRE 130, in act 9130. In act 9130, the instructing video encoder 685 compresses a scaled frame received from the camera via the associated scaler 680 into an interframe frame based on the previous reference frame, and in normal streaming The inter-frame is sent as the next inter-frame to all receiving MREs 130 (including the required MRE 1300. In addition, the instructing encoder 685 releases all other previous reference frames, and the method 9000 terminates. The SCM in the MRM 120 240 changing the sequence number of the relay frame received from the presentation MRE 130 and transmitting it to the receiving MRE 130 as a relay frame, the SCM 240 can convert the serial number received from the demand MRE into The appropriate serial number that has been sent from the presentation MRE 130 to the MRM 120.

If it is determined in act 9012 that a previous reference frame is not available, task 9000 may continue to action 9210 and communicate with SCM 240 of MRM 120. In communication with the SCM 240, the MRECM 540 presenting the MRE 130 can be instructed to open a new connection for a temporary stream parallel to the normal stream, an ID can be assigned to the temporary stream, the encoded image of the temporary stream The size (resolution) is defined as the same as the normal stream, and the number of quality-enhanced frames in the temporary stream can be defined.

If the temporary stream is a new stream sent from the presentation MRE 130, the MRECM 540 requiring the MRE 130 in block 9210 may obtain one or more IDs of the new relay stream from the SCM 240 and about the stream. Video parameters, such as the size of the video image (resolution), associated segments in the CP layout Location, etc., as disclosed above in the description of Figures 9A and 8A. If the temporary stream is not a new stream sent from the presentation MRE 130, the MRECM 540 of the demand MRE 130 may only be notified that the next frame of the stream received from the presentation MRE 130 belongs to the temporary stream.

After obtaining information about the transition period from the MCM 540 of the SCM 240, the presence MRE 130, and the demand MRE 130 in act 9210, the method 9000 can begin assigning the desired resources in action 9230 to handle the transition period in the two MREs 130. . The resources may be communication and bandwidth resources in the MRENI 520, internal communication resources between the MRENI 520 and the MREVM 550, and external communication resources between the MRENI 520 and the NI 220 in the MRM 120. Additional allocated resources may be video resources in the MREVM 550, and the like.

At act 9240, the allocated resources are organized to handle the temporary stream. At the MREVM 550 presenting the MRE 130, the input of the temporary encoder 687 is associated with the encoder of the normal stream. In an embodiment, for example, the correlation is used to obtain the same scaled video image from the associated scaler 680 and the appropriate quantization parameters for obtaining the regular interframe and the SP frame and SSP frame. set. In other embodiments, the switch contacts the reference frame of the encoder used to obtain the normal stream and uses it as one of the video images encoded by the temporary encoder 687. In addition, the output of the temporary encoder 687 presenting the MRE 130 can be associated with the MREVRTP 690 to be associated with the appropriate ID number and a timestamp.

In an embodiment where an SP frame is used on a normal stream and an SSP frame is used on the temporary stream, the video decoder 640 requiring the MRE 130 will receive low One of the quality compresses the in-frame frame, then receives a few enhanced quality inter-frames, and then receives the SSP frame. This sequence complies with the compression standard. The frame following the SSP frame will be the interframe frame from the normal stream.

In addition, if the temporary stream is a new stream sent from the presentation MRE 130 via the MRM 120, the MRECM 540 of the demand MRE 130 can be organized in action 9240 to handle the new stream, and the RTP video input buffer 610 And video RTP parser 620 can be notified of the ID of the new stream and associated with a CVSM 630. The MRE video encoder 640 can be associated with the CVSM, and the like.

Also, in another embodiment of compressing (with lossless compression) and transmitting a reference frame of a normal encoder (except for related activities of some embodiments as described above) on the temporary stream, the correlation of the MRE 130 is required. The MRE video decoder 640 is notified in action 9240 when to change the compression criteria from the normal compression to the only compression (lossless compression) used to compress the reference frame, and vice versa.

After organizing the MRE 130 and the MREVM 550 of the demand MRE 130 in action 9240, the transition period can be further autonomously performed by the MRE VM 550 presenting the MRE 130 and the demand MRE 130 and the SCVRP 250 of the MRM 120. Therefore, method 9000 can be terminated.

10A and 10B illustrate a unique diagram of a video stream generated from MRE 1 (presentation MRE 130) and transmitted to MRE 2 and MRE 3 (received MRE 130 and demand MRE 130, respectively) via an MRM 120. A timing diagram of one of the transition periods in one embodiment of the block. 10A and 10B are for illustrative purposes only and are not drawn to scale.

Figure 10A has two sections, and the upper section illustrates the presentation of the MRE The normal stream of 130 (MRE 1) and the lower section illustrate a temporary stream with respect to presenting MRE 130 (MRE 1). The two segments are illustrated by five columns and a time axis suitable for one of the two segments. The first column illustrates one of the scaled camera video images (CF1 to CFn+1) received from the video camera of the MRE 1 after being scaled by the ratio adjuster and the FM 680. The second column illustrates one of the streams of coded video frames (IF1 to EFn+1) at the output of the normal encoder of MRE 1. This stream of coded frames is transmitted as a normal stream to the MRM 120, one of the management MRC agendas. The third column illustrates one of the reference frames (RFm-1 to RFn) stored in the encoder of the MRE 1 due to the associated scaled video frame received from the camera. The third column is the last column of the upper section.

The lower section of Figure 10A illustrates the temporary stream beginning in the fourth column, and the fourth column illustrates one of the encoded video frames (TIm to Tm+2 and SSTn) at the output of one of the temporary encoders of MRE 1. Streaming. This stream of coded frames is transmitted towards the MRM 120 as a temporary stream. The last column illustrates one of the reference frames (RTm to RTm+2 and RTFn) stored in the temporary encoder 687 of the MRE 1.

In this example, as illustrated by the arrows between the third column and the fourth column, the temporary stream can be generated by encoding the reference frame of the normal encoder 685. The first frame (TIm) is encoded as the same size (resolution) but one of the lower quality frames. The next two frames are the enhanced interframes. Each frame improves quality and reduces the difference between the reference frame stored in the encoder of the normal stream and the associated reference frame stored in the temporary encoder 687. In other embodiments, the number of enhanced interframes can be two Any number outside the frame, such as for example, any number between a frame and 15 frames. The last frame SSTn of the temporary stream is a unique frame in which the reference frame RTFn has the same value as the reference frame RFn of the normal encoder to synchronize the reference frame RTFn with the reference frame RFn. In an embodiment, lossless compression may be used to compress the difference between the RFn at the input of the temporary encoder and the reference frame RTm+2 stored in the temporary encoder 687 to generate the SSTn frame, and thus the RTFn The value of each pixel in the same is the same as the value of the corresponding pixel in the RFn.

After scaling, the video frame CF1 is received from the MRE 1 camera and the normal stream from MRE 1 is initiated at T1. The normal stream encoder compresses the video frame CF1 into an in-frame frame IF1. The next scaled camera frame is encoded as an interframe frame. The compression standard can be H.264 AVC, H.264 annex G, MPEG-4, and the like.

At T2, MRE 3 requests to join the receiving MRE of the normal stream. Therefore, at T2, a request from the MRE 3 to send an in-frame request from one of the frames in the MRE 1 is requested. The presentation MRE 130 may initiate one of the temporary streams illustrated in the lower section of Figure 10A, rather than by transmitting an in-frame to the plurality of receiving MREs 130 (MRE 1) of the normal stream. The owner responds to the request.

The first frame CFm from the camera is compressed after the in-frame request is compressed based on the reference frame generated when the previous frame RFm-1 is compressed. In the embodiment of FIG. 10A, the reference frame RFm generated by compressing the CFm on the normal stream into an inter-frame frame TIm is compressed into one of the in-frame frames TIm of the temporary stream. The frame TIm is sent as the first frame on the temporary stream. Heading towards MRE 3. The compression of the temporary stream is accomplished by using the same compression standard used in the encoder of the normal stream, but the quality of the in-frame may be lower than the quality of the normal stream. This quality can vary in bit rate, sharpness, and the like.

The next few reference frames RFm+1, RFm+2 of the normal stream encoder are encoded by the temporary encoder as an enhanced quality interframe frame to generate the next interframe frame Tm+1 and Tm+. 2. For example, the number of enhanced quality in-frames can be any number between 1 and 15. The number of enhanced quality frames can be defined during the establishment of a temporary stream.

At T3, when it is assumed that the quality of both the normal stream and the temporary stream is similar, the next scaled camera frame CFn is compressed into an interframe frame EFn. The reference frame RFn generated when the CFn is compressed may be compressed by the temporary encoder 687 with lossless compression to generate a unique frame SSTn. For example, by using a fact that the difference between two consecutive frames in a video conference is small compared to a normal video image, the lossless compression can be performed on the difference between the RFn and the reference frame RTm+2 of the temporary stream. . Examples of lossless compression may be ZIP, Lempel-Ziv-Welch (LZW), Lossless JPEG 2000, and the like. Therefore, the reference frame RTFn generated at the temporary encoder 687 has the same value as RFn, so that the reference frame RTFn is synchronized with the reference frame RFn. Furthermore, decoding the unique frame at the decoder of MRE 3 will result in a reference frame with the same value of RFn and RTFn. Thus, as illustrated in FIG. 10B, at this point in time, the temporary stream can be terminated and the normal stream can be switched by MRM 120 to demand MRE 3.

At the demand MRE 3, the inverse of the lossless coding can be used to complete the coding difference. decoding. The result will be added to the value of the previous decoder reference frame similar to RTm+2. The result of adding the decoding differences can be stored in the decoder of the MRE 3 as a reference frame RTFn and can be displayed on the MRE 3. At T4, the temporary stream can be terminated.

Looking at the other side of the MRM 120 at Figure 10B, at T1, the first frame (in-frame IF1) presenting the normal stream of MRE 1 is sent towards MRE 2, followed by a plurality of inter-frames. The normal stream can be relayed to a plurality of receiving MREs 130 (not shown) that are equivalent to MRE 2. At T2, MRE 3 requests to receive the normal stream and request an in-frame. In response, the MRM 120 begins to send the frame of the temporary stream towards the demand MRE 3. The first frame is the frame inside the frame TIm, followed by a few enhanced quality inter-frames (Tm+1 and Tm+2). Next, at T3, the unique frame SSTn is sent as the last frame of the temporary stream. At T4, the inter-frame frames Fn+1, Fn+2 and the next frame from the normal stream are also oriented toward MRE 3.

11A and 11B are timing diagrams illustrating one transition period in another embodiment of moving from one stream to another using an SP frame and an SSP frame. The two streams are generated from the same video camera of the same size (resolution). The video streams are generated by MRE 1 (presentation MRE 130) and are streamed via MRM 120 to MRE 2 and MRE 3 (receive MRE 130 and demand MRE 130, respectively). 11A and 11B are for illustrative purposes only and are not drawn to scale.

Similar to FIG. 10A, FIG. 11A also has two sections, the upper section illustrating normal streaming from MRE 1 and the lower section illustrating temporary streaming from MRE 1. By five columns and for one of the two sections The axis diagram illustrates the two sections. The first column illustrates one of the scaled camera video images (CF1 to CFn+1) received from the video camera of the MRE 1 after being scaled by the ratio adjuster and the FM 680. The second column illustrates one of the streams of coded video frames (IF1 to EFn+1, including SPn) at the output of the normal encoder of MRE 1. This stream of coded frames is transmitted as a normal stream to the MRM 120, one of the management MRC agendas. The third column illustrates one of the reference frames (RFm-1 to RFm+2 and RSPn) stored in the normal encoder 685 of the MRE 1 due to the associated scaled video frame received from the camera. flow. The third column is the last column of the upper section.

The lower portion of Figure 11A illustrates the temporary stream beginning in the fourth column, and the fourth column illustrates the encoded video frames (TIm to Tm+2 and SSPn) at the output of one of the MRE 1 temporary encoders 687. A stream. This stream of coded frames is transmitted towards the MRM 120 as a temporary stream. The last column illustrates one of the reference frames (RTm to RTm+2 and RSSPn) stored in the temporary encoder 687 of the MRE 1.

In this example, as illustrated by the arrows at the top of the compressed frames TIm through SSPn, the same scaled adjustments at the output of the proportional adjuster and FM 680 associated with the normal encoder 685 can be encoded. A temporary stream is generated by the video frame. Therefore, two encoders (normal encoder 685 and temporary encoder 687) encode the same input frame. The first frame (TIm) is encoded as the same size (resolution) but one of the lower quality frames. The next two frames are the enhanced interframes. Each frame improves the quality and reduces the scaled frame of the input of the temporary encoder 687 and stores it temporarily. The difference between the associated reference frames in encoder 687. In other embodiments, the number of enhanced inter-frames can be any number other than the two frames, such as any number between one frame and 15 frames, for example. The final frame of the temporary stream is a scaled camera frame CFn, which is compressed into a regular SSP frame and generated as an SSPn frame. The same scaled camera frame CFn is scaled by the normal encoder 685 into a conventional SP frame. The result of SP coding and SSP coding is that the reference frame RSPn of the normal stream and the reference frame RSSPn of the temporary stream have the same value, so that the reference frame RSPn is synchronized with the reference frame RSSPn. Since the reference frame of the decoder of the normal stream and the reference frame of the decoder of the temporary stream have the same value, the temporary stream can be terminated, and the MRM 120 can view the following figure of the normal stream. The box is relayed to the required MRE 3.

After scaling, the normalized stream from MRE 1 is initiated at T1 after receiving the scaled video frame CF1 from the camera of MRE 1. The normal stream encoder 685 compresses the video frame CF1 into an in-frame frame IF1. The next scaled camera frame is encoded as an interframe frame. The compression standard can be any standard capable of processing SP frames and SSP frames, such as H.264 AVC, H.264 annex G, MPEG-4, and the like.

At T2, MRE 3 requests to join the normal streamed receive MRE 130. Therefore, at T2, a request from the MRE 3 to send an in-frame request from one of the frames in the MRE 1 is requested. In an embodiment, the presentation MRE 130 may initiate one of the temporary streams as illustrated in the lower section of FIG. 11A, rather than by sending an in-frame to the plurality of receptions of the normal stream. The owner of the MRE 130 (MRE 1) responds to the request.

The first frame CFm from the camera is scaled after being scaled based on the reference frame generated when the previous frame RFm-1 is compressed. In the embodiment of Fig. 11A, the same scaled camera frame CFm is compressed into one of the in-frame frames TIm of the temporary stream. The in-frame frame TIm is transmitted toward the MRE 3 as the first frame on the temporary stream. The compression of the temporary stream is accomplished by using the same compression standard used in the encoder 685 of the normal stream, but the quality of the frame may be lower than the quality of the normal stream. This quality can vary in bit rate, sharpness, and the like.

The next few scaled reference frames CFm+1, CFm+2 are encoded by the temporary encoder 687 as an enhanced quality interframe to produce the next interframe frames Tm+1 and Tm+2. For example, the number of enhanced quality inter-frames can be any number between 1 and 15.

At T3, when it is assumed that the quality of both the normal stream and the temporary stream is similar, the next scaled camera frame CFn is compressed by the normal encoder into a regular SP frame SPn and sent on the normal stream. . The same scaled camera frame CFn is compressed by the temporary encoder 687 into a conventional SSP frame SSPn and transmitted as the last frame on the temporary stream. Therefore, the reference frame RSSPn generated at the temporary encoder has the same value as the reference frame RSPn of the normal encoder. Furthermore, decoding the SPn frame at the decoder of MRE 2 will result in a reference frame having the same value as RSPn. Furthermore, decoding the SSPn frame at the decoder of MRE 3 will result in a reference frame having the same value as RSSPn. Since RSSPn and RSPn have the same value, the encoder/decoder includes the same reference frame. So as in As illustrated in FIG. 11B, at this point in time, the temporary stream can be terminated at T4, and the normal stream can be switched by MRM 120 to demand MRE 3.

Looking at the output side of the MRM 120 in Figure 11B, at T1, the first frame (in-frame IF1) presenting the normal stream of MRE 1 is sent towards MRE 2, followed by a plurality of inter-frames. The normal stream can be relayed to a plurality of receiving MREs 130 (not shown) that are equivalent to MRE 2. At T2, MRE 3 requests to receive the normal stream and request an in-frame. In response, the MRM 120 begins to relay the frame of the temporary stream towards the demand MRE 3. The first frame is the frame TIm in the frame, followed by a few enhanced quality inter-frames Tm+1 and Tm+2. Next, at T3, the unique frame SSPn is sent as the last frame of the temporary stream. At T4, the inter-frame frames Fn+1, Fn+2 and the next frame from the normal stream are also oriented toward MRE 3.

The above description is intended to be illustrative and not limiting. For example, the embodiments described above can be used in combination with each other. Many other embodiments will be apparent to those of ordinary skill in the examination of the above description. Therefore, the scope of the invention should be determined by reference to the scope of the appended claims and the scope of the claims. In the scope of the accompanying claims, the terms "including" and "including" are used as the concise English equivalent of the terms "including" and "including".

1‧‧‧Media Relay Endpoint (MRE)

2‧‧‧Media Relay Endpoint (MRE)

3‧‧‧Media Relay Endpoint (MRE)

100‧‧‧Multimedia conference system

110‧‧‧Network

120‧‧‧Media Relay Multipoint Control Unit (MRM)

130‧‧‧Media Relay Endpoints

220‧‧‧Network Interface Module

230‧‧‧ Agenda Compressed Audio Instant Protocol (RTP) Processor

240‧‧‧Signal and control module

250‧‧‧ agenda compressed video instant protocol (RTP) processor

305‧‧‧Compressed Real-Time Agreement (RTP) Audio Data Interface

310‧‧‧Instant Agreement (RTP) Audio Input Buffer

320‧‧‧Instant Messaging (RTP) Header Profiler and Organizer

330‧‧‧Media Relay Endpoint (MRE) Sequential Audio Memory

340‧‧‧ busbar

350‧‧ Real-time agreement (RTP) compressed audio stream generator

352‧‧‧Media Relay Endpoint (MRE) multiplexer sequencer

354‧‧‧Media Relay Endpoint (MRE) Instant Protocol (RTP) Audio Output Buffer

360‧‧‧Optical Energy Processor

365‧‧‧Control bus

405‧‧‧Compressed Real-Time Agreement (RTP) video data interface

410‧‧‧ Instant Protocol (RTP) Video Input Buffer

420‧‧• Video Instant Protocol (RTP) Header Profiler and Organizer

430‧‧‧Media Relay Endpoint (MRE) sequential video memory

440‧‧ ‧ busbar

450‧‧‧ Instant Protocol (RTP) compressed video stream generator

452‧‧‧Media Relay Endpoint (MRE) multiplexer sequencer/video media relay endpoint (MRE) multiplexer sequencer

454‧‧‧Media Relay Endpoint (MRE) Instant Protocol (RTP) Video Output Buffer

465‧‧‧Control bus

520‧‧‧Media Relay Endpoint (MRE) Network Interface Module

530‧‧‧Media Relay Endpoint (MRE) Audio Module

540‧‧‧Media Relay Endpoint (MRE) Control Module

550‧‧‧Media Relay Endpoint (MRE) Video Module

610‧‧‧RTP video input buffer

620‧‧• Video Instant Protocol (RTP) header parser and organizer

630‧‧‧Compressed video clip memory

640‧‧‧Media Relay Endpoint (MRE) Video Decoder

650‧‧‧Fragment frame memory (FM)

655‧‧‧Background frame memory

660‧‧‧Media Relay Endpoint (MRE) Continuous Rendering (CP) Image Builder

670‧‧‧Media Relay Endpoint (MRE) Continuous Presentation (CP) Frame Memory Module

680‧‧‧Proportional adjuster and frame memory (FM)

685‧‧‧Video Encoder

687‧‧‧ Temporary encoder

690‧‧•Media Relay Endpoint (MRE) Video Instant Protocol (RTP) processor

710‧‧ Real-time agreement (RTP) audio input buffer

720‧‧‧Instant Messaging (RTP) Header Profiler and Organizer

730‧‧‧Media Relay Endpoint (MRE) Sequential Audio Memory

740‧‧‧Media Relay Endpoint (MRE) Audio Decoder

750‧‧‧Audio Mixer

760‧‧‧Encoder

770‧‧‧Media Relay Endpoint (MRE) Audio Instant Protocol (RTP) processor

1 is a block diagram illustrating a multimedia conferencing system including one of a variety of novel electronic video conferencing systems, in accordance with an embodiment.

2 is a block diagram of one of the related elements of a Media Relay MCU (MRM) in accordance with an embodiment.

3 is a simplified block diagram of a related element having an agenda compressed audio RTP processor in accordance with an embodiment of the present invention.

4 is a simplified block diagram of one of the related elements of a compressed video RTP processor having an agenda according to an embodiment.

5 is a simplified block diagram of one of the associated elements of a Media Relay Endpoint (MRE) in accordance with an embodiment.

6 is a simplified block diagram of one of the elements associated with a portion of an MRE video module (MREVM) in accordance with an embodiment.

7 is a block diagram illustrating related components of a portion of an MRE audio module (MREAM) in accordance with an embodiment.

8A is a flow chart illustrating one of the related actions of a conference setting method in accordance with an embodiment.

8B is a flow chart illustrating one of the related actions of a signal conversion and control module implemented by one of the MRMs in accordance with an embodiment.

9 (including FIGS. 9A and 9B) is a flow chart illustrating one of the related actions of the MRE control module conference setting technique according to an embodiment.

10A and 10B are timing diagrams illustrating one transition period of a particular frame generated from an MRE and transmitted to one of two other MRE video streams via an MRM, in accordance with an embodiment.

11A and 11B are timing diagrams illustrating one transition period of a particular frame generated from an EP via one MRM and transmitted to one of two other MRE video streams, in accordance with an embodiment.

120‧‧‧Media Relay Multipoint Control Unit (MRM)

220‧‧‧Network Interface Module

230‧‧‧ Agenda Compressed Audio Instant Protocol (RTP) Processor

240‧‧‧Signal and control module

250‧‧‧ agenda compressed video instant protocol (RTP) processor

Claims (22)

A method for video communication, comprising: transmitting, from a first media relay endpoint, a first compressed video stream to be transmitted toward a second media relay endpoint; and the first media relay endpoint Generating a second compressed video stream to be transmitted to a third media relay endpoint, including: causing the first media relay endpoint to be one of the first encoders and the first reference frame and the first One of the second relay frames of the second encoder of the media relay endpoint is synchronized; transmitting, by the first reference frame, one of the first compressed video streams encoded by the first encoder And compressing a video frame from the second reference frame and transmitting a second compressed video frame encoded by the second encoder in the second compressed video stream. The method of claim 1, further comprising: receiving a request to send an in-frame to the third media relay endpoint, wherein the first media relay endpoint generates a third medium to be directed toward the third media The action of the second compressed video stream transmitted by the relay endpoint is performed in response to the request for a frame within the frame. The method of claim 1, wherein one of the first encoders of the first media relay endpoint is one of the first reference frame and one of the second encoders of the first media relay endpoint The operation of the second reference frame synchronization includes: generating a first sequence reference in the first encoder by encoding the first sequence video frame of the first video stream by the first encoder And generating, by the second encoder, a second sequence of video frames in the second video stream to generate a second sequence reference frame in the second encoder, based on the first sequence Encoding the second sequence of video frames with reference to a frame, wherein the second sequence of reference frames is successively more synchronized with the first sequence of reference frames, and wherein the second sequence of reference frames is the last reference frame and the One of the first sequence reference frames is finally synchronized with the reference frame. The method of claim 3, wherein the second sequence of video frames comprises: an intra-frame frame having a quality lower than one of the video frames of the first video stream; and an inter-frame frame of one of the quality increments. The method of claim 3, wherein the last video frame of one of the second sequence of video frames is a unique compressed frame. The method of claim 3, wherein losslessly compressing a difference between a current frame of the first sequence reference frame and one of the second sequence reference frames immediately following the reference frame to generate the second sequence of video images The last frame of the box. The method of claim 6, wherein the ZIP is used to losslessly compress the difference. The method of claim 3, wherein the first sequence of video frames is a sequence of interframes. The method of claim 1, wherein the first compressed video stream and the second compressed video stream comply with H.264. The method of claim 1, further comprising: Terminating the second video stream; and transmitting the first video stream to be transmitted to the second media relay endpoint and the third media relay endpoint. The method of claim 1, wherein the first video stream for relaying towards a plurality of media relay endpoints is transmitted. The method of claim 1, wherein one of the first encoders of the first media relay endpoint is one of the first reference frame and one of the second encoders of the first media relay endpoint The action of the second reference frame synchronization includes: generating a first sequence reference frame in the first encoder when encoding the first compressed video stream; and in encoding the second compressed video stream A second sequence reference frame is generated in the second encoder, wherein the video image that is scaled from a same sequence encodes the first compressed video stream and the second compressed video stream. The method of claim 12, wherein the second compressed video stream comprises: an in-frame frame having a quality lower than one of the video frames of the first compressed video stream; and an inter-frame frame of one of the quality increments. The method of claim 13, wherein the first compressed video stream comprises one of the inter-frame frames. The method of claim 12, wherein the current video frame of the same sequence of the scaled video images is encoded by the first encoder as a handover prediction (SP) frame and encoded by the second encoder as a primary Switch Forecast (SSP) frame. The method of claim 15, further comprising terminating the second compressed video stream after transmitting the SSP frame, wherein one of the second compressed video streams is an SSP frame. A media relay for providing a continuous presentation layout configuration at a first media relay endpoint during a multipoint video conference between a first media relay endpoint and a plurality of media relay endpoints a point control unit, the continuous presentation layout configuration comprising a plurality of segments, each segment displaying video from a selected media relay endpoint of the plurality of media relay endpoints, the media relay multipoint control unit comprising: a network The interface receives the relay compressed video data chunk from the plurality of media relay endpoints and sends the relay compressed video data chunk to the first media relay endpoint; a signaling and control module, The plurality of media relay endpoints select two or more media relay endpoints; and a compressed video processor that parses the received relay compressed video data chunks; Or the parsed relay compressed video data chunks received by the two or more media relay endpoints are organized into a group of two or more streams of the compressed video data chunks; and Transmitting through the network interface The group of two or more streams of compressed video data chunks are oriented toward the first media relay endpoint, wherein the compressed video processor is responsive to receipt from the signaling and control module from the selected Two or more media relay endpoints One of the presentation media relay endpoints requires the media endpoint to request an in-frame frame instruction to perform the following steps: concurrently obtaining a normal stream of one of the relay compressed video data chunks from the presentation media relay endpoint And obtaining, by the presentation media relay endpoint, a temporary stream of the relay compressed video data chunk; parsing the obtained relay compressed video data chunk on the temporary stream; and compressing the parsed relay The video data chunk is organized into a temporary stream of one of the relay compressed video data chunks; and the temporary stream of the relayed compressed video data chunks transmitted through the network interface faces the demanding media relay endpoint. The media relay multipoint control unit of claim 17, wherein the compressed video processor is further configured to relay from the presentation media relay in response to an instruction received from the signaling and control module The normal relay compressed video data chunk obtained by the point terminates the temporary stream towards the demand media relay endpoint. A media relay endpoint, comprising: a video processor, comprising: a first encoder; and a temporary encoder, wherein the video processor is configured to: transmit by the first encoder for compression Retrieving one of the first video streams toward a second media relay endpoint; receiving a third media relay endpoint requires a frame in the frame Transmitting, by the temporary encoder, for relaying a temporary video stream toward the third media relay endpoint; causing one of the first encoders to be in the first reference frame and the temporary encoder One of the second reference frames is synchronized; and the temporary video stream is terminated. The media relay endpoint of claim 19, wherein the indication is an in-frame request received from the third media relay endpoint. The media relay endpoint of claim 19, wherein the video processor causes a handover prediction (SP) frame generated by the first encoder to be synchronized with a temporary handover prediction (SSP) frame generated by the temporary encoder. A media relay endpoint (MRE), comprising: an MRE video processor, which receives a group of streams of relay compressed video data chunks originating from a group of selected MREs; The received relay compressed video data chunks are organized into a plurality of groups, each group corresponding to one of the MREs of the selected MRE; and each group of the relay compressed video data chunks received by the organization is decoded; The relay-compressed video data chunks received from the decoded organizations are successively rendered in a layout configuration, wherein the MRE video processor is configured to: obtain a block of compressed video data transmitted from a first MRE One of the temporary streams; organize, decode, and translate the temporary stream from the layout Video synchronization; the synchronization allocation is used to decode a reference frame in one of the decoders of the common video stream received by the first MRE; and organize, decode, and translate the common stream from the layout configuration Video.