JP2011507347A - System and method for storing program code and data in an application hosting center - Google Patents

Abstract

Systems and methods for storing program code and data within an application hosting center are provided. In one embodiment of a computer-implemented method, program code and / or data used to execute an online application is subdivided into a first format and a second format, and the first format of program code and data is relatively A first type of memory that provides a low latency memory access and a second type of program code and data that provides a relatively long latency memory access compared to the first type of memory. A streaming interactive video stream representing the video generated by the application is retrieved from the first memory and the second memory in response to a client request to store in the memory and execute the online application to the client. Send.
[Selection] Figure 2a

Description

  The present invention relates generally to the field of data processing systems that improve the ability of users to manipulate and access audio and video media.

  Related Application: This application is a continuation-in-part (CIP) application of No. 10 / 315,460 filed Dec. 10, 2002 entitled “APPARATUS AND METHOD FOR WIRELESS VIDEO GAMING” assigned to the assignee of the present application. It is.

  Recorded audio and video media have represented the world since the time of Thomas Edison. At the beginning of the 20th century, recorded audio media (cylinders and records) and video media (nickelodeon and movies) were widely distributed, but both technologies were still in their infancy. In the late 1920s, video was combined with audio on a mass market basis, and then became a color video with audio. Radio broadcasting has gradually evolved into a broadcast mass market audio media with a large advertising support form. With the establishment of the television (TV) broadcast standard in the mid 1940s, television joined the radio as a form of broadcast mass market media that delivered pre-recorded or live video to the home.

  Until the middle of the 20th century, most homes in the United States had a gramophone record player that played recorded audio media, a radio that received live broadcast audio, and a television receiver that showed live audio / video (A / V) media. Was. In many cases, these three “media players” (record player, radio and TV) have been combined into a single cabinet sharing a common speaker to become a home “media center”. Media options are limited to consumers, but the media “ecosystem” was fairly stable. Most consumers knew how to use "media players" and were able to fully enjoy their abilities. At the same time, publishers of media (mainly video and television studios and music companies) can distribute their media, both in theater and home, without suffering from the popular pirated or “secondary sale” or resale of used media. Could be distributed. Typically, the issuer does not earn revenue from secondary sales, and therefore the revenue that the issuer may otherwise earn from second-hand media buyers for new sales has decreased. Although there are certainly used records sold in the middle of the 20th century, such sales did not have a major impact on record issuers. This is because music tracks are heard hundreds or thousands of times, unlike videos or video programs that adults typically only watch once or several times. Thus, music media does not “corrupt” much more than video / video media (ie, continues to be valuable to adult consumers). When purchasing a record, if the consumer likes the music, the consumer will probably hold it for an extended period of time.

  From the mid-20th century to today, the media ecosystem has undergone a series of fundamental changes in both consumer and issuer profits and damages. When audio recorders, especially cassette tapes with high-quality stereo sound, were widely introduced, there was certainly a high degree of consumer convenience. However, it also marked the beginning of the pirated version, which is now widely spread with consumer media. Certainly, many consumers have used cassette tapes to tape their own records purely for convenience, but more and more consumers (eg, in dormitories with quick access to each other's record collection). Student) made a pirated copy. Consumers also taped music played over the radio rather than buying records and tapes from publishers.

  The advent of consumer-oriented VCRs can be set to record new TV VCRs to watch TV shows and can be viewed later, leading to further consumer convenience and “on-demand” basis for movies and TV programs It also led to the creation of a video rental business that can be accessed on the Internet. The rapid development of mass market home media devices since the mid 1980s has led to an unprecedented level of choice and consumer convenience, as well as a rapid expansion of the media publishing market.

  Today, consumers face excessive media choices and excessive media devices, many of which are tied to specific types of media or specific publishers. Enthusiastic media consumers have stacks of devices connected to TVs and computers in various rooms of the house, so that the cable “to one or more TV receivers and / or personal computers (PCs)”. A mouse burrow ", as well as a group of remote controllers, has arisen. (For the purposes of this application, the term “personal computer” or “PC” refers to desktops, Macintosh® or other non-Windows® computers, Windows compatible devices, Unix® variations, laptops, etc. Any device suitable for use at home or in the office, including: video game consoles, VCRs, DVD players, audio surround-sound processors / amplifiers, satellite set-top boxes, Cable TV set top box, etc. And since there are compatibility issues for enthusiastic consumers, there are multiple similar function devices. For example, a consumer may own both an HD-DVD and a Blu-ray DVD player, or have both a Microsoft Xbox® and Sony Playstation® video game system. Indeed, because some games are not compatible across game console versions, consumers may own both XBox and subsequent versions, for example, Xbox 360®. In many cases, consumers are confused as to which video input and which remote controller to use. Even after the disc is placed in the correct player (eg DVD, HD-DVD, Blu-ray, Xbox or Playstation), video and audio inputs are selected for the device and the correct remote controller is found. Still face technical challenges. For example, in the case of a wide screen DVD, the user first determines the correct aspect ratio (eg, 4: 3, full, zoom, wide zoom, cinema wide, etc.) on his TV or monitor screen and then sets It becomes necessary to do. Similarly, the user will first need to determine the correct audio surround sound system format (eg, AC-3, Dolby Digital, DTS, etc.) and then set it. In many cases, consumers are unaware that they are not enjoying the media content at their full capability of their television or audio system (for example, watching movies crushed with the wrong aspect ratio or not surround sound) Or listening to audio in stereo).

  More and more Internet-based media devices have been added to the device stack. Audio devices such as the Sonos® digital music system stream audio directly from the Internet. Similarly, a device such as a Slingbox® entertainment player can record a video and stream it through a home network or the Internet and view it on a PC at a remote location. Internet Protocol Television (IPTV) services provide cable TV-type services through a digital subscriber line (DSL) or other home Internet connection. Recently, efforts have also been made to integrate multiple media functions into a single device, such as a Moxi® Media Center or a PC running Windows XP Media Center Edition. Each of these devices provides an element of convenience for the functions it performs, but lacks ubiquity and easy access to most media. Moreover, such devices often require hundreds of dollars in manufacturing costs due to the need for expensive processing and / or local storage. In addition, these modern consumer electronic devices typically consume large amounts of power while idle, meaning that they become expensive over time and waste energy resources. . For example, a device may continue to operate if the consumer neglects to turn it off or switches to a different video input. And since none of the devices is a complete solution, they must be integrated with other stacks of devices in the home, which still leaves the user in the wire's mouse burrow and remote control sea It will be.

  Moreover, many new Internet-based devices typically provide media in a more general form than it is available when it functions properly. For example, devices that stream video over the Internet often stream only video material, rather than interactive “extra” that often accompany a DVD, such as video, game “production” or director commentary. This is often due to the fact that the interactive material is generated in a specific format intended for the specific device that handles it interactively. For example, DVD, HD-DVD and Blu-ray discs have their own specific bidirectional format. Home media devices or local computers that are developed to support all popular formats require a level of sophistication and flexibility that would probably be prohibitively expensive and complex for consumers to operate. It will be.

  In addition to this issue, if a new format is introduced in the future, the local device may not have the hardware capability to support the new format, which means that consumers can upgrade Means you have to buy a local media device. For example, if high-resolution video or stereo video (eg, one video stream for each eye) is introduced at a later date, the local device may not have the computing power to decode the video or may be new May not have hardware to output video in different formats (for example, if each eye is given 60fps and 120fps video is synchronized with shuttered glasses, stereo feeling is achieved) Assuming that if the consumer's video hardware can only support 60 fps video, this option is not available without purchasing upgrade hardware).

  The aging and complexity issues of media devices become a serious problem when it comes to sophisticated interactive media, especially video games.

Modern video game applications consist primarily of four major non-portable hardware platforms: Sony Playstation® 1, 2 and 3 (PS1, PS2 and PS3), Microsoft Xbox® and Xbox 360. (Registered trademark), Nintendo Gamecube (registered trademark) and Wii ^™ , and PC-based games. Each of these platforms, unlike the others, games that are written to run on one platform will not run on another platform. There is also a compatibility problem from one generation of devices to the next. While most software game developers create software games that are designed independently of a specific platform, in order to run a specific game on a specific platform, the game should be used on a specific platform. In order to adapt, a dedicated layer of software (often referred to as a “game development engine”) is required. Each platform is sold to the consumer as a “console” (ie, a stand-alone box attached to a TV or monitor / speaker) or is the PC itself. Typically, video games are sold on optical media such as Blu-ray DVD, DVD-ROM, or CD-ROM, including video games implemented as sophisticated real-time software applications. As home broadband speeds increase, video games are increasingly being used for download.

  The specific requirements for platform compatibility with video game software are extremely demanding due to the real-time nature and high computational requirements of advanced video games. For example, the general compatibility of manufacturing applications (eg, Microsoft Word) from one PC to another with a faster processing unit or core, as well as from one generation of video games to the next (eg, Full game compatibility is expected from XBox to XBox 360 or from Playstation 2 (PS2) to PlayStation 3 (PS3). However, this is not the case with video games. Video game manufacturers typically seek the best possible performance for a given price point when a video game generation goes on sale, so many written for previous generation systems There are often rapid architectural changes to the system that will not work on later generation systems. For example, XBox is based on the x86 family of processors, while XBox 360 is based on the PowerPC family.

  Although techniques for emulating traditional architectures can be used, it is often impossible to achieve exactly the same behavior in emulation if the video game is a real-time application. This is detrimental to consumers, video game console manufacturers, and video game software publishers. For consumers, this means that old and new generations of video game consoles need to remain connected to the TV in order to be able to play all games. For console manufacturers, this means the costs associated with emulation and the hassle of adopting a new console. In addition, publishers need to release multiple versions of new games to reach all potential consumers, that is, release versions for each video game brand (eg XBox, Playstation). In addition, it is often necessary to release a version for a given brand version (eg, PS2 and PS3). For example, separate versions of the electronic arts “Madden NFL 08” have been developed for XBox, XBox 360, PS2, PS3, Gamecube, Wii and PC, among other platforms.

  Portable devices such as cellular (cell) phones and portable media players also present challenges to game developers. Increasingly, such devices are connected to a wireless data network and are capable of downloading video games. However, there are a wide variety of cell phones and media devices on the market with a wide range of different display resolutions and computing capabilities. Also, such devices typically have constraints on power consumption, cost and weight, so advanced graphics such as graphics processing units (GPUs) such as those manufactured at NVIDEO in Santa Clara, California. It lacks acceleration hardware. As a result, game software developers typically develop a given game title simultaneously for many different types of portable devices. The user knows that a given game title is not available for his particular cell phone or portable media player.

  In the case of home game consoles, hardware platform manufacturers typically impose royalty on software game developers for their ability to publish games on their platform. Cell phone telegraph companies also typically impose royalties on game publishers to download games to cell phones. In the case of PC games, there are no royalties paid to publish the game, but game developers typically have a consumer service burden to support a wide range of possible PC configuration and installation issues. Because it is large, it faces high costs. Also, PCs typically have few barriers to game software piracy. They are easy to reprogram by technically knowledgeable users, and can easily create pirated versions of games (eg, via the Internet) for easy distribution. Thus, software game developers are costly and disadvantageous for publishing on game consoles, cell phones and PCs.

  For console and PC software game publishers, this is not the only cost. In order to distribute the game through the retail channel, the issuer imposes a wholesale price that is lower than the retail price of the retail store to gain a margin. Publishers also typically have to pay for manufacturing costs and distribute physical media that holds the game. The issuer may also consider, for example, a contingency that may occur if the game cannot be sold, if the price of the game falls, or if the retailer has to refund part or all of the wholesale price and / or take the game from the buyer. “Price protection costs” are often imposed by retailers to cover this situation. In addition, retail stores typically charge publishers to help market a game with advertising flyers. Furthermore, retail stores are increasingly buying back games from users who have finished playing the games and selling them as used games, and typically do not share used game revenue with game publishers. In addition to the cost burden imposed on game publishers, pirated versions of games are often created and distributed over the Internet, where users download and copy them for free.

  As Internet broadband speeds are increasing and broadband connections to the Internet and “cafes” where Internet-connected PCs are rented, especially in the United States and around the world, games are becoming more common It has been increasingly distributed via download to a PC or console. Also, broadband connections are increasingly being used to play multiplayer and large scale multiplayer online games (both of which are referred to by the acronym “MMOG” in this disclosure). These changes alleviate some of the problems and costs associated with retail distribution. Downloading online games presents several drawbacks to game publishers in that distribution costs are typically small and little or no cost from unsold media. However, downloaded games are still pirated and due to their size (often gigabytes in size) take a very long time to download. In addition, small disk devices such as those sold with portable computers or video game consoles are filled with multiple games. However, to the extent that games or MMOGs require an online connection for playable games, the problem of piracy is mitigated. This is because the user is usually required to have a valid user account. Unlike linear media (eg video and music) that can be copied by a camera that shoots video on a display screen or a microphone that records audio from speakers, each video game experience is unique and simple video / audio Cannot copy using record. Therefore, even in areas where copyright laws are not enforced and piracy is rampant, MMOG is shielded from piracy and can therefore support business. For example, Vivendi SA's “World of Warcraft” MMOG has been successfully deployed around the world without suffering from piracy. And many online or MMOG games, such as Linden Lab's “Second Life” MMOG, for game operators through an economic model built into the game to buy, sell and form assets using online tools. Generate revenue. Thus, in addition to conventional game software purchases or contracts, mechanisms can be used to pay for using online games.

  Although online nature or MMOG can often mitigate piracy, online game operators still face the remaining challenges. Many games require substantial local (ie, in-home) processing resources for online or MMOG to function properly. A user may not be able to play a game if they have a low-performance local computer (eg, one that does not have a GPU, such as a low-end laptop). In addition, as game consoles become older, they will gradually retreat from the latest and become unable to handle more advanced games. Even assuming that the user's local PC can handle the computational demands of the game, there is often installation complexity. The driver may not be compatible (for example, if a new game is downloaded, a new version of the graphics driver is installed, which may cause a previously installed game based on the old version of the graphics driver to be May be activated). The console runs out of local disk space as more games are downloaded. Complex games are typically found by game developers when bugs are found and fixed, or when changes are made to the game (eg, the game level is too difficult or too easy to play) You receive patches downloaded over time from game developers. These patches require a new download. However, sometimes not all users complete the download of all patches. At other times, downloaded patches introduce other compatibility or disk space consumption issues.

  Also, during game play, large data downloads may be required to provide graphics or behavior information to the local PC or console. For example, if a user enters a MMOG room and encounters a scene or character that has behaviors that are created with graphic data or are not available on the user's local machine, the scene or character data must be downloaded. Don't be. As a result, if the internet connection is not fast enough, there will be a substantial delay during game play. And if the scene or character encountered encounters storage space or computing power that goes beyond the local PC or console, the user may not be able to proceed with the game or continue with poor quality graphics. Thus, online or MMOG games often limit their memory and / or computational complexity requirements. In addition, they often limit the amount of data transfer during the game. Online or MMOG may also narrow the market for users who can play games.

  In addition, it is becoming increasingly common for users with technical knowledge to modify the game so that a local copy of the game can be reverse engineered and cheated. A cheat is as simple as repeating a button press faster than humanly possible (eg, shooting a gun very fast). In games that support in-game asset transactions, cheats reach a level of sophistication that results in fraudulent transactions involving assets of actual economic value. When online or MMOG economic models are based on such asset transactions, this has a substantially detrimental effect on the game operator.

The cost of developing new games allows PCs and consoles to create increasingly sophisticated games (with more realistic graphics such as real-time ray tracing and more realistic behavior such as real-time physical simulation). It is increasing as In the early days of the video game industry, video game development is a process very similar to application software development, i.e., the majority of development costs are software development, and graphic, audio, and behavioral elements or "assets". For example, it is not developed for moving images having a wide range of special effects. Today, many sophisticated video game development efforts closely resemble special effects video development rather than software development. For example, many video games provide 3D world simulations and generate increasingly photorealistic (ie computer graphics that look as realistic as live-motion video photography) characters, props and environments. To do. One of the most challenging aspects of photorealistic game development is to create a computer-generated human face that is indistinguishable from a living human face. Face capture technology, such as Contour ^™ reality capture, developed by Mova, San Francisco, California, captures and tracks the exact shape of the performer's face at high resolution during exercise. This technique allows 3D faces to be rendered on a PC or game console that are virtually indistinguishable from captured live motion faces. Capturing and rendering “photoreal” human faces is useful in a number of ways. First, very recognizable celebrities or sports players (often hired at high costs) are often used in video games, where imperfections are obvious to the user and disrupt the viewing experience May cause discomfort or discomfort. Also, in many cases, a high degree of detail is required to achieve a high degree of photorealism, i.e., as polygons and / or textures change from frame to frame as the face moves, It is required to render a large number of polygons and high resolution textures.

  When a large number of polygonal scenes with detailed textures change rapidly, the PC or game console supporting the game will have enough polygon and texture data for the required number of animation frames generated in the game segment. You may not have enough RAM to store. In addition, a single optical drive or single disk drive typically available on a PC or console is usually much slower than RAM and typically has a GPU in rendering polygons and textures. Cannot maintain the maximum data rate that is acceptable. Current games typically load most of the polygons and textures into RAM, which means that the complexity and duration of a given scene is primarily limited by the capacity of the RAM. For example, in the case of facial animation, this can cause the PC or game console to have a non-photoreal low resolution before the game pauses and loads polygons and textures (and other data) for more frames. Or a photoreal face that can be animated only for a limited number of frames.

  Watching the progress bar move slowly across the screen when the PC or console displays a message similar to "Loading ..." is an inherent drawback by today's users of complex video games It is tolerated. Delay while the next scene loads from disk (where "disk" refers to non-volatile optical or magnetic media, as well as non-disk media, eg, semiconductor "flash" memory, unless otherwise indicated) Takes several seconds or minutes. This is a waste of time and is quite frustrating for the game player. As mentioned above, many or all of the delays are due to the time it takes to load polygons, textures or other data from the disk, but the processor and / or GPU in the PC or console will have data for the scene. Some load time may be spent during preparation. For example, a soccer video game allows a player to select from a number of players, teams, stadiums, and weather conditions. Thus, different polygons, textures and other data (collectively “objects”) are required for a scene based on what particular combination is selected (eg, different teams are required to With different colors and patterns). Many or all of the various permutations can be listed, many or all objects can be pre-calculated and stored on the disk used to store the game. However, if the number of permutations is large, the amount of storage required for all objects may be too large to fit on the disk (impossible to download). Thus, existing PC and console systems are typically limited in both the complexity and play duration of a given scene and are plagued with long load times for complex scenes.

  Another significant limitation with conventional video game systems and application software systems is the large database of 3D objects such as polygons and textures that need to be loaded for processing to a PC or game console, for example. Is being used gradually. As mentioned above, such databases require a long load time when stored locally on disk. However, load times are usually more severe when the database is stored at a remote location and accessed through the Internet. In such a situation, downloading a large database may take minutes, hours, or days. In addition, such databases are often generated at great expense (eg 3D models of detailed high-masted sailing vessels for use in games, movies or historical documentaries) and sold to local end users Is intended to be. However, if the database is downloaded to a local user, it may be subject to piracy. In many cases, the user simply evaluates the database to see if it meets the user's needs (eg, whether the 3D costume of the game character has a satisfactory look or appearance when the user makes a particular move). I would like to download a database for Long load times are a hindrance for users who evaluate 3D databases before making a decision to purchase.

  In MMOG, a similar problem occurs particularly as a game in which a user can gradually use customized characters. In the case of a PC or game console for displaying a character, access to a database of 3D geometry (polygons, textures, etc.) and behavior for that character (eg, if the character has a shield, It is necessary to obtain whether or not the strength is sufficient to divert. Typically, when the MMOG is first played by the user, a large number of databases for the characters are available in advance along with an initial copy of the game, which is available locally on the game's optical disc or downloaded to the disc. The However, as the game progresses, if the user encounters a character or object whose database is not locally available (for example, if another user generates a customized character), that character or object is displayed. By the time, the database must be downloaded. This causes a substantial delay in the game.

  Given the sophistication and complexity of video games, another challenge for video game developers and publishers with traditional video game consoles often costs two to three years at a cost of tens of millions of dollars. Multiply it to develop a video game. Assuming that a new video game console platform is introduced approximately once every five years, game developers will be able to obtain video games at the same time that the new platform is launched. It is necessary to start development work just before the launch of a new game console just before these game years. Many consoles are sometimes released from competing manufacturers at about the same time (eg, within one or two years of each other), but what must be examined is the popularity of each console, for example which consoles Is it the biggest sales of video game software? For example, in recent console cycles, Microsoft XBox 360, Sony Playstation 3 and Nintendo Wii are scheduled to be introduced in approximately the same general time frame. However, some years before its introduction, game developers must essentially “bet” which console platforms will be more successful than others and devote development resources accordingly. Video production companies must also allocate limited production resources based on an estimate of the movie's success potential well before the movie's release. Given the increased level of investment required for video games, game production is becoming increasingly similar to video production, and game production companies routinely allocate production resources based on estimates of the future success of a particular video game. Dedicated. However, unlike a video company, this bet is not based solely on the success of the production itself, but rather on the success of the game console intended to run the game. Although launching games for multiple consoles at once can mitigate risk, this additional effort increases costs and often delays the actual launch of the game.

The application software and user environment on the PC is not only more visually appealing to the user, but also more computationally intensive, dynamic and interactive in order to be more effective and intuitive It becomes. For example, the new Windows Vista ^™ operating system and successive versions of the Macintosh® operating system incorporate visual animation effects. Advanced graphics tools such as Maya ^™ from Autodesk provide very sophisticated 3D rendering and animation capabilities that push the limits of modern CPUs and GPUs. However, the computational requirements of these new tools create a number of practical problems for users and software developers of such products.

  Since the visual representation of the operating system (OS) must function on a wide variety of computers, including previous generation computers that are no longer sold but can still be upgraded with a new OS, OS graphics requirements are It is largely limited by the lowest common denominator of the computer that is the target of the OS, typically including computers that do not include a GPU. This severely limits the graphics capabilities of the OS. In addition, battery-operated portable computers (eg, laptops) limit visual display capabilities. This is because high computational activity in the CPU or GPU typically increases power consumption and shortens battery life. Portable computers typically include software that automatically reduces processor activity to reduce power consumption when the processor is not utilized. In some computer models, the user can manually reduce processor activity. For example, Sony's VGN-SZ280P laptop displays “Stana” on one side (for low performance, long battery life) and “Speed” on the other side (for high performance, short battery life) Included switches. An OS running on a portable computer must be able to function usefully even when the computer runs at part of its peak performance capability. Thus, OS graphics performance often remains far below the latest computing power available.

  Applications that are resistant to high end computing such as Maya are often sold with the expectation that they will be used in high performance PCs. This typically establishes the lowest common denominator requirement of very high performance, more expensive and less portable. As a result, such applications have target viewers that are much more limited than general purpose OSs (or general purpose production applications such as Microsoft Office), and are typically general purpose OS software or general purpose application software. They are sold in much smaller quantities. Also, in many cases, it is difficult for prospective users to try out applications that are robust to such calculations in advance, further limiting the potential audience. For example, before a student wants to learn how to use Maya, or a potential buyer who is already knowledgeable about such applications invests in a purchase (including buying a high-end computer that can run Maya), Maya Suppose you want to try. A student or potential buyer can download a demo version of Maya or get a physical media copy of it, but there is no computer that can run Maya with its full capabilities (eg, handling complex 3D scenes) In some cases, a complete inform assessment of the product cannot be performed. This limits the audience of such high-end applications. This also contributes to high selling prices. This is because development costs are usually amortized over a much smaller number of purchases than general purpose applications.

  High-priced applications also create a great incentive for individuals and businesses to use pirated copies of application software. As a result, high-end application software has been plagued by the proliferation of pirated copies, despite significant efforts by publishers of such software to mitigate piracy through various technologies. Even when using pirated high-end applications, users cannot eliminate the need to invest in expensive modern PCs to perform pirated copies. Thus, pirated software users can use software applications at a fraction of their actual retail price, but still need to purchase or obtain expensive PCs in order to fully utilize the application.

  The same is true for users of high-performance pirated video games. Pirates can get the game at a fraction of their actual price, but expensive computing hardware (such as a GPU-enhanced PC or XBox 360) required to play the game properly It is still required to purchase a high-end video game console. Assuming video games are typically a distraction for consumers, the additional cost for high-end video game systems becomes prohibitive. This situation is even worse in countries where the average annual salary of current workers is much lower than in the United States (eg, China). As a result, only a very small percentage of the population owns high-end video game systems or high-end PCs. In such countries, “Internet cafes” in which users pay a fee for using a computer connected to the Internet are very common. In many cases, such Internet cafes have only older models or low-end PCs that do not have high performance features such as GPUs that allow players to play computationally intensive video games. This is an important factor in the success of games running on low-end PCs, such as Vivendi's “World of Warcraft”, which has been very successful in China and is commonly played in Internet cafes there. In contrast, computationally strong games such as “Second Life” are very unlikely to be played on a PC installed in a Chinese internet cafe. Such a game is virtually inaccessible to users who can only access a low-performance PC in an Internet cafe.

  There is also a barrier for users who are considering buying a video game and want to try it out by first downloading a demo version of the game to their home via the Internet. Video game demos are often full-featured versions of games where some features are disabled or the amount of play on the game is limited. This may involve a long process (possibly hours) of downloading a gigabyte game before it can be installed and run on either a PC or console. In the case of a PC, calculate the special drivers needed for the game (eg DirectX or OpenGL drivers), download the correct version and install them, then the PC can play the game Including determining if. This latter step is where the PC has sufficient processing (CPU and GPU) capability, sufficient RAM, and a compatible OS (for example, some games run on Windows XP but not Vista). Including determining whether or not. Thus, after a long process of trying to run a video game demo, the user can find that the video game demo is probably not playable given the user's PC configuration. Unfortunately, when a user downloads a new driver to try out the demo, those driver versions may not be compatible with other games or applications that the user normally uses on the PC, so If you install the demo, games or applications that were able to run before may become inoperable. These barriers not only frustrate the user, but also become a barrier for video game software publishers and video game developers who bring games to market.

  Another problem that leads to economic inefficiencies relates to the fact that a given PC or game console is typically designed to accept a certain level of performance requirements for applications and / or games. For example, some PCs have some RAM, have a slow or fast CPU, and if they have a GPU, have a slow or fast GPU. Some games or applications take advantage of the total computational power of a given PC or console, but many games or applications do not. If the user's choice for a game or application does not reach the peak performance capability of the local PC or console, the user will waste money on the PC or console for features that are not used. In the case of a console, the console manufacturer has paid more than needed to subsidize the console cost.

  Another problem that exists when buying and enjoying a video game relates to allowing the user to see others playing the game before the user decides to purchase the game. There are a number of conventional solutions for recording video games for later playback. For example, US Pat. No. 5,558,339 records game information body information including game controller actions during “game play” on video game client computers (owned by the same user or different users). Teaches. This status information can later be used to play some or all game actions on a video game client computer (eg, a PC or console). A significant disadvantage of this solution is that in order for the user to view the recorded game, the user has a video game client computer that can play the game, and the game play is not played when the recorded game state is played. You must have a video game application running on that computer to be identical. In addition, the video game application must be written so that there is no difference in execution between the recorded game and the played game.

  For example, game graphics are generally calculated for each frame. For many games, the game logic sometimes determines whether the scene is particularly complex or if there are other delays that slow down execution (eg, another process of removing CPU cycles from a game application on a PC). Takes less than or longer than one frame to calculate the graphic to be displayed for the next frame. In such a game, a “threshold” frame is finally generated which is calculated slightly shorter than one frame time (for example, several CPU clock cycles shorter). When the same scene is recalculated using exactly the same game state information, it can easily take several CPU clock cycles longer than one frame time (for example, the internal CPU bus is slightly out of phase with the external DRAM bus). And if you introduce a time delay of a few CPU cycles, even if there is no significant delay due to another process that takes a few milliseconds of CPU time out of the game processing). Therefore, when the game is played, the frame is calculated in two frame times instead of one frame time. Some behaviors are based on how often the game calculates new frames (eg, when the game samples input from the game controller). While the game is being displayed, this difference in time reference for different behaviors does not affect game play, but causes the played game to produce different results. For example, if a basketball trajectory is calculated at a constant 60 fps speed, but the game controller input is sampled based on the calculated frame speed, the calculated frame speed is recorded by the game. 53fps when played, but 52fps when the game is played, which will make a difference in whether basketball is prevented from entering the basket, resulting in different results. Thus, recording a video game using game state requires a very careful game software design to ensure that playback using the same game state information produces exactly the same results.

  Another conventional solution for recording video games is to simply record the video output of a PC or video game system (eg, to a VCR, DVD recorder, or to a video capture board on the PC). The video can then be rewound and played, or the recorded video can typically be compressed and uploaded to the Internet. The drawback of this solution is that when the 3D game sequence is played, the user is limited to viewing the sequence only from the viewpoint where the sequence was recorded. In other words, the user cannot change the viewpoint of the scene.

  Furthermore, when a compressed video of a recorded game sequence that is played on a home PC or game console is available to other users over the Internet, the compressed video can be viewed in real time even if the video is compressed in real time. It is impossible to upload to the Internet. The reason is that many homes around the world connected to the Internet have very asymmetric broadband connections (eg, DSL and cable modems typically have a much wider downstream bandwidth than the upstream bandwidth). Because it has a width). Compressed high resolution video sequences often have a bandwidth that is wider than the upstream bandwidth capacity of the network, making them impossible to upload in real time. Thus, after the game sequence is played, there will be a significant delay (possibly minutes or hours) before another user on the Internet can watch the game. This delay can be tolerated in certain situations (eg, looking at a previously played game player's performance), but the ability to see a game (eg, a basketball tournament played by a champion player) live, or a game played live Eliminate the “instant play” ability when done.

  Another conventional solution allows viewers to watch video games live on a television set only under the control of a television producer. Some channels in the United States and other countries offer video game viewing channels, and television viewers watch some video game users (eg, top rated players playing in tournaments) on the video game channel. Can do. This is accomplished by feeding the video output of the video game system (PC and / or console) to a video distribution and processing device for the television channel. This is different from when a television channel broadcasts a live basketball game where multiple cameras send live video from different angles around the basketball court. Thus, television channels can take advantage of their video / audio processing to act on the device to manipulate the output from various video game systems. For example, a television channel can superimpose text indicating the status of different players on top of the video from the video game (as if it were overlay text in a live basketball game), and the television channel You can record audio from commentators who can discuss the actions that occur inside. In addition, the video game output can be combined with a camera that records the video of the actual player of the game (eg, showing an emotional response to the game).

  One problem with this solution is that in order to gain live broadcast excitement, such a live video feed must be available in real time to the video distribution and processing device on the television channel. However, as noted above, this is often not useful when the video game system is run from home, especially if a portion of the broadcast includes live video from a camera that captures the game player's real-world video. Is possible. Further, in the tournament state, as described above, there is a problem that a game player in the home changes and cheats the game. For these reasons, such video game broadcasts over television channels are where players and video game systems gather at a common location (eg, a television studio or stadium), where the television production device has multiple video games. Often configured to accept video feeds from the system and potential live cameras.

  Such traditional video game TV channels are similar in experience to live sporting events, for example, video game players are depicted as “athletes” for both action in the video game world and action in the real world. However, these video game systems are often limited to players that are physically very close to each other. And since a television channel was broadcast, each broadcast channel can only show one video stream selected by the producer of the television channel. Because of these limitations and the high cost of airtime, production equipment, and production personnel, television channels typically show only the highest rated players that play in top tournaments.

  Furthermore, a given television channel that broadcasts a full screen video of a video game to all television viewers shows only one video game at a time. This severely limits television viewer options. For example, a television viewer may not be interested in a game that is played at a given time. Another viewer is only interested in watching a particular player's game play not featured by the television channel at a given time. In other cases, the viewer is only interested in seeing how a specialized player handles a particular level in the game. Still other viewers desire to control the viewpoint of viewing the video game, which is different from that selected by the production team or the like. In short, television viewers have countless preferences for watching video games that are unacceptable by certain broadcasts on the television network, even when many different television channels are viewed. For all the reasons described above, conventional video game television channels have significant limitations in presenting video games to television viewers.

  Another drawback of conventional video game systems and application software systems is that they are complex and usually plagued by errors, crashes and / or unintended and undesired behavior (collectively “bugs”). That is. Games and applications are typically passed through a debugging and tuning process (often referred to as “Software Quality Assurance” or SQA) before launch, but almost always the games or applications are a wide audience in the field. Bugs suddenly appear when released. Unfortunately, it is difficult for software developers to identify and track many bugs after release. It is difficult for software developers to notice bugs. Even when learning about bugs, there is only a limited amount of information available to identify what caused the bug. For example, when a user calls the game developer's customer service line and plays a game, the screen begins to flash and then turns dark blue, leaving a message indicating that the PC is freezing. This gives the SQA team very little information useful for tracking bugs. A game or application that is connected online can sometimes provide a lot of information in some cases. For example, a “watchdog” process can sometimes be used to monitor a game or application for “crash”. The watchdog process collects statistical information about its state (eg, stack state, memory usage, how far the game or application has progressed, etc.) when the game or application process crashes, and then collects that information. You can upload to the SQA team via the Internet. However, in complex games or applications, such information can take a very long time to decipher to accurately determine what the user has done at the time of the crash. Therefore, it is impossible to determine what event sequence caused the crash.

  Yet another problem associated with PCs and game consoles is that they suffer from service problems that are very inconvenient for consumers. These service issues also affect PC or game console manufacturers. This is because the manufacturer needs to send a special box to safely carry a broken PC or console and bear the cost of repair if the PC or console is within the warranty period. . Game or application software publishers are also affected by lost sales (or online service usage) due to the PC and / or console being in repair.

FIG. 1 shows a conventional video game system such as Sony Playstation® 3, Microsoft XBox 360®, Nintendo Wii ^™ , Windows-based personal computer, or Apple Macintosh. Each of these systems is a central processing unit (CPU) for executing program code, typically a graphics processing unit (GPU) for performing advanced graphics operations, and for communicating with external devices and users. Multiple forms of input / output (I / O) are provided. For simplicity, these components are shown coupled together as a single unit 100. The conventional video game system of FIG. 1 also plays an optical media drive 104 (for example, a DVD-ROM drive), a hard drive 103 for storing video game program codes and data, a multiplayer game, , Network connection 105 for downloading patches, demos or other media, random access memory (RAM) 101 for storing program code currently being executed by the CPU / GPU 100, input commands from the user during game play And a display device 102 (eg, SDTV / HDTV or computer monitor).

  The conventional system shown in FIG. 1 suffers from a number of limitations. First, the optical drive 104 and the hard drive 103 tend to have a very low access speed compared to the RAM 101. When functioning directly through the RAM 101, the CPU / GPU 100 can actually process much more polygons per second than is possible when program code and data are read directly from the hard drive 103 or optical drive 104. . This is because RAM 101 is generally very wide in bandwidth and is not bothered by the relatively long seek delay of the disk mechanism. However, in these conventional systems, only a limited amount of RAM is provided (eg, 256-512 Mbytes). Therefore, a “Loading ...” sequence is often required in which the RAM 101 is periodically filled with data for the next sequence of the video game.

  Some systems attempt to superimpose program code loading at the same time as gameplay, but this can only be done when there is a known sequence of events (for example, when driving a car and going down the road). , While driving, you can load the approaching building geometry by the roadside). For complex and / or rapid scene changes, this type of superposition usually does not work. For example, if the user is in the middle of a battle and the RAM 101 is completely filled with data representing objects in the field of view at that moment, the user quickly left the field of view to see objects that are not currently loaded in the RAM 101. If you move to, there will be a discontinuity of action. This is because there is not enough time to load a new object from the hard drive 103 or optical media 104 into the RAM 101.

  In the system of FIG. 1, another problem arises due to the limited storage capacity of the hard drive 103 and optical media 104. Although disk storage devices with relatively large storage capacities (eg, 50 gigabytes or more) can be manufactured, they still cannot provide sufficient storage capacity for some scenes currently encountered in video games. For example, as described above, a soccer video game allows a user to select from a large number of teams, players and stadiums around the world. For each team, each player and each stadium, a very large number of texture maps and environment maps are required to characterize the 3D curved surface around the world (eg, each team has a unique jersey, each of which Requires a unique texture map).

  One technique used to address this latter problem is that the game pre-calculates textures and environment maps as they are selected by the user. This may involve a number of computationally intensive processes, including video decompression, 3D mapping, shading, data structure organization, etc. As a result, there may be a delay for the user while the video game performs these calculations. One way to reduce this delay is in principle to perform all of these calculations, including team, player roster, and stadium permutations when the game is first developed. Thus, the release version of the game downloads all this preprocessed data stored on the optical media 104 or one or more servers on the Internet to the hard drive 103 over the Internet when the user makes a selection. With a given pre-processed data for a given team, player directory, stadium selection. However, as a practical matter, such pre-loaded data for each possible permutation in game play will likely be terabytes of data that far exceeds the capacity of today's optical media devices. Furthermore, the data for a given team, player list, and stadium selection will probably be more than a few hundred megabytes. For example, in a 10 Mbps home network connection, downloading this data through the network connection 105 takes more time than calculating the data locally.

  Thus, the conventional game architecture shown in FIG. 1 causes the user to experience significant delays between major scene transitions in complex games.

  Another problem with the conventional solution shown in FIG. 1 is that video games tend to be more advanced year by year, requiring higher CPU / GPU power. Therefore, even assuming a limited amount of RAM, video game hardware requirements exceed the peak processing power levels available for these systems. As a result, users need to upgrade their game hardware every few years to keep pace (or play new games at a lower quality level). As a result of the ever-increasing trend of video games, video game play machines for home use are typically economical because their cost is usually determined by the requirements of the highest performance game they can support. Inefficient. For example, XBox 360 is used to play games such as "Gears of War" that require a high performance CPU, GPU, and hundreds of megabytes of RAM, or XBox 360 is several kilobytes of RAM And is used to play Pac Man, a 1970s game that requires only a very low performance CPU. In fact, XBox 360 has enough computing power to host many Pac Man games at once.

  Video game machines are typically turned off for most of the week. According to the Nielsen Entertainment Study July 2006 issue on active games over 13 years old, on average, active games spend 14 hours a week playing console video games, ie a week It is only 12% of the total time. This means that, on average, video game consoles are idle for 88% of the time, and expensive resources are inefficiently used. Assuming that video game consoles are often subsidized by manufacturers to reduce the purchase price (in the hope that the subsidy will be reduced by royalties from future video game software purchases) This is particularly significant.

  Video game consoles also incur costs associated with most consumer electronic devices. For example, system electronics and mechanisms need to be housed in an enclosure. The manufacturer needs to provide a repair warranty. The retail store that sells the system needs to collect margins in selling the system and / or selling video game software. All of these factors add to the cost of the video game console, which must be subsidized by the manufacturer, routed to the consumer, or both.

  Furthermore, piracy is a major problem for the video game industry. The security mechanisms used in virtually all major video game systems are “cracked” year by year and unauthorized copies of video games are taken. For example, the XBox 360 security system was cracked in July 2006 and users can now download unauthorized copies online. Downloadable games (eg, PC or Mac games) are particularly susceptible to piracy. In some parts of the world where piracy is weak, there is essentially no market for success for standalone video game software. This is because users can buy pirated copies as easily as legal copies at a fraction of the cost. Also, in many parts of the world, the cost of game consoles is a high percentage of revenue, and even if the pirated version is cracked, only a few people can afford the latest game system.

  In addition, the used game market reduces revenue for the video game industry. When a user gets tired of the game, the user can sell the game to a store, and the store resells the game to other users. This unauthorized but general practice significantly reduces the game publisher's revenue. Similarly, when there is a platform transition every few years, there is usually a 50% drop in sales. This means that when the user knows that a new version platform is about to be released, it stops purchasing games for the old platform (eg, when Playstation 3 is about to be released, the user purchases a game for Playstation 2 Because it stops. The loss of sales and increased development costs associated with the new platform will have a huge impact on game developers' profits.

  Also, the new game console is very expensive. XBox 360, Nintendo Wii, and Sony Playstation 3 are all retailed for hundreds of dollars. High power personal computer game systems are priced up to $ 8000. This represents a significant investment for the user, especially considering that the hardware will become obsolete after a few years and many systems will be purchased for children.

  One solution to the above problem is an online game that is delivered as compressed video and audio where game program code and data are hosted on a server and streamed on demand to client machines over a digital broadband network. Some companies, such as the Finnish G-Cluster (currently a subsidiary of Softbank Broadmedia in Japan), offer these services online. Similar gaming services are available in local networks, such as in a hotel, and are provided by DSL and cable television providers. The main drawback of these systems is the latency, ie, the time it takes for the signal to travel to and from a game server typically located at the operator's “head end”. Fast action video games (also known as “twitch” video games) are very much between the time a user performs an action on the game controller and the time the display screen is updated to show the result of the user action. Require a short waiting time. In order for the user to feel like the game responds “instantly”, a short waiting time is required. Users can be satisfied with different waiting intervals based on the type of game and the level of proficiency of the user. For example, a slow casual game (such as backgammon) or a game that plays the role of slow action can tolerate a 100ms latency, but in a fast action game, if the latency exceeds 70 or 80ms, the user , The performance in the game is inadequate and unacceptable. For example, in a game that requires a fast reaction time, the accuracy sharply decreases as the waiting time increases from 50 to 100 ms.

  When a game or application server is installed in a nearby controlled network environment or installed in an environment where the network path to the user is predictable and / or can tolerate bandwidth peaks, For both latency consistency, it is much easier to control latency (eg, as users observe certain movements from digital video streaming through the network). This level of control can be done between the cable TV network headend and the cable TV subscriber's home, or from the DSL central office to the DSL subscriber's home, or locally in the commercial office from the server or user. It can be achieved in an area network (LAN) environment. In addition, special grade point-to-point private connections can be obtained between companies with guaranteed bandwidth and latency. However, in a game or application system that hosts a game in a general server center connected to the Internet, and then streams compressed video to a user via a broadband connection, latency is incurred due to a number of factors, and the development of conventional systems Cause severe restrictions.

  In a typical broadband home, the user can have a DSL or cable modem for broadband services. Such broadband services typically incur a round trip latency of about 25 ms (sometimes more) between the user's home and the general Internet. In addition, there is a round trip delay from routing data to the server center via the Internet. Latency through the Internet varies based on the route the data is given and the delay incurred when it is routed. In addition to routing delays, round trip delays are also incurred due to the speed of light traveling through the optical fiber interconnecting most Internets. For example, every 1000 miles incurs a round trip latency of about 22 ms due to the speed of light through the optical fiber and other overhead.

  Additional latency occurs due to the data rate of data streamed over the Internet. For example, if a user has a DSL service sold as a “6 Mbps DSL service”, in fact, the user will probably only get a downstream throughput of less than 5 Mbps and probably a digital subscriber line access multiplexer (DSLAM). Due to various factors, such as congestion during peak load times, the connection quality will be seen periodically. Also, if the local shared coaxial cable that is looped through the adjacent part of the cable modem system network or somewhere else is congested, the data of the cable modem used for the connection sold as “6 Mbps cable modem service” A similar problem occurs that reduces the rate much lower. If data packets at a constant rate of 4 Mbps are streamed in one direction in the User Datagram Protocol (UDP) format via such a connection from the server center, if everything works well, the data packets are In typical situations, packets are dropped if they pass without incurring significant latency, but there is congestion (or other obstacles) and only 3.5 Mbps is available to stream data to the user. Either data loss occurs, or packets are queued at the congestion point until packets can be sent, resulting in additional latency. Different congestion points have different queuing capacities holding delayed packets, so in some cases, packets that cannot pass congestion are dropped immediately. In other cases, several megabits of data are queued and eventually sent out. However, in almost all cases, the queue at the congestion point has a capacity limit, and when these limits are exceeded, the queue overflows and packets are dropped. Therefore, in order to avoid incurring additional latency (or worse, packet loss), it is necessary not to exceed the data rate capacity from the game or application server to the user.

  There is also latency due to the time required to compress the video at the server and decompress the video at the client device. In addition, the video game running on the server also incurs latency while calculating the next frame to be displayed. Currently available video compression algorithms are plagued by either high data rates or long latency. For example, Motion JPEG is an intra frame-only lossy compression algorithm characterized by a short waiting time. Each video frame is compressed independently of each other's video frame. When the client device receives a frame of compressed motion JPEG video, it immediately decompresses and displays the frame, greatly reducing latency. However, since each frame is compressed separately, the algorithm cannot take advantage of the similarity between successive frames, and as a result, intra-frame-only video compression algorithms are plagued by very high data rates. ing. For example, 60 fps (frames / second) 640 × 480 motion JPEG video requires data of 40 Mbps (megabits / second) or more. Such high data rates for such low resolution video windows can be prohibitively expensive in many broadband applications (and certainly for most consumer Internet-based applications). Furthermore, since each frame is compressed independently, defects in the frame that may result from lossy compression will likely appear at different locations in successive frames. This appears to the viewer as a moving visual defect when the video is decompressed.

  MPEG2, H.M. from Microsoft Corporation Other compression algorithms such as H.264 or VC9 can achieve high compression ratios when used in conventional configurations, but at the expense of long latency. Such an algorithm uses inter-frame and intra-frame compression. Periodically, such an algorithm performs intra-frame dedicated compression of the frame. Such a frame is known as a key frame (typically referred to as an “I” frame). These algorithms then typically compare the I frame with both the previous and successive frames. Rather than compressing the previous and successive frames independently, the algorithm determines what has changed in the video from the I frame to the previous and successive frames, and then determines those changes as The change preceding the I frame is stored as a “B” frame, and the change following the I frame is stored as a “P” frame. This results in a data rate that is much slower than intraframe-only compression. However, this typically comes at the expense of long latency. I frames are typically much larger than B or P frames (often more than 10 times), so that they take a proportionally longer time to transmit at a given data rate.

  For example, if the I frame is 10 times the size of the B and P frames, there are 29 B frames + 30 P frames = 59 inter frames per single I intra frame, or “frame group Consider a state where there are a total of 60 frames for each (GOP) ". Therefore, at 60 fps, there is one 60 frame GOP per second. Assume that the transmission channel has a maximum data rate of 2 Mbps. In order to obtain the highest quality video on the channel, the compression algorithm generates a 2 Mbps data stream, which, given the above ratio, is 2 megabits (Mb) / (59 + 10) = 30,394 bits / intra Frame and 303,935 bits / I frame. When a compressed video stream is received by a decompression algorithm, each frame needs to be decompressed and displayed at regular intervals (eg, 60 fps) in order to play the video steadily. To obtain this result, if a frame experiences transmission latency, all frames need to be delayed by at least that latency, so the worst case frame latency is the latency for each video frame. Define Since I-frames are the largest, the longest transmission latency is introduced and the entire I-frame must be received before the I-frame can be decompressed and displayed (or the inter-frame depends on the I-frame) . Assuming that the channel data rate is 2 Mbps, 303,935 / 2 Mb = 145 ms is required to transmit the I frame.

  The inter-frame video compression system described above that uses most of the bandwidth of the transmission channel will experience long latency due to the large size of the I frame relative to the average size of the frame. Or, in other words, traditional interframe compression algorithms achieve a lower average data rate per frame than intraframe-only compression algorithms (eg, 2 Mbps vs. 40 Mbps) but still remain high per frame due to large I frames. Annoyed by peak data rates (eg, 303,935 * 60 = 18.2 Mbps). Note that the analysis assumes that both P and B frames are much smaller than I frames. This is generally true, but not true for frames where high video complexity is not correlated to previous frames, high motion or scene changes. In such a situation, the P or B frame is as large as the I frame (if the P or B frame is larger than the I frame, sophisticated compression algorithms typically “force” the I frame. And replace the P or B frame with an I frame). Thus, a digital video stream always has an I frame size data rate peak. Thus, for compressed video, when the average video data rate approaches the data rate capacity of the transmission channel (which is often the case, assuming high data rate requirements for video), from I frames or large P or B frames High peak data rates result in long frame latency.

  Of course, the above description only characterizes the compression algorithm latency caused by large B, P or I frames in a GOP. When B frames are used, the waiting time is longer. This is because all the B frames after the B frame and the I frame must be received before the B frame can be displayed. Thus, in a picture group (GOP) sequence such as BBBBBBIPPPPPBBBBBBIPPPPP with 5 B frames preceding each I frame, the first B frame is displayed by the video decompressor until subsequent B and I frames are received. Can not do it. Thus, if the video is streamed at 60 fps (ie 16.67 ms / frame), no matter how fast the channel bandwidth is before the first B frame can be decompressed, It takes 16.67 * 6 = 100 ms to receive, which is exactly 5 B frames. Compressed video sequences with 30 B frames are very common. And at low channel bandwidths, such as 2 Mbps, the latency effect caused by the size of the I frame is primarily added to the latency effect due to waiting for the B frame to arrive. Thus, in a 2 Mbps channel with a very large number of B frames, it is very easy to exceed 500 ms latency using conventional video compression techniques. If the B frame is not used (at the expense of a low compression ratio for a given quality level), it does not incur the B frame latency, but the latency caused by the peak frame size described above is still suffer.

  This problem is exacerbated by the nature of many video games. Video compression algorithms that use the GOP structure described above are optimized for use primarily with raw video or video material intended for passive viewing. Typically, the camera (whether it is a real camera or a virtual camera in the case of computer-generated animation) and the scene are relatively stable. Simply because if the camera or scene moves too suddenly, the video or movie material is (a) typically uncomfortable to watch, and (b) This is because the viewer usually cannot follow the action exactly when moving (e.g. when shooting a child blowing a candle on a birthday cake, the camera is bumped and suddenly moves away from the cake) When moved with a gui, the viewer typically concentrates on the child and the cake, ignoring short breaks when the camera suddenly moves). In the case of video interviews or video teleconferencing, the camera is held in a fixed position and does not move at all, producing very few data peaks. However, 3D high action video games are characterized by constant movement (eg, consider a 3D race where all frames are in rapid motion during the race, or the virtual camera always moves guilty) Think of a first-person shooter). Such a video game results in a frame sequence with large and frequent peaks where the user needs to see clearly what happened during these sudden movements. Thus, compression defects are almost unacceptable in 3D high action video games. Thus, the video output of many video games, due to their nature, produces a compressed video stream with very high and frequent peaks.

  Assuming that users of fast action video games are not very tolerant of long latencies, and assuming all the causes of latencies mentioned above, to date, server-hosted streaming video over the Internet There were restrictions on video games. In addition, users of applications that require a high degree of interactivity are similarly plagued with limitations if the application is hosted on general internet and stream video. Such services include hosting servers in a commercial setting, headend (for cable broadband), or central office (for digital subscriber line (DSL)), or LAN (or a special grade private connection). Directly into the network, the route and distance from the client device to the server is controlled to minimize latency, and requires a network configuration that can accept peaks without incurring latency. LANs (typically rated at 100 Mbps to 1 Gbps) and sufficient bandwidth lease lines can typically support peak bandwidth requirements (eg, a peak bandwidth of 18 Mbps is a small LAN capacity of 100 Mbps). Part).

  Also, the peak bandwidth request can be accepted by the residential broadband infrastructure if special acceptance is made. For example, in cable TV systems, digital video traffic is given a dedicated bandwidth that can handle peaks like large I frames. And in a DSL system, a high speed DSL modem can be prepared to allow for high peaks or a special grade connection that can handle high data rates. However, conventional cable modem and DSL infrastructures attached to the general Internet have a much lower tolerance for the peak bandwidth requirements of compressed video. Thus, online services that host video games or applications at a server center long distance from the client device and then stream the compressed video output over the Internet through a traditional residential broadband connection, in particular, have very low latency. With respect to games and applications that require it (eg, first person shooters and other multi-user interactive action games, or applications that require fast response time), they suffer from significant latency and peak bandwidth limitations.

  The present disclosure will be more fully understood from the accompanying drawings and the following detailed description. However, the subject matter disclosed herein is merely illustrative of the invention and is not limited to the specific embodiments shown.

1 shows the architecture of a conventional video game system. 1 illustrates a high level system architecture according to one embodiment. 1 illustrates a high level system architecture according to one embodiment. The actual data rate, rated data rate, and required data rate for communication between the client and server are shown. Fig. 4 illustrates a hosting service and client used by one embodiment. Fig. 4 illustrates an exemplary latency associated with communication between a client and a hosting service. 2 illustrates a client device according to one embodiment. Fig. 6 illustrates a client device according to another embodiment. FIG. 4c is a block diagram of the client device of FIG. 4c. 4d is a block diagram of the client device of FIG. 4d. FIG. Fig. 3 illustrates one form of video compression that can be used according to one embodiment. 6 illustrates one form of video compression that can be used in another embodiment. The peak of the data rate associated with the transmission of a low complexity, low action video sequence is shown. Shows peak data rates associated with transmission of high complexity, high action video sequences. Figure 3 illustrates a video compression technique used in one embodiment. Figure 3 illustrates a video compression technique used in one embodiment. Fig. 4 illustrates an additional video compression technique used in one embodiment. Fig. 4 illustrates a technique used in one embodiment to mitigate data rate peaks. Fig. 4 illustrates a technique used in one embodiment to mitigate data rate peaks. Fig. 4 illustrates a technique used in one embodiment to mitigate data rate peaks. Fig. 4 illustrates one embodiment for efficiently packing video tiles into packets. Fig. 4 illustrates one embodiment for efficiently packing video tiles into packets. Fig. 4 illustrates an embodiment using forward error correction techniques. Fig. 4 illustrates an embodiment using forward error correction techniques. Fig. 4 illustrates an embodiment using forward error correction techniques. Fig. 4 illustrates an embodiment using forward error correction techniques. 1 illustrates one embodiment illustrating a multi-core processing unit for compression. FIG. 4 illustrates communication between geographic positioning and hosting services according to embodiments. FIG. FIG. 4 illustrates communication between geographic positioning and hosting services according to embodiments. FIG. Fig. 6 illustrates the latency associated with communication between a client and a hosting service. The server center architecture of the hosting service is shown. FIG. 4 illustrates a screen shot of one embodiment of a user interface including multiple raw video windows. FIG. 17 shows the user interface of FIG. 16 after selecting a particular video window. Figure 18 shows the user interface of Figure 17 after zooming a particular video window to full screen size. 6 illustrates co-user video data overlaid on a multiplayer game screen. 3 illustrates a user page for a game player in a hosting service. Illustrate 3D interactive advertising. Figure 2 illustrates a series of steps for generating a photorealistic image with a textured surface from a live acting surface capture. Fig. 4 illustrates a user interface page that allows selection of linear media content. It is a graph which shows the time length which passes before a web page is raw with respect to a connection speed.

  In the following description, specific details are set forth such as device types, system configurations, communication methods, etc., in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details are not required to implement the embodiments described herein.

  2a-b shows that the user's home 211 ("user's home" is where the user is located) via the Internet 206 (or other public or private network) under contract services. FIG. 2 illustrates a high-level architecture of two embodiments as hosted by the hosting service 210 and accessed by the client device 205 (including outdoor when using mobile devices). Client device 205 may be a general purpose computer such as a Microsoft Windows or Linux-based PC or Apple Macintosh computer that has an internal or external display device 222 and is wired or wirelessly connected to the Internet, or video and audio. May be a dedicated client device, such as a set top box (wired or wirelessly connected to the Internet), or a mobile device that is wirelessly connected to the Internet.

  Any of these devices may have its own user input devices (eg, keyboard, buttons, touch screen, trackpad or inertial sensing rod, video capture camera and / or motion tracking camera, etc.). Alternatively, an external input device 221 (eg, keyboard, mouse, game controller, inertial sensing rod, video capture camera, and / or motion tracking camera, etc.) connected by wire or wireless may be used. As discussed in more detail below, the hosting service 210, including those with CPU / GPU processing power of high power, including a variety of performance levels server. While playing a game or using an application in the hosting service 210, the client device 205 for home or office receives keyboard and / or controller input from the user and then sends controller input to the hosting service 210 over the Internet 206. This hosting service 210 executes the game program in response and generates a continuous frame of video output (a series of video footage) for the game or application software (e.g., the user presses a button, when commanded to move the character on the screen to the right, the game program generates a sequence of video images showing the character moving to the right). This sequence of video images is then compressed using a video compressor low-latency, then the hosting service 210 transmits the video stream of low-latency through the Internet 206. Home or office client device then decodes the compressed video stream and renders the decompressed video images on a monitor or TV. As a result, the computing and graphics hardware requirements of client device 205 are significantly relaxed. The client 205 need only have the processing power to forward keyboard / controller input to the Internet 206 and to decode and decompress the compressed video stream received from the Internet 206, which is essentially Today, that CPU can run in software care (for example, Intel's Core Duo CPU running at approximately 2 GHz will run 720p HDTV encoded using compressors such as H.264 and Windows Media VC9. Can be thawed). And in the case of a client device, a dedicated chip also allows video decompression to such a standard in real time, at a much lower cost, and at a much lower power consumption than a general purpose CPU as required by modern PCs. Can be accomplished with In particular, to perform the functions of transferring controller input and decompressing video, the home client device 205 is a specialized graphics processing unit (GPU), optical drive or hard drive, such as the conventional one shown in FIG. Does not require a video game system.

  As games and application software become more complex and more photorealistic, they require high performance CPUs, GPUs, more RAM, and larger and faster disk drives, and hosting services 210's While computing power continues to upgrade, end users are not required to update the client platform 205 for home or office. This is because the processing requirements remain constant for the display resolution and frame rate for a given video decompression algorithm. Thus, the system shown in FIGS. 2a-b does not suffer from the hardware limitations and compatibility issues found today.

  In addition, since the game and application software runs only on the server of the hosting service 210, a copy of the game or application software (in the form of optical media or downloaded software) is not present in the user's home or office. None (“office” as used herein includes non-residential settings, including, for example, school classrooms unless otherwise indicated). This significantly reduces the risk of illegally copying (pirated) games or application software, as well as the risk of using valuable databases by pirated games or applications. In fact, special servers are required to play games or application software that are impractical for home or office use (eg, very expensive, large or noisy devices are required) Even if a pirated copy of the game or application software is obtained, it cannot operate at home or in the office.

  In one embodiment, the hosting service 210 provides software development tools to a game or application software developer 220 (generally referring to a software development company, a game or movie studio, or a game or application software publisher) 220 for this development. A person designs a video game to design a game that can be run on the hosting service 210. Such tools allow developers to take advantage of hosting service features not normally available on stand-alone PCs or video game consoles (eg, fast access to very large databases of complex geometries (“geometry "Study shape" here refers to polygons, textures, rigging, lighting, behavior, and other components and parameters that define a 3D database unless otherwise indicated)).

  Under this architecture, different business models are possible. Under one model, the hosting service 210 collects contract fees from end users and pays royalties to the developer 220 as shown in FIG. 2a. In another implementation shown in FIG. 2b, developer 220 collects subscription fees directly from the user and pays hosting service 210 to host the game or application content. These basic principles are not limited to a specific business model for providing online gaming or application hosting.

Compressed Video Characteristics As noted above, one significant problem with providing video game services or application software services online is latency. A waiting time of 70 to 80 ms (from the point where the input device is operated by the user to the point where the response is displayed on the display device) is the upper limit of games and applications that require a fast response time. However, this is very difficult to achieve due to the numerous practical and physical constraints in the architectural environment shown in FIGS. 2a and 2b.

  As shown in FIG. 3, when a user subscribes to Internet services, the connection is typically rated by the nominal maximum data rate 301 to the user's home or office. Based on the provider's policy and the capabilities of the routing device, its maximum data rate can be enforced slightly more strictly, but typically the actual data rate will be low for one of many different reasons. . For example, there may be a lot of network traffic in the DSL central office or local cable modem loop, or there may be noise on the cable causing the packets to be dropped, or the provider establishing the maximum number of bits / month / user . Currently, maximum downstream data rates for cable and DSL services typically range from several hundred kilobits per second (Kbps) to 30 Mbps. Cellular services are typically limited to hundreds of Kbps downstream data. However, the speed of broadband services and the number of users subscribing to broadband services increase rapidly with time. Currently, one analysis estimates that 33% of US broadband subscribers have downstream data rates of 2 Mbps or higher. For example, one analysis estimates that by 2010, over 85% of US broadband subscribers will have a data rate of 2 Mbps or higher.

  As shown in FIG. 3, the maximum data rate 302 actually available can vary over time. Thus, in low latency online games or application software contexts, it is sometimes difficult to predict the actual available data rate for a particular video stream. A data rate 303 is required to maintain a given level of quality in a given number of frames per second (fps) at a given resolution (eg, 640 × 480 @ 60 fps) for a certain amount of scene complexity. A number of problems can arise if the movement increases above the maximum data rate 302 that is actually available (as shown by the peaks in FIG. 3). For example, some Internet services will simply drop packets, causing data loss and video distortion / loss on the user's video screen. Other services temporarily buffer (ie queue) additional packets and provide packets to clients at available data rates, increasing latency, ie for many video games and applications. It will lead to unacceptable results. Eventually, some internet service providers see the increase in data rate as a malicious attack, eg, a denial of service attack (a well-known technique used by hackers to disable network connectivity), and Disconnect the user's Internet connection for a specified period. Accordingly, the embodiments described herein take steps to ensure that the data rate required for a video game does not exceed the maximum available data rate.

Hosting Service Architecture FIG. 4a illustrates the architecture of the hosting service 210 according to one embodiment. The hosting service 210 can be located in a single server center or can be distributed across multiple server centers (a low latency connection where the path to one server center has a lower latency than other server centers). To users, load balance among users, and provide redundancy when one or more server centers fail). The hosting service 210 ultimately includes hundreds, thousands, or millions of servers 402 that can serve a very large user base. The hosting service control system 401 provides overall control over the hosting service 210 and commands routers, servers, video compression systems, billing and accounting systems, and the like. In one embodiment, the hosting service control system 401 is embodied in a distributed processing Linux-based system coupled to a RAID array that is used to store a database for user information, server information, and system statistics information. In the above description, the various actions embodied by the hosting service 210 are initiated and controlled by the hosting service control system 401 unless it is due to another specific system.

  Hosting service 210 includes a number of servers 402 such as those currently available from Intel, IBM, Hewlett-Packard, and the like. Alternatively, server 402 can be assembled with a custom configuration of components, or ultimately integrated so that all servers are embodied as a single chip. This figure shows only a few servers 402 for clarity of illustration, but in an actual deployment there may be as few as one server 402 or there may be millions of servers 402 or more. Also good. Server 402 may all be configured similarly (as an example of some configuration parameters, with the same CPU type and performance, with or without a GPU, and with a GPU, With the same GPU type and performance, with the same number of CPUs and GPUs, and with the same amount and type / speed of RAM, and with the same RAM configuration), or various subsets of servers 402 have the same configuration (For example, 25% of the servers can be configured in one way, 50% can be configured in a different way, and 25% can be configured in another way) or each The server 402 may be different.

  In one embodiment, the server 402 is diskless, i.e. its own local mass storage (optical or magnetic storage, or semiconductor-based storage, e.g. flash memory or other mass storage means performing a similar function. Each server accesses a shared mass storage device through a high speed backplane or network connection. In one embodiment, this high-speed connection is a storage area network (SAN) 403 connected to a series of independent disk redundant arrays (RAID) 405, and the connection between the devices is implemented using Gigabit Ethernet. It becomes. As will be apparent to those skilled in the art, SAN 403 synthesizes multiple RAID arrays 405 together to ultimately produce a wide bandwidth, ie, the bandwidth available from the RAM used for current game consoles and PCs. Used to approach or potentially exceed it. RAID arrays based on rotating media such as magnetic media often have significant seek time access latencies, but RAID arrays based on semiconductor storage devices should be implemented with fairly short access latencies. Can do. In another configuration, some or all of the servers 402 locally provide some or all of their own mass storage. For example, the server 402 stores frequently accessed information, such as its operating system, and a copy of the video game or application in a low-latency local flash-based storage device, but with geometric or game state information. To access a large database from time to time, a SAN is used to access a rotating media based RAID array 405 with high seek latency.

  Further, in one embodiment, the hosting service 210 uses the low latency video compression logic 404 described in detail below. The video compression logic 404 can be implemented in software, hardware, or a combination thereof (some embodiments of which are described below). Video compression logic 404 includes logic for compressing audio and visual material.

  In operation, the control signal logic 413 of the client 415 is activated by the user while playing a video game or using an application in the user's home 211 via a keyboard, mouse, game controller or other input device 421. Send control signals 406a-b (typically in the form of UDP packets) representing button presses (and other forms of user input). Control signals from a given user are routed to the appropriate server 402 (or multiple servers if multiple servers respond to the user's input device). As shown in FIG. 4a, the control signal 406a is routed to the server 402 via the SAN. Alternatively or additionally, the control signal 406b is routed directly to the server 402 via a hosting service network (eg, an Ethernet-based local area network). Regardless of how they are transmitted, the server (s) execute the game or application software in response to the control signals 406a-b. Although not shown in FIG. 4a, various network components such as firewall (s) and / or gateway (s) may be connected to the edge of hosting service 210 (eg, hosting service 210 and Internet 410). And / or traffic at the edge of the user's home 211 between the Internet 410 and the home or office client 415 can be processed. The graphics and audio output of the executed game or application software, i.e., a new sequence of video images, is fed to the low latency video compression logic 404, which is a low latency video compression technique as described herein. And compressing the compressed video stream, typically with compressed or uncompressed audio, over the Internet 410 (or as described below, the general Internet To the client 415 (via an optimal high-speed network service that bypasses). The low latency video decompression logic 412 at the client 415 then decompresses the video and audio streams, renders the decompressed video stream, and typically plays the decompressed audio stream on the display device 422. To do. Alternatively, the audio may or may not be played on a separate speaker from the display device 422. Note that the input device 421 and the display device 422 are shown as separate devices in FIGS. 2a and 2b, but may be integrated into a client device such as a portable computer or mobile device.

  The home or office client 415 (previously described as the home or office client 205 in FIGS. 2a and 2b) is a very inexpensive and low power device with very limited computational or graphic performance and local mass. The storage device is very limited or not at all. In contrast, each server 402 coupled to the SAN 403 and multiple RAIDs 405 is a very high performance computing system, and in fact when multiple servers are used cooperatively in a parallel processing configuration. There is almost no limit to the amount of computation and graphics processing power that can be retained. Then, the computing power of server 402 is given to the user because of the low latency video compression 404 and the short latency video compression 412 perceptible to the user. When the user presses a button on the input device 421, the video on the display 422 responds to the button press with no perceptually significant delay as if the game or application software is running locally. And updated. Thus, a home or office client 415, which is a very low performance computer or inexpensive chip that embodies the low latency video decompression and control signal logic 413, can be any effective from a remote location that may be available locally. Is given to the user. This gives the user the power to play the most advanced processor intensive (typically new) video games and highest performance applications.

  FIG. 4 c shows a very basic and inexpensive home or office client device 465. This device is an embodiment of a home or office client 415 from FIGS. 4a and 4b. This is about 2 inches long. It has an Ethernet jack 462 that interfaces with an Ethernet cable over Power over Ethernet (PoE), from which power and connection to the Internet is obtained. NAT can be performed in a network that supports network address translation (NAT). In an office environment, many new Ethernet switches have PoE and take PoE directly to the office Ethernet jack. In such situations, only an Ethernet cable from the wall jack to the client 465 is required. If the available Ethernet connection does not carry power (eg in a home with DSL or cable modem but no PoE), an inexpensive wall “brick” that accepts an unpowered Ethernet cable and an output Ethernet with PoE. (Ie power supply) can be used.

  Client 465 includes control signal logic 413 (FIG. 4a) coupled to a Bluetooth wireless interface that interfaces with a Bluetooth input device 479 such as a keyboard, mouse, game controller and / or microphone and / or headset. One embodiment of client 465 also outputs video coupled to display device 468 capable of supporting 120 fps video at 120 fps and sends a signal to shuttered glasses 466 (typically via infrared) for each successive one. The shutter can be alternately actuated on one eye and then the other eye on the frame. The effect perceived by the user is a stereo 3D image that “jumps” the display screen. One such display device 468 that supports such operations is the Samsung HL-T5076S. Since each eye's video stream is separate, in one embodiment, two independent video streams are compressed by the hosting service 210, the frames are interleaved in time, and the frames are independent 2 in the client 465. Defrosted as one thawing process.

  Client 465 also has a low latency video decompression logic 412 that decompresses incoming video and audio and outputs it via HDMI (High Definition Multimedia Interface), and SDTV (Standard Definition Television) or HDTV (High Definition Sharpness). TV) 468 to provide video and audio to the TV or plug 463 to a monitor 468 that supports HDMI. If the user's monitor 468 does not support HDMI, HDMI to DVI (Digital Visual Interface) can be used, but audio is lost. Under the HDMI standard, display capabilities (eg, supported resolution, frame rate) 464 are communicated from the display device 468, and this information is then sent back to the hosting service 210 over the Internet connection 462, and thus the display The compressed video can be streamed in a format suitable for the device.

  FIG. 4d shows a home or office client device 475 that is the same as the home or office client device 465 shown in FIG. 4c except that it has more external interfaces. Client 475 can also accept PoE for power or can extend from an external power adapter (not shown) that plugs into the wall. Using the client 475's USB input, the video camera 477 provides the compressed video to the client 475, which is uploaded by the client 475 to the hosting service 210 and used as described below. Built in the camera 477 is a short-latency compressor using a compression technique described below.

  In addition to having an Ethernet connector as an Internet connection, the client 475 also has an 802.11g wireless interface to the Internet. Both interfaces can use NAT in a network that supports NAT.

  In addition to having an HDMI connector for outputting video and audio, the client 475 also has a dual link DVI-I connector that includes an analog output (and provides a VGA output with a standard adapter cable). It also has an analog output for composite video and S-video.

  For audio, the client 475 has left and right analog stereo RCA jacks and a TOSLINK output for digital audio output.

  In addition to the Bluetooth wireless interface to the input device 479, it also has a USB jack for interfacing with the input device.

  FIG. 4 e illustrates one embodiment of the internal architecture of client 465. All or some of the illustrated devices can be embodied in a field programmable logic array, custom ASIC, or a number of custom designed or off-the-shelf devices.

  An Ethernet 497 with PoE is attached to the Ethernet interface 481. Power 499 is derived from Ethernet 497 with PoE and connected to the remaining devices in client 465. The bus 480 is a common bus for communication between devices.

  A control CPU 483 that runs a small client control application from flash 476 (in most cases a small CPU such as a 100 MHz MIPS R4000 series CPU with embedded RAM) is sufficient for the network (ie Ethernet interface). Implements the protocol stack for, and communicates with the hosting service 210 and configures all devices within the client 465. It also handles the interface with the input device 469 and, if necessary, protects with “forward error correction” and sends the packet back to the hosting service 210 with the user controller data. Control CPU 483 also monitors packet traffic (eg, if packets are lost or delayed, also stamps their arrival). This information is returned to the hosting service 210 so that it can constantly monitor the network connection and adjust what is sent accordingly. The flash memory 476 is initially loaded with the control program of the control CPU 483 at the time of manufacture, as well as a serial number unique to a particular client 465 unit. This serial number allows the hosting service 210 to uniquely identify the client 465 unit.

  The Bluetooth interface 484 communicates wirelessly to the input device 469 through its antenna internal to the client 465.

  Video decompressor 486 is a low latency video decompressor configured to implement the video decompression described herein. A large number of video decompressors exist as off-the-shelf or intelligent property (IP) designs that can be integrated into FPGAs or custom ASICs. H. One company that provides IP for H.264 decoders is Ocean Logic of Manly of NSW Australia. The effect of using IP is that the compression technique used here does not follow the compression standard. Some standard decompressors are flexible enough to be configured to accept the compression techniques described herein, while others are not. However, IP offers complete flexibility in redesigning decompressors as needed.

  The output of the video decompressor is coupled to the video output subsystem 487, which couples the video to the video output of the HDMI interface 490.

  Audio decompression subsystem 488 can be implemented using standard audio decompressors available or can be implemented as IP, or audio decompression can implement, for example, a Vorbis audio decompressor. It can also be embodied in a control processor 483 that can.

  An apparatus that implements audio decompression is coupled to an audio output subsystem 489 that couples audio to the audio output of the HDMI interface 490.

  FIG. 4 f illustrates one embodiment of the internal architecture of client 475. As will be apparent, this architecture is the same as the client 465 except for the additional interface and any external DC power from the wall power adapter, which external DC power is used as such. The power coming from the Ethernet PoE 497 is replaced. Functions common to the client 465 will not be described below, and additional functions will be described below.

  The CPU 483 communicates with and configures additional devices.

  WiFi subsystem 482 provides wireless Internet access as an alternative to Ethernet 497 through its antenna. The WiFi subsystem is available from a wide range of manufacturers, including Atheros Communications of Santa Clara, California.

  USB subsystem 485 provides an alternative to Bluetooth communication for wired USB input device 479. USB subsystem, a fairly standard, readily available for FPGA and ASIC, as with and video decompression, often built in performing off-the-shelf devices other functions.

  Video output subsystem 487 generates a wider range of video output than in client 465. This provides DVI-I 491, S-video 492 and composite video 493 in addition to providing HDMI 490 video output. Also, when the DVI-I 491 interface is used for digital video, the display capability 464 is returned from the display device to the control CPU 483 so that the hosting service 210 can be notified of the capabilities of the display device 478. All the interfaces provided by the video output subsystem 487 are very standard interfaces and are readily available in numerous forms.

  Audio output subsystem 489 outputs audio digitally through digital interface 494 (S / PDIF and / or Toslink) and outputs audio in analog form through stereo analog interface 495.

Round Trip Latency Analysis Of course, to understand the above paragraph, the round trip latency between the user's action using the input device 421 and viewing the result of that action on the display device 420 is 70-80 ms. Must be: This waiting time must consider all factors in the path from the input device 421 in the user's house 211 to the hosting service 210 and back to the user's house 211 to the display device 422. FIG. 4b shows the various components and networks through which the signal must travel, and above these components and networks are timelines that list the latency that can be expected in an actual implementation. Note that FIG. 4b has been simplified to show only important path routing. Other routing of data used for other features of the system is described below. Double-headed arrows (eg, arrow 453) represent round trip latency, single-headed arrows (eg, arrow 457) represent one-way latency, and “˜” represents an approximate measure. There are real-world situations where the listed latency cannot be achieved, but in many cases in the United States, DSL and cable modem connections to the user's home 211 are used to achieve these latency in the environment described in the next paragraph. I must point out what I can do. Also, cellular wireless connections to the Internet will work reliably in the system shown, but most current US cellular data systems (such as EVDO) suffer from very long latencies and the wait shown in FIG. 4b. Note also that time cannot be achieved. However, these basic principles could be implemented in future cellular technologies that can implement this level of latency.

  Starting from the input device 421 in the user's house 211, when the user operates the input device 421, a user control signal is transmitted to the client 415 (this may be a stand-alone device such as a set-top box or a PC Or may be software or hardware running on another device, such as a mobile device), and packetized (in one embodiment in UDP format) with the destination address for arriving at the hosting service 210 Is given. The packet also includes information indicating from which user the control signal comes. The control signal packet (s) are then forwarded through the firewall / router / NAT (Network Address Translation) device 443 to the WAN interface 442. The WAN interface 442 is an interface device provided to the user's house 211 by the user's ISP (Internet Service Provider). The WAN interface 442 may be a cable or DSL modem, WiMax transceiver, fiber transceiver, cellular data interface, internet protocol over power line interface, or others of multiple interfaces to the internet. Further, the firewall / router / NAT device 443 (and potentially the WAN interface 442) may be integrated into the client 415. One example is a mobile phone that includes software for implementing the functionality of a home or office client 415 and means for wirelessly routing and connecting to the Internet according to a standard (eg, 802.11g).

  The WAN interface 442 then routes the control signal to what is referred to as a “point of presence” 441 for the user's Internet service provider (ISP), which is connected to the user's home 211. A facility that provides an interface between a WAN transport and a general Internet or private network. The nature of the existence point varies based on the nature of the Internet service provided. In the case of DSL, this will typically be the central office of the telephone company where the DSLAM is located. In the case of a cable modem, this is typically a cable multisystem operator (MSO) headend. In the case of a cellular system, this will typically be the control room associated with the cellular tower. However, whatever the nature of the point of presence, this routes the control signal packet (s) to the general Internet 410. The control signal packet (s) is then routed to the WAN interface 441 to the hosting service 210 through what is most often a fiber transceiver interface. WAN 441 then routes the control signal packet to routing logic 409 (which is implemented in a number of different ways, including Ethernet switches and routing servers), which evaluates the user's address, and Route control signals to the correct server 402 for a given user.

  Server 402 then takes the control signal as input to the game or application software running on server 402 and uses that control signal to process the next frame of the game or application. When the next frame is generated, video and audio are output from the server 402 to the video compressor 404. Video and audio are output from the server 402 to the compressor 404 via various means. First, the compressor 404 can be incorporated into the server 402, and thus compression can be implemented locally within the server 402. Alternatively, video and / or audio is in packetized form, via a network connection such as an Ethernet connection, to a network that is a private network between the server 402 and the video compressor 404, or through a shared network such as the SAN 403. Can be output. Alternatively, the video may be output from server 402 through a video output connector, such as a DVI or VGA connector, and then captured by video compressor 404. Audio may also be output from the server 402 as digital audio (eg, via a TOSLINK or S / PDIF connector) or as analog audio, and the analog audio may be output by audio compression logic within the video compressor 404. Digitized and encoded.

  When video compressor 404 captures a video frame and the audio generated during that frame time from server 402, it compresses the video and audio using the techniques described below. Once video and audio are compressed, they are packetized with an address, sent back to the user's client 415 and routed to the WAN interface 441, which routes video and audio packets over the general Internet 410. The Internet routes video and audio packets to the user's ISP presence point 441, which routes the video and audio packets to the user's home WAN interface 442, which interface the video and audio packets. Route to firewall / router / NAT device 443, which then routes video and audio packets to client 415

  The client 415 decompresses the video and audio, then displays the video on the display device 422 (or the client's built-in display device) and the audio to the display device 422 or a separate amplifier / speaker or an amplifier built into the client. / Send to speaker.

  In order for the user to recognize that there is no perceptual delay in the entire process described above, the round trip delay needs to be less than 70 or 80 ms. Some of the latency delays in the round trip path are under the control of the hosting service 210 and / or the user, others are not. Nevertheless, based on the analysis and testing of a large number of real-world scenarios, the approximate measurements are:

  The one-way transmission time for transmitting the control signal 451 is typically less than 1 ms, and round trip routing through the user's house 452 is typically consumer grade firewall / router / accessible over Ethernet. This is achieved in about 1 ms using a NAT switch. User ISPs vary widely in their round trip delay 453, but for DSL and cable modem providers it is typically 10-25 ms. The round trip latency in the general Internet 410 varies greatly based on how traffic is routed and whether the route is flawed (these issues are discussed below), but typically The typical Internet provides a very optimal route, and latency is determined primarily by the speed of light through the optical fiber given the distance to the destination. As further described below, we have established 1000 miles as the furthest approximate distance that is expected to leave the hosting service 210 away from the user's home 211. At 1000 miles (2000 miles round trip), the actual transit time of signals through the Internet is about 22 ms. The WAN interface 441 to the hosting service 210 is typically a commercial grade fiber high speed interface with negligible latency. Thus, typical Internet latency 454 is typically 1 to 10 ms. The latency of the one-way routing 455 through the hosting service 210 is less than 1 ms. Server 402 typically calculates a new frame for a game or application in less than one frame time (16.7 ms at 60 fps), so the maximum one-way latency that is reasonable to use is 16 ms. With the optimal hardware implementation of the video and audio compression algorithms described herein, compression 457 can be completed in 1 ms. The less optimal form takes about 6 ms to compress (of course, the less optimal form takes longer, but such an implementation affects the overall round trip latency and waits for 70-80 ms. Other latency needs to be reduced to maintain the time goal (e.g., an acceptable distance through the general Internet can be reduced). The round trip latency of the Internet 454, user ISP 453, and user house routing 452 has already been taken into account, so the rest is the latency of video decompression 458, which is implemented by dedicated hardware for video decompression 458. Depending on whether it is implemented in software on a client device 415 (such as a PC or mobile device) and based on the size of the display and the performance of the decompression CPU. Typically, decompression 458 takes 1 to 8 ms.

  Therefore, by adding all the worst case latency actually seen, the worst case round trip latency that can be expected to be experienced by the user of the system shown in FIG. 4a can be determined. They are 1 + 1 + 25 + 22 + 1 + 16 + 6 + 8 = 80 ms. And indeed (with the note below), this uses a prototype type version of the system shown in FIG. 4a, with an off-the-shelf Windows PC as a client device and a domestic DSL and cable modem in the United States. The approximate round trip latency found using the connection. Of course, in a scenario better than the worst case, a very short latency is obtained, but it cannot depend on the development of widely used commercial services.

  To obtain the latency listed in FIG. 4b over the general Internet, the video compressor 404 of FIG. 4a and the video decompressor 412 of the client 415 generate a packet stream with very specific characteristics and the hosting service 210 Packet sequence generated over the entire path from the device to the display device 422 is not subject to delay or excessive packet loss and is used by the user via the user's Internet connection through the WAN interface 442 and the firewall / router / NAT 443 in particular. It must be consistently within the bandwidth constraints that it can. In addition, the video compressor must generate a packet stream that is robust enough to allow the inevitable packet loss and packet reordering that occurs in normal Internet and network transmissions.

Low Latency Video Compression To achieve the goal, one embodiment takes a novel video compression solution that reduces latency and relaxes peak bandwidth requirements for transmitting video. Before describing these embodiments, an analysis of current video compression techniques is performed with reference to FIGS. 5 and 6a-b. Of course, these techniques can be used on a basic basis if the user is given sufficient bandwidth to handle the data rates required by these techniques. Note that audio compression is not addressed here other than that it is implemented simultaneously and synchronously with video compression. There are conventional audio compression techniques that meet the requirements of this system.

  FIG. 5 is one specific prior art for compressing video, where each individual video frame 501-503 is compressed by compression logic 520 using a specific compression algorithm to produce a series of compressed frames. 2 illustrates the prior art for generating 511-513. One embodiment of this technique is “motion JPEG”, where each frame is compressed according to a joint picture expert group (JPEG) compression algorithm, based on a discrete cosine transform (DCT). . A variety of different types of compression algorithms may be used, but remain adapted to these basic principles (eg, wavelet-based compression algorithms such as JPEG-2000).

  One problem with this type of compression is that it reduces the data rate of each frame, but does not use the similarity between successive frames to reduce the data rate of the entire video stream. For example, assuming a frame rate of 640 * 480 * 24 bits / pixel = 640 * 480 * 24/8/1024 = 900 kilobytes / frame (KB / frame) for a given quality of video, as shown in FIG. Motion JPEG only compresses the stream by a factor of 10 and produces a 90 KB / frame data stream. At 60 frames / second, this requires a channel bandwidth of 90 KB * 8 bits * 60 frames / second = 42.2 Mbps, which is much more for almost all home Internet connections in the United States today It has a wide bandwidth and a very wide bandwidth for a large number of office Internet connections. In fact, if a constant data stream is required with such a wide bandwidth and only one user is useful in an office LAN environment, a large percentage of the 100 Mbps Ethernet LAN bandwidth is consumed. This puts a heavy burden on Ethernet switches that support. Therefore, compression for moving video is inefficient when compared to other compression techniques (as described below). In addition, single-frame compression algorithms such as JPEG and JPEG-2000 that use the lossy compression algorithm can cause compression flaws that are not noticeable in still images (eg, flaws in dense leaves in a scene are It may not be visible as a defect because it is not exactly known to the eye what it should look like. However, when the scene moves, the defect becomes noticeable because it is detected by the eye that the defect changes from frame to frame, even though the defect is in the area of the scene that is not noticed by still images. As a result, “background noise” that is similar in appearance to “snow” noise seen during marginal analog TV reception is perceived in the sequence of frames. Of course, this form of compression can still be used in some of the embodiments described herein, but generally speaking, high data for a given perceptual quality to avoid background noise in the scene. A rate (ie, a low compression ratio) is required.

  H. H.264, or other types of compression, such as Windows Media VC9, MPEG2 and MPEG4, all take advantage of the similarity between successive frames, so the compression of the video stream is more efficient. All of these techniques rely on the same general technique for compressing video. Therefore, H.I. Although the H.264 standard is described, the same general principles apply to various other compression algorithms. Compression H. X264 open source software library for H.264 and decompression H.264 A large number of H.264 including the FFmpeg open source software library for H.264 H.264 compressors and decompressors can be used.

  FIGS. 6a and 6b illustrate a conventional compression technique in which a series of uncompressed video frames 501-503, 559-561 are converted by a compression logic 620 into a series of “I frames” 611, 671, “P frames. "612-613" and "B frame" 670. The vertical axis in FIG. 6a generally represents the size of each frame that is obtained by encoding (but the frame is not drawn to an appropriate scale). As mentioned above, video coding using I-frames, B-frames and P-frames is well known to those skilled in the art. Briefly, I frame 611 is DCT-based compression of fully uncompressed frame 501 (similar to the compressed JPEG video described above). The P frame 612-613 is generally significantly smaller in size than the I frame 611. This is because the advantage of the data of the previous I frame or P frame is taken in, that is, the data indicating the change between the previous I frame or P frame is included. B frame 670 is similar to P frame, but B frame uses a frame in a subsequent reference frame and a potential frame in a preceding reference frame.

  In the following description, it is assumed that the desired frame rate is 60 frames / second, each I frame is about 160 Kb, the average P and B frames are 16 Kb, and a new I frame is generated every second. With this set of parameters, the average data rate is 160 Kb + 16 Kb * 59 = 1.1 Mbps. This data rate is well within the maximum data rate for many current broadband Internet connections to homes and offices. This technique also tends to avoid the background noise problem from intraframe dedicated encoding. This is because P and B frames track the difference between frames and compression defects do not tend to appear or disappear from frame to frame, alleviating the background noise problem.

  One problem with this type of compression is that the average data rate is relatively low (eg, 1.1 Mbps), but a single I-frame takes many frames to transmit. For example, using the prior art, to stream video at 1.1 Mbps with 160 Kbps I frames every 60 frames, typically a 2.2 Mbps network connection (eg, 2.2 Mbps peak from FIG. 3a). DSL or cable modem) with a maximum available data rate 302 is sufficient. This is accomplished by keeping the 1 second video in the decompression queue until the video is decompressed. In 1 second, 1.1 Mb of data is transmitted, which is easily accepted by the maximum available data rate of 2.2 Mbps, even assuming that the available data rate drops periodically by about 50%. It is done. Unfortunately, this conventional solution results in a 1 second latency for the video because there is a 1 second video buffer at the receiver. Such a delay is sufficient for many conventional applications (eg, linear video playback), but with much longer latency for fast action video games that cannot tolerate a latency greater than 70-80 ms. is there.

  Attempts have been made to eliminate the 1 second video buffer, but not enough latency reduction for fast action video games. As an example, the use of the B frame described above requires reception of all B and I frames preceding the I frame. Assuming that 59 non-I frames are roughly divided between P and B frames, there are at least 29 B frames, and I frames are received before the B frames can be displayed. Therefore, regardless of the available bandwidth of the channel, a delay of 29 + 1 = 30 frames, each 1/60 second wide, ie a waiting time of 500 ms is required. Obviously this is far too long.

  Therefore, another solution is to eliminate B frames and use only I and P frames. (One result is an increase in data rate for a given quality level, but for consistency in this example, each I frame is 160 Kb and the average P frame is 16 Kb in size. Yes, and therefore continue to assume that the data rate is still 1.1 Mbps.) This solution eliminates the inevitable latency introduced by B-frames. This is because the decoding of each P frame only depends on the previously received frame. The problem with this solution is that because I frames are much larger than average P frames, transmission of I frames has substantial latency in narrow bandwidth channels that are typical in most homes and many offices. It is to increase. This is shown in FIG. 6b. Except for I frames where the peak data rate 623 required for an I frame far exceeds the maximum available data rate 622 (and also the rated maximum data rate 621), the data rate 624 of the video stream is the maximum available. Lower than data rate 621. The data rate required by the P frame is less than the maximum available data rate. Even if the peak of the maximum available data rate of 2.2 Mbps is kept steadily at the peak rate of 2.2 Mbps, it takes 160 Kb / 2.2 Mb = 71 ms to transmit an I-frame and can be used If the maximum data rate 622 drops by 50% (1.1 Mbps), 142 ms is required to transmit the I frame. Therefore, the waiting time for transmitting an I frame is somewhere between 71 and 142 ms. This waiting time is added to the waiting time shown in FIG. 4b, and in the worst case this is added to 70 ms, so that it appears on the display device 422 from the moment the user operates the input device 421. Up to 141 to 222 ms total round trip latency, which is much higher. And when the maximum available data rate falls below 2.2 Mbps, the latency increases further.

  Note also that generally “jamming” the ISP at the peak data rate 623 will produce severe results that far exceed the available data rate 622. Different ISP devices behave differently, but when receiving packets at a data rate much higher than the available data rate 622, the following behavior is quite common between DSL and cable modem ISPs. (A) delay the packet by queuing the packet (introducing latency), (b) drop some or all of the packet, and (c) disable the connection for a period of time (probably related to the ISP It is a malicious attack such as a “denial of service” attack). Therefore, transmitting a packet stream at all data rates with the characteristics shown in FIG. 6b is not a feasible option. Peak 623 may be queued at hosting service 210 and transmitted at a data rate lower than the maximum available data rate, introducing the unacceptable latency described above.

  Furthermore, the video stream data rate sequence 624 shown in FIG. 6b is a very “compliant” video stream data rate sequence and is expected to result from compressing video from a video sequence that does not change significantly and has little motion. Data rate sequences of the type (e.g. common in video teleconferencing where the camera is in a fixed position and hardly moves, and objects in the scene, for example a person sitting in a chair and showing little movement) To be appropriate).

  The video stream data rate sequence 634 shown in FIG. 6c is a typical sequence for what would be expected to be seen from a video with much more action, such as generated in a video or video game or some application software. . Note that in addition to the I frame peak 633, there are also P frame peaks such as 635 and 636 that are very large and often exceed the maximum available data rate. These P frame peaks are not as large as I frame peaks, but are much too large to be carried by the channel at all data rates, and, like I frame peaks, P frames The peak must be transmitted slowly (thus increasing latency).

  In a high bandwidth channel (eg, a 100 Mbps LAN, or a wide bandwidth 100 Mbps private connection), the network can tolerate large peaks, such as I frame peak 633 or P frame peak 636, and in principle: A short waiting time can be maintained. However, such networks are often shared among a large number of users (eg, in an office environment), and such “peak” data is especially relevant when network traffic is routed to a private shared connection. (E.g., from a remote data center to an office), affecting LAN performance. First, it should be noted that this example is a relatively low resolution video stream of 640 × 480 pixels at 60 fps. 1920 x 1080 HDTV streams at 60 fps are easily handled by modern computers and displays, and 2560 x 1440 resolution displays are becoming increasingly available at 60 fps (eg, Apple's 30 "display) and 60 fps. A 1920x1080 high-action video sequence requires 4.5 Mbps using H.264 compression for a reasonable quality level, assuming an I-frame peak of 10 times the nominal data rate, which is 45 Mbps or less Although there are still significant P frame peaks, many users have the same 100 Mbps network (eg, private network connection between office and data center) When receiving a video stream via a network, how the peaks from the video stream of a large number of users align, overwhelm the network bandwidth, and reduce the bandwidth of the switch backplane that supports users on the network. It's easy to see if it's potentially overwhelming, even with Gigabit Ethernet networks, if enough users align enough peaks at once, they can overwhelm the network or network switch, and 2560x1440. As resolution video becomes more commonplace, the average video stream data rate will be 9.5 Mbps, possibly resulting in a peak data rate of 95 Mbps, of course, a 100 Mbps connection between the data center and the office (this is Today, which is a very fast connection) is completely sunk by peak traffic from a single user, so streaming with high peaks even though LAN and private network connections can tolerate more peak streaming video. Video is undesirable and requires special planning and adaptation by the office IT department.

  Of course, for standard linear video applications, these matters are not a problem. This is because the data rate is “smoothed” at the transmission point, the data for each frame is below the maximum available data rate 622, and the client's buffer decompresses the sequence of I, P and B frames. It is because it memorizes until it is done. Thus, the data rate across the network is kept close to the average data rate of the video stream. Unfortunately, this introduces latency even when B-frames are not used, ie it is unacceptable for short latency applications such as video games and applications that require fast response time.

  One conventional solution to mitigate video streams with high peaks is to use a technique often referred to as “constant bit rate” (CBR) encoding. The term CBR seems to mean that all frames are compressed to have the same bit rate (ie size), but it usually refers to a certain number of frames (here one frame) This is a compression paradigm where the maximum bit rate is allowed. For example, in the case of FIG. 6c, the compression algorithm uses 70% or more of the rated maximum data rate 621 when the CBR constraint is applied to an encoding that limits the bit rate to, for example, 70% of the rated maximum data rate 621. Thus, the compression of each frame is limited so that the frames that are normally compressed are compressed with a small number of bits. As a result, frames that require more bits than usual to maintain a given quality level result in a bit “deficiency” and the video quality of these frames requires more than 70% of the rated maximum data rate 621. Not worse than for other frames. This solution results in acceptable results for some types of compressed video where (a) there is little expected motion or scene change and (b) the user can accept periodic quality degradation. Can be generated. A good example of an application suitable for CBR is video teleconferencing. This means that if there are only a few peaks and the quality drops for a short time (eg, the camera pans to produce significant scene movement and large peaks, during which it is sufficient for high quality video compression. This is because most users will accept it if there is no bit and the video quality is degraded. Unfortunately, CBR is not well suited for many other applications that have very complex or many moving scenes and / or where a moderate level of quality is required.

  The low latency compression logic 404 used in one embodiment uses a number of different techniques to address a range of problems associated with streaming low latency compressed video while maintaining high quality. . First, the low latency compression logic 404 only generates I and P frames, alleviating the need to wait multiple frames of time to decode each B frame. Further, as shown in FIG. 7a, in one embodiment, the low latency compression logic 404 subdivides each uncompressed frame 701-760 into a series of “tiles” and each tile is an I or P frame. Encode individually as either. The group of compressed I and P frames is referred to herein as “R frames” 711-770. In the particular example shown in FIG. 7a, each uncompressed frame is subdivided into a 16 tile 4 × 4 matrix. However, these basic principles are not limited to specific subdivision schemes.

In one embodiment, the low latency compression logic 404 divides the video frame into a number of tiles and encodes (ie, compresses) one tile from each frame as an I frame (ie, the tile is full). Compressed as if it were 1/16 individual video frames of video size, the compression used for this “mini” frame is I frame compression), and the remaining tiles are encoded as P frames (ie, (Ie, the compression used for each 1/16 “mini” frame is P-frame compression). The tiles compressed as I and P frames are referred to as “I tile” and “P tile”, respectively. In each successive video frame, the tile to be encoded as an I tile is changed. Thus, for a given frame time, only one of the tiles in the video frame is an I tile and the remaining tiles are P tiles. For example, in FIG. 7 a, tile 0 of uncompressed frame 701 is encoded as I tile I ₀ and the remaining 1-15 tiles are encoded as P tiles P ₁ -P ₁₅ to form R frame 711. In the next uncompressed video frame 702, tile 1 of uncompressed frame 701 is encoded as I tile I ₁ and the remaining tiles 0 and 2 to 15 are encoded as P tiles P ₀ and P ₂ to P ₁₅ , R A frame 712 is formed. Accordingly, the tiles I and P tiles are interleaved one after another over successive frames. This process continues until the last tile in the matrix is encoded as an I tile (ie, I ₁₅ ) and an R tile 770 is generated. The process is then resumed to generate another R frame, such as frame 711 (ie, encoding an I tile for tile 0, etc.). Although not shown in FIG. 7a, in one embodiment, the first R frame of a video sequence of R frames contains only I tiles (ie, subsequent P frames are the basis for calculating motion). To have video data). Alternatively, in one embodiment, the startup sequence uses the same I tile pattern as normal, but does not include P tiles for tiles that are not yet encoded with I tiles. In other words, some tiles are not encoded with data until the first I tile arrives, thus avoiding the startup peak at the video stream data rate 934 of FIG. 9a, described in detail below. Further, as described below, a variety of different sizes and shapes can be used for the tiles while still conforming to these basic principles.

Video decompression logic 412 executed at client 415 decompresses each tile as if it were a separate video sequence of small I and P frames, and then renders each tile to frame buffer drive display device 422. For example, I ₀ and P ₀ from R frames 711 to 770 are used to decompress and render tile 0 of the video footage. Similarly, tile 1 is reconstructed using I ₁ and P ₁ from R frames 711-770, and so on. As described above, I-frame and P-frame decompression is a well-known technique, and I-tile and P-tile decompression is accomplished by obtaining multiple instances of video decompression performed by client 415. Can do. Although the multiplication process appears to increase the computational burden on the client 415, in practice it is not. This is because the tile itself is proportionally smaller with respect to the number of additional processes, so the number of pixels displayed is one process and uses conventional full size I and P frames. Because it is the same as the case.

  This R frame technique significantly reduces the bandwidth peaks typically associated with the I frames shown in FIGS. 6b and 6c. This is because a given frame is usually made up of P frames that are typically smaller than I frames. For example, assuming again that a typical I frame is 160 Kb, the I tile of each frame shown in FIG. 7 a will be approximately 1/16 of this amount, or 10 Kb. Similarly, assuming that a typical P frame is 16 Kb, the P frame of each tile shown in FIG. 7 a is approximately 1 Kb. The final result is an R frame of approximately 10 Kb + 15 * 1 Kb = 25 Kb. Therefore, each 60 frame sequence is 25 Kb * 60 = 1.5 Mbps. Thus, at 60 frames / second, this requires a channel that can maintain a bandwidth of 1.5 Mbps, but fairly low peaks for I tiles are distributed throughout the 60 frame interval.

  Note that in the previous example assuming the same data rate for I and P frames, the average data rate was 1.1 Mbps. This is because, in the previous example, a new I frame is introduced only once every 60 frames of time, whereas in this example, an I frame cycle is created in a time of 16 frames and thus a time equal to the I frame. This is because tiles are introduced every 16 frames, resulting in a significantly higher average data rate. In fact, even more frequently introducing I frames does not increase the data rate linearly. This is due to the fact that the P frame (or P tile) encodes the difference from the previous frame to the next frame. Thus, if the previous frame is quite similar to the next frame, the P frame will be very small, while if the previous frame is quite different from the next frame, the P frame will be very growing. However, since the P frame is derived primarily from the previous frame, not the actual frame, the resulting encoded frame contains a larger number of bits (eg, visual defects) than the I frame. be able to. Error accumulation occurs when one P frame is followed by another P frame, and worsens when there is a long sequence of P frames. Here, an elaborate video compressor detects that the video quality is degraded after a series of P frames and allocates more bits to its subsequent P frames as needed to increase the quality, or If it is the most efficient course of action, replace the P frame with an I frame. Thus, when a long sequence of P-frames is used (eg, 59 P-frames as in the example above), typically when the scene has a great deal of complexity and / or motion, As is further away from the I frame, more bits are needed in the P frame.

  Alternatively, when viewing the P frame from the opposite perspective, the P frame that follows the I frame tends to require fewer bits than the P frame that is further away from the I frame. Accordingly, in the example shown in FIG. 7a, there is no P frame that is 15 frames or more away from the preceding I frame, while in the previous example, the P frame can be 59 frames away from the I frame. Therefore, the more frequently there are I frames, the smaller the P frames. Of course, the exact relative size will vary based on the nature of the video stream, but in the example of FIG. 7a, if the I tile is 10Kb, the P tile will average only 0.75Kb in size and 10Kb + 15 * 0.75Kb = 21.25Kb, or at 60 frames / second, the data rate is 21.25Kb * 60 = 1.3Mbps, or 1.1Mbps with 59 P frames followed by I frames The data rate is about 16% higher than the stream. Again, the relative results between these two solutions for video compression will vary based on the video sequence, but typically using R frames is more than using I / P frame sequences. Experiments have shown that it requires about 20% more bits for a given quality level. However, of course, R frames dramatically reduce peaks and enable video sequences with much lower latency than I / P frame sequences.

  R frames can be configured in a variety of different ways, depending on the nature of the video sequence, channel reliability, and available data rates. In another embodiment, a different number of tiles than 16 in a 4x4 configuration is used. For example, 2 tiles can be used in a 2x1 or 1x2 configuration, 4 tiles can be used in a 2x2, 4x1 or 1x4 configuration, and 6 tiles can be used in a 3x2, 2x3, 6x1 or 1x6 configuration. Eight tiles can be used in a 4x2 (as shown in Fig. 7b), 2x4, 8x1 or 1x8 configuration. Note that the tiles do not need to be square and the video frames need not be square or rectangular. The tiles can be divided into shapes that are most suitable for the video frame and application to be used.

  In another embodiment, the cycle of I and P tiles is not fixed to the number of tiles. For example, in an 8 tile 4x2 configuration, a 16 cycle sequence can still be used as shown in Figure 7b. The sequential uncompressed frames 721, 722, 723 are each divided into 8 tiles, 0-7, and each tile is individually compressed. From R frame 731, only tile 0 is compressed as an I tile, and the remaining tiles are compressed as P tiles. For subsequent R frame 732, all 8 tiles are compressed as P tiles, then for subsequent R frame 733, tile 1 is compressed as an I tile and all other tiles are compressed as P tiles. The Thus, the sequence continues for 16 frames, and I tiles are generated only for every other frame, and the last I tile is generated for tile 7 during the 15th frame time (not shown in FIG. 7b). ), And during the 16th frame time, the R frame 780 is compressed using all P tiles. The sequence is then resumed, with tile 0 being compressed as an I tile and the other tiles being compressed as P tiles. Similar to the previous embodiment, each first frame of the entire video sequence is typically all I tiles, giving a reference of P tiles forward from that point. The cycle of I and P tiles need not be an even multiple of the number of tiles. For example, with eight tiles, each frame with an I tile is followed by two frames, all with P tiles, followed by another I tile. In yet another embodiment, some tiles have been found to have more movement and require more frequent I tiles, for example, while other areas are more static (e.g., If few frequent I tiles are required (indicating game scores), the I tile is sequenced more frequently than the other tiles. Further, although each frame is shown with a single I tile in FIGS. 7a-b, multiple I tiles may be encoded within a single frame (based on the transmission channel bandwidth). Conversely, a frame or frame sequence can be transmitted without I tiles (ie, only with P tiles).

  The reason why the solution described in the previous paragraph works well is that the distribution of I tiles over each single frame seems to cause a big peak, but the behavior of the system is not that simple. is there. Each tile is compressed separately from the other tiles, so the smaller the tile, the less efficient the encoding of each tile. This is because the compressor for a given tile cannot take advantage of similar video features and similar motion from other tiles. Therefore, dividing the screen into 16 tiles generally results in lower encoding efficiency than dividing the screen into 8 tiles. However, if the screen is divided into 8 tiles and all I-frame data is introduced every 8 frames instead of every 16 frames, the overall data rate is very high. Thus, by introducing all I frames every 16 frames instead of every 8 frames, the overall data rate is reduced. Also, by using 8 large tiles instead of 16 small tiles, the overall data rate is reduced, reducing the data peaks caused by the large tiles to some extent.

  In another embodiment, the low latency video compression logic 404 of FIGS. 7a and 7b preconfigures the allocation of bits to various tiles in an R frame with settings based on known characteristics of the video sequence to be compressed. Or automatically based on ongoing analysis of video quality in each tile. For example, in a competitive video game, the front of the player's car (which has relatively little movement in the scene) occupies most of the lower half of the scene, while the upper half of the scene is almost always moving. Completely filled with roads, buildings and landscapes. If compression logic 404 assigns an equal number of bits to each tile, the lower half tile (tile 4-7) of the uncompressed frame 721 screen of FIG. 7b is typically the uncompressed frame 721 of FIG. The screen is compressed with higher quality than the upper half tiles (tiles 0-3). If this particular game or this particular scene of the game is known to have such characteristics, the hosting service 210 operator allocates more bits to the top tile of the screen than to the bottom tile of the screen. The compression logic 404 can be configured as follows. Alternatively, the compression logic 404 can evaluate the quality of tile compression after the frame is compressed (using one or more of a number of compression quality metrics such as peak signal-to-noise ratio (PSNR)). And gradually determine that a tile will consistently produce a high quality result over a time window until the various tiles reach a similar quality level and gradually give more bits to the tile that produces a lower quality result. Assign to. In another embodiment, compression logic 404 assigns bits to obtain a high quality for a particular tile or group of tiles. For example, an overall good perceptual appearance can be given so that the quality of the central part is higher than the edge of the screen.

  In one embodiment, to improve the resolution of certain areas of the video stream, video compression logic 404 may use areas of the video screen with relatively large scene complexity and / or motion as scene complexity and / or motion. Encode using tiles that are smaller than a relatively small video screen area. For example, as shown in FIG. 8, small tiles are used around a moving character 805 in one area of one R frame 811 (potentially a series of R frames with the same tile size ( (Not shown))). Then, when the character 805 moves to a new area of the video, small tiles are used around this new area in another R frame 812 as shown. As mentioned above, a variety of different sizes and shapes can be used as “tiles” while still meeting these basic principles.

  The cyclic I / P tiles described above substantially reduce the peak data rate of the video stream, but especially for rapidly changing or very complex video footage that occurs in video, video games and certain application software. In some cases, the peaks cannot be completely eliminated. For example, during a sudden scene transition, a complex frame may be followed by another completely different complex frame. Even if a large number of I tiles may precede the scene transition by some frame time, they do not help in this situation. This is because the new frame material is not related to the previous I tile. In this state (and other states where many videos change even if everything has changed), video compressor 404 can code more efficiently as I tiles, if not all of the P tiles. As a result, it is determined that a very large peak occurs in the data rate for the frame.

  As noted above, for most consumer grade Internet connections (and many office connections), data exceeding the maximum available data rate shown as 622 in FIG. 6c is “jam” along with the rated maximum data rate 621. There are cases where it is simply not feasible. The rated maximum data rate 621 (eg, “6 Mbps DSL”) is essentially a marketing number for users considering purchasing an Internet connection, but generally does not guarantee a level of performance. For the purposes of the present invention this is irrelevant. This is because only the maximum available data rate 622 is a problem when video is streamed through a connection. As a result, in FIGS. 9a and 9c, when describing a solution to the peak problem, the rated maximum data rate is removed from the graph and only the maximum data rate 922 available is shown. The video stream data rate should not exceed the maximum available data rate 922.

  To address this, the first thing the video compressor 404 should do is determine a peak data rate 941, which is the data rate that the channel can handle steadily. This rate can be determined by a number of techniques. One such technique, in FIGS. 4a and 4b, gradually sends a progressively higher data rate test stream from the hosting service 210 to the client 415, and the client provides feedback to the hosting service regarding packet loss and latency levels. It is what makes you When packet loss and / or latency begins to show a sharp increase, it is an indication that the maximum available data rate 922 is being reached. Thereafter, the hosting service 210 gradually reduces the data rate of the test stream until the client 415 reports that the test stream is received at a packet loss level that is acceptable for a reasonable period of time and the latency is approximately minimal. be able to. This establishes a peak maximum data rate 941, which is then used as the peak data rate for streaming video. Over time, the peak data rate 941 fluctuates (eg, when another user in the home begins to use the Internet connection heavily) and the client 415 constantly monitors it, increasing packet loss or latency. Indicates that the maximum available data rate 922 falls below the previously established peak data rate 941, and if so, indicates the peak data rate 941. Similarly, over time, if client 415 finds that packet loss and latency are kept at an optimal level, the video compressor will slowly increase the data rate to increase the maximum available data rate. Packet loss and / or long latency that the maximum available data rate 922 has been exceeded (for example, if another user at home stops heavy use of the Internet connection) You can again find a low level for the peak data rate 941, but wait again until it is probably higher than the previous level of the increased data rate test. Thus, by using this technique (and other techniques similar to it), the peak data rate 941 can be found and adjusted periodically as needed. The peak data rate 941 establishes the maximum data rate that can be used by the video compressor to stream video to the user. The logic for determining the peak data rate can be embodied in the user's home 211 and / or hosting service 210. In the user's home 211, the client device 415 performs calculations to determine the peak data rate and returns that information to the hosting service 210, where the hosting service server 402 received from the client 415. Based on statistical information (eg, packet loss, latency, maximum data rate, etc.), calculations are performed to determine the peak data rate.

  FIG. 9a illustrates a video stream data rate 934 having substantial scene complexity and / or motion generated using the cyclic I / P tile compression technique shown in FIGS. 7a, 7b and 8 and described above. Illustrate. Note that the video compressor 404 is configured to output compressed video at an average data rate that is lower than the peak data rate 941, and most of the time, the video stream data rate is kept below the peak data rate 941. Comparison of video stream data rate 634 and data rate 934 shown in FIG. 6c generated using I / P / B or I / P frames shows that cyclic I / P tile compression produces a very smooth data rate It shows that Further, at frame 2x peak 952 (which approaches 2x peak data rate 942) and frame 4x peak 954 (which approaches 4x peak data rate 944), the data rate exceeds peak data rate 941 and is acceptable. I can't. In fact, even high action video from rapidly changing video games, peaks above peak data rate 941 occur in less than 2% of the frame, peaks in excess of 2x peak data rate 942 occur infrequently, and 3x peak A peak exceeding the data rate 943 hardly occurs. However, when they occur (e.g., during scene transitions), the data rate required by them is necessary to produce a good quality video image.

  One way to solve this problem is to simply configure the video compressor 404 such that the maximum data rate output is the peak data rate 941. Unfortunately, the quality of the video output that results during the peak frame is insufficient because the compression algorithm is “deficient” in bits. As a result, a compression defect appears when there is a sudden transition or fast movement, and eventually the user realizes that a defect always appears when there is a sudden change or rapid movement, which is extremely annoying.

  The human visual system is very sensitive to visual defects that appear during sudden changes or rapid movements, but is less sensitive to detection of frame rate reduction in such conditions. In fact, when such a sudden change occurs, the human visual system is fascinated by tracking the change, the frame rate drops from 60 fps to 30 fps in a short time, and then instantly 60 fps. It seems that he / she does not notice to return to. And in the case of a very rapid transition, such as a sudden scene change, the human visual system is unaware that the frame rate drops to 20 fps or 15 fps and then immediately returns to 60 fps. As long as the frame rate decrease only occasionally occurs for the observing person, the video will appear as if it has run continuously at 60 fps.

  This property of the human visual system is exploited by the technique shown in FIG. 9b. Server 402 (from FIGS. 4a and 4b) generates an uncompressed video output stream at a constant frame rate (60 fps in one embodiment). The time line shows each frame 961-970 in 1/60 second each. Each uncompressed video frame starts at frame 961 and is output to the low latency video compressor 404, which compresses the frame in less than one frame time and generates compressed frame 1 981 as the first frame. The data generated for compressed frame 1 981 may be large or small based on a number of factors, as described above. If the data is small enough to be transmitted to the client 415 within a frame time (1/60 second) or less than the peak data rate 941, it is transmitted during the transmission time (xmit time) 991 (arrowed) The length indicates the width of the transmission time). At the next frame time, server 402 generates uncompressed frame 2 962, which is compressed into compressed frame 2 982, and to client 415 during transmission time 992, which is less than the frame time at peak data rate 941. Sent.

  Then, at the next frame time, server 402 generates uncompressed frame 3 963. When this is compressed by the video compressor 404, the resulting compressed frame 3 983 is more data than can be transmitted at the peak data rate 941 per frame time. Therefore, it is transmitted during a transmission time (2x peak) 993 taking up the entire frame time and a portion of the next frame time. Here, during the next frame time, server 402 generates another uncompressed frame 4 964 and outputs it to video compressor 404, but the data is ignored and is shown at 974. This is because the video compressor 404 is configured to ignore further uncompressed video frames that arrive while still transmitting the previous compressed frame. Of course, the video decompressor of client 415 does not receive frame 4 and simply continues to display frame 3 on display device 422 for a duration of 2 frames (ie, the frame rate is simply reduced from 60 fps to 30 fps).

  For the next frame 5, server 402 outputs uncompressed frame 5 965, which is compressed into compressed frame 5 985 and transmitted within one frame during transmission time 995. The video decompressor of the client 415 decompresses the frame 5 and displays it on the display device 422. Server 402 then outputs uncompressed frame 6 966 and video compressor 404 compresses it to compressed frame 6 986, where the resulting data is very long. The compressed frame is transmitted at the peak data rate 941 during the transmission time (4 × peak) 996, but it takes approximately 4 frames to transmit the frame. During the next 3 frame times, video compressor 404 ignores 3 frames from server 402 and client 415 decompressor holds frame 6 steadily on display device 422 during the 4 frame time period ( That is, the frame rate is simply reduced from 60 fps to 15 fps). Finally, server 402 then outputs frame 10 970, video compressor 404 compresses it to compressed frame 10 987, which is transmitted during transmission time 997, and the decompressor of client 415 is The frame 10 is decompressed and displayed on the display device 422, and the video resumes again at 60 fps.

  Video compressor 404 drops video frames from the video stream generated by server 402, but does not drop audio data regardless of how the audio arrives, and does not drop audio data when the video frame is dropped. Note that it continues to compress and send them to the client 415, which continues to decompress the audio data and feeds the audio to the device used by the user and plays the audio. Thus, the audio continues to be unattenuated during the period when the frame is dropped. Compressed audio consumes a relatively small percentage of bandwidth compared to compressed video and as a result does not significantly affect the overall data rate. Although not shown in any data rate diagram, a data rate capacity is always reserved for the compressed audio stream within the peak data rate 941.

  The example described above with respect to FIG. 9b was chosen to show how the frame rate drops during the data rate peak, but when the cyclic I / P tile technique described above is used. It has not been shown that high data rate peaks and frames dropped thereby are rare even in high scene complexity / high action sequences such as occur in video games, movies and some application software. Thus, the decrease in frame rate is occasional and short, and the human visual system does not detect it.

  When the frame rate reduction mechanism described above is applied to the video stream data rate shown in FIG. 9a, the resulting video stream data rate is shown in FIG. 9c. In this example, 2x peak 952 is reduced to flat 2x peak 953, 4x peak 955 is reduced to flat 4x peak 955, and the total video stream data rate 934 is kept below peak data rate 941.

  Thus, using the techniques described above, high-action video streams can be transmitted with low latency over common Internet and consumer grade Internet connections. In addition, in an office environment in a LAN (eg, 100 Mbps Ethernet or 802.11g wireless) or a private network (eg, a 100 Mbps connection between a data center and an office), high action video streams can be transmitted without peaks. , Allowing a large number of users (eg, sending 1920 × 1080 at 60 fps at 4.5 Mbps) to use a LAN or shared private data connection without having a superimposed peak overwhelming the network or network switch backplane.

Data Rate Adjustment In one embodiment, the hosting service 210 first accesses the maximum available data rate 622 and channel latency to determine an appropriate data rate for the video stream and then responds thereto. To dynamically adjust the data rate. In order to adjust the data rate, the hosting service 210 can change, for example, the video resolution and / or the number of frames / second of the video stream sent to the client 415. The hosting service can also adjust the quality level of the compressed video. When changing the resolution of the video stream, for example, from 1280x720 resolution to 640x360, the video decompression logic 412 of the client 415 can scale up the video to maintain the same video size on the display screen.

  In one embodiment, in a situation where the channel is completely dropped out, the hosting service 210 pauses the game. In the case of a multiplayer game, the hosting service reports to the other user that the user has dropped out of the game and / or pauses the game to the other user.

Dropped or delayed packets In one embodiment, due to packet loss between video compressor 404 and client 415 in FIG. 4a or 4b, or packets are received out of order and too slow to decompress The video decompression logic 412 can mitigate visual defects when data is lost due to arrivals that are too late to meet the frame latency requirements. In streaming I / P frame implementations, if there are lost / delayed packets, the entire screen is affected, potentially freezing the screen completely for a period of time, or viewing other screen widths Display the top defect. For example, if a lost / delayed packet causes an I frame loss, the decompressor will lack a reference for all subsequent P frames until a new I frame is received. If a P frame is lost, it will affect the P frame for all subsequent screens. Based on how long it takes for an I frame to appear, this can be a long or short visual effect. Using the interleaved I / P tiles shown in FIGS. 7a and 7b, lost / delayed packets are very unlikely to affect the entire screen. This is because it only affects the tiles contained in the affected packet. If the data for each tile is transmitted in an individual packet, it affects only one tile if the packet is lost. Of course, the width of the visual defect depends on whether the I tile packet is lost and how many frames it takes to see the I tile if the P tile is lost. However, if different tiles on the screen are updated frequently (potentially every frame) in an I-frame, even if one tile on the screen is affected, the other tiles are not. In addition, if an event causes the loss of a large number of packets at once (for example, interrupting the data flow for a short time with a power spike following the DSL line), one tile can affect more than other tiles. However, certain tiles are only affected for a short time because they are quickly updated with new I tiles. Also, in streaming I / P frame implementation, if the I frame is not only the most important frame, but also a very large frame, and therefore there are events that cause dropped / delayed packets, The probability that an I-frame will be affected is higher than a very small I-tile (ie, if any part of the I-frame is lost, there is no chance of decompressing the I-frame). For all these reasons, using I / P tiles results in much fewer visual defects when packets are dropped / delayed than with I / P frames.

  In one embodiment, an attempt is made to reduce the effects of lost packets by intelligently packaging compressed tiles into TCP (Transmission Control Protocol) or UDP (User Datagram Protocol) packets. For example, in one embodiment, tiles are aligned to packet boundaries if possible. FIG. 10a shows how tiles are packed into a series of packets 1001-1005 without embodying this feature. More specifically, in FIG. 10a, tiles cross packet boundaries and are packed inefficiently such that single packet loss results in multiple frame losses. For example, if the packet 1003 or 1004 is lost, three tiles are lost, resulting in a visual defect.

  In contrast, FIG. 10b shows tile packing logic 1010 for intelligently packing tiles into packets to reduce the effects of packet loss. First, tile packing logic 1010 aligns tiles to packet boundaries. Thus, tiles T1, T3, T4, T7, and T2 are each aligned with the boundaries of packets 1001-1005. Tile packing logic also attempts to fit tiles within a packet in the most efficient way possible without crossing packet boundaries. Based on the size of each tile, tiles T1 and T6 are combined in one packet 1001, T3 and T5 are combined in one packet 1002, and tiles T4 and T8 are combined in one packet 1003. Tile T8 is added to packet 1004 and tile T2 is added to packet 1005. Thus, under this skim, the loss of a single packet will result in the loss of two or fewer tiles (rather than the three tiles shown in FIG. 10a).

  One additional benefit over the embodiment shown in FIG. 10b is that tiles are transmitted in a different order than they are displayed in the video. In this way, if adjacent packets are lost from the same event that prevents transmission, they affect areas that are not adjacent to each other on the screen, making defects on the display less noticeable.

  In one embodiment, forward error correction (FEC) techniques are used to protect some portions of the video stream from channel errors. As known in the art, FEC techniques such as Reed-Solomon and Viterbi generate error correction data information and attach it to data transmitted over the communication channel. If an error occurs in the underlying data (eg, an I frame), the error can be corrected using FEC.

  FEC codes increase the data rate of transmission and therefore ideally are only used where they are most needed. If data is transmitted that does not cause very noticeable visual defects, it is preferable not to use FEC codes to protect the data. For example, a P tile immediately before a lost I tile causes a visual defect on the screen by 1/60 second (ie, the tile on the screen is not updated). Such visual defects can be barely detected by the human eye. If the P tile is further back from the I tile, losing the P tile will become more noticeable. For example, if the tile cycle pattern is that an I tile is followed by 15 P tiles, and then the I tile is obtained again, if the P tile immediately following the I tile is lost, the tile is 15 It will show the wrong video for the duration of the frame (at 60 fps it is 250 ms). The human eye easily detects a break in the stream for 250 ms. Thus, the further the P tile is behind the new I tile (ie, the closer the P tile is to the I tile), the more pronounced the defect. As mentioned above, in general, the closer the P tile is to the I tile, the smaller the data for that P tile. Therefore, not only is it important to protect the P tile following the I tile from being lost, but it is also important that the size is small. In general, the smaller the data that needs to be protected, the smaller the FEC code required to protect it.

  Thus, as shown in FIG. 11a, in one embodiment, only the I tile is given an FEC code due to the importance of the I tile in the video stream. Accordingly, FEC 1101 includes error correction code for I tile 1100 and FEC 1104 includes error correction code for I tile 1103. In this embodiment, no FEC is generated for P tiles.

  In one embodiment shown in FIG. 11b, FEC codes are also generated for P tiles that are most likely to cause visual defects if lost. In this embodiment, FEC 1105 provides error correction code for the first three P tiles, but not for subsequent P tiles. In another embodiment, an FEC code is generated for the P tile with the smallest data size (which tends to occur earliest after the I tile and self-select the P tile that is most important to protect).

  In another embodiment, rather than sending the FEC code with the tile, the tile is sent twice, each time with a different packet. If one packet is lost / delayed, the other packet is used.

  In one embodiment shown in FIG. 11c, FEC codes 1111 and 1113 are generated for audio packets 1110 and 1112 transmitted from the hosting service simultaneously with the video, respectively. It is particularly important to maintain audio integrity in the video stream. This is because distorted audio (e.g., ticks or soots) leads to a particularly unwanted user experience. The FEC code helps to ensure that the audio content is rendered at the client computer 415 without distortion.

  In another embodiment, rather than sending the FEC code with the audio data, the audio data is sent twice, each time in a different packet. If one packet is lost / delayed, the other packet is used.

  Further, in one embodiment shown in FIG. 11d, FEC codes 1121 and 1123 are used for user input commands 1120 and 1122 (eg, button presses) sent upstream from client 415 to hosting service 210, respectively. This is important. This is because the lack of button presses or mouse movements in a video game or application leads to an undesirable user experience.

  In another embodiment, rather than sending the FEC code with the user input command data, the user input command data is sent twice, each time in a different packet. If one packet is lost / delayed, the other packet is used.

  In one embodiment, the hosting service 210 evaluates the quality of the communication channel with the client 415 to determine whether FEC should be used, and if so, applies the FEC to any part of the video, audio and user commands. Decide what to do. Evaluating the “quality” of the channel includes functions such as evaluating packet loss, latency, etc., as described above. If the channel is not particularly reliable, the hosting service 210 can apply FEC to all of the I tiles, P tiles, audio, and user commands. In contrast, if the channel is reliable, the hosting service 210 applies FEC to audio and user commands only, or applies FEC to audio or video, or does not use FEC at all. While still meeting these basic principles, various other permutations of FEC applications can be used. In one embodiment, the hosting service 210 constantly monitors the channel status and changes the FEC policy accordingly.

  In another embodiment, referring to FIGS. 4a and 4b, a packet is lost / delayed and the tile data is lost, or perhaps because of a particularly bad packet loss, the FEC may have lost tile data. Can not be modified, the client 415 evaluates how many frames remain before a new I tile is received and compares it to the round trip latency from the client 415 to the hosting service 210. If the round-trip latency is less than the number of frames until the new I tile naturally arrives, the client 415 sends a message requesting the new I tile to the hosting service 210. This message is routed to the video compressor 404, which generates an I tile rather than a P tile for the tile where the data is lost. If the system shown in FIGS. 4a and 4b is designed to give a round trip latency of typically less than 80 ms, then the tile is modified within 80 ms (at 60 fps, the frame is 16. It is 67ms wide, so in the time of all frames, the tile is modified within 83.33ms, which is the time of 5 frames, with a latency of 80ms, ie this is a noticeable interruption, It is much less noticeable than the 250ms break for 15 frames). When compressor 404 generates an I tile from its normal cyclic order, if the bandwidth of the frame exceeds the available bandwidth by the I tile, compressor 404 delays the cycle of the other tiles, Make the other tiles receive P tiles during that frame time (even if one tile should normally be an I tile during that frame), then start with the next frame and continue with normal cycling , And usually the tile receiving the I tile in the previous frame will receive the I tile. This action delays the phase of the R frame cycle for a short time, but is usually not visually noticeable.

Video and Audio Compressor / Decompressor Implementation FIG. 12 illustrates one particular embodiment of compressing 8 tiles in parallel using a multi-core and / or multi-processor 1200. In one embodiment, a dual processor quad-core Xeon CPU computer system operating at 2.66 GHz or higher is used, each core being an open source x264 H.264. H.264 compressor is implemented as an independent process. However, various other hardware / software configurations may be used while still meeting these basic principles. For example, each of the CPU cores is implemented by FPGA. H.264 compressor. In the example shown in FIG. 12, cores 1201-1208 are used to simultaneously process I tiles and P tiles as eight independent threads. As is well known in the art, current multi-core and multi-processor computer systems were originally integrated with multi-threaded operating systems such as Microsoft Windows XP Professional Edition (64-bit or 32-bit edition) and Linux. Sometimes multithreading is possible.

  In the embodiment shown in FIG. 12, each of the eight cores works only for one tile, so it operates primarily independently of the other cores that each perform x264 individual instantiations. Capturing uncompressed video at 640x480, 800x600, or 1280x720 resolution using a PCI Express x1-based DVI capture card, such as the Sendero Video Imaging IP Development Board from Microtronics, Oosterhout, The Netherlands The card's FPGA then uses direct memory access (DMA) to transfer the captured video over the DVI bus to the system RAM. The tiles are arranged in a 4x2 array 1205 (they are shown as square tiles, but in this embodiment are 160x240 resolution). Each instantiation of x264 is configured to compress one of eight 168x240 tiles, and after the first I tile compression, each core enters a cycle and each one frame is another frame Are synchronized, as shown in FIG. 12, to phase out and compress one I tile and then seven P tiles.

  At each frame time, the resulting compressed tiles are combined into a packet stream using the techniques described above, and then the compressed tiles are sent to the destination client 415.

  Although not shown in FIG. 12, if the data rate of the 8 tiles to be combined exceeds a specific peak data rate 941, it is necessary until the data of the 8 tiles to be combined has been transmitted. Accordingly, a total of 8x264 processes are suspended during multiple frame times.

  In one embodiment, the client 415 is implemented as software on a PC running eight instances of FFmpeg. The receive process receives 8 tiles and each tile is routed to FFmpeg, which decompresses the tile and renders it at the appropriate tile location on the display device 422.

  The client 415 receives keyboard, mouse, or game controller input from the PC input device driver and sends it to the server 402. The server 402 then applies the received input device data to a game or application executed on the server 402, which is a PC that runs Windows using an Intel 2.16 GHz Core Duo CPU. Server 402 then generates a new frame and outputs it through its DVI output, either from the motherboard-based graphics system, or through the NVIDIA 8800GTX PCI Express card DVI output.

  At the same time, server 402 outputs the audio formed by the game or application through its digital audio output (eg, S / PDIF), which is the digital audio input of a dual quad-core Xeon-based PC that embodies video compression. Combined with The Vorbis open source audio compressor is used to compress audio simultaneously with video, using the core available for process threads. In one embodiment, the core that completes tile compression performs audio compression first. The compressed audio is then transmitted with the compressed video and decompressed at the client 415 using a Vorbis audio decompressor.

Light passing through a glass, such as a hosting service server center distribution optical fiber, travels at a fraction of the speed of light in a vacuum, and therefore the exact propagation speed of light in the optical fiber can be determined. In practice, however, the optimal latency on the Internet is observed to reflect a transmission rate of nearly 50% of the speed of light, considering routing delay times, transmission inefficiencies, and other overhead. . Thus, the optimal 1000 mile round trip latency is about 22 ms and the optimal 3000 mile round trip latency is about 64 ms. Thus, a single server on one coast of the United States is too far away to service clients on the other coast (as far as 3000 miles away) with the desired latency. However, as shown in FIG. 13a, the server center 1300 of the hosting service 210 is located in the central part of the United States (eg, Kansas, Nebraska, etc.), and the distance to any point on the US continent is about 1500 miles or less. In this case, the round-trip Internet waiting time can be reduced to about 32 ms. Referring to FIG. 4b, the worst-case latency allowed for user ISP 453 is typically 25 ms, but latency close to 10 to 15 ms has been observed in DSL and cable modem systems. Please be careful. FIG. 4b also assumes that the maximum distance from the user's house 211 to the hosting center 210 is 1000 miles. Thus, a typical user ISP round trip latency of 15 ms is used, and at a maximum internet distance of 1500 miles for a round trip latency of 32 ms, the user operates the input device 421 and sees a response on the display device 422. The total round-trip waiting time from is 1 + 1 + 15 + 32 + 1 + 16 + 6 + 8 = 80 ms. Thus, a response time of 80 ms is typically obtained over an internet distance of 1500 miles. This allows a user's home with a sufficiently short user ISP latency 453 in the US continent to access a single server located in the center.

  In another embodiment shown in FIG. 13b, the server centers HS-1 through HS6 of the hosting service 210 are strategically located in the United States (or other geographic region) and some large hosting service server center (eg, HS2 And HS5) are located in the vicinity of a population center. In one embodiment, the server centers HS1 to HS6 exchange information over a network 1301, which is the Internet or a private network or a combination of both. In a state where there are a plurality of server centers, it is possible to provide a service with a short waiting time to a user having a long user ISP waiting time 453.

  The distance on the Internet is certainly a factor that contributes to round trip latency through the Internet, but sometimes other factors that are less related to latency will come into play. Sometimes the packet stream is routed to a remote location through the Internet and back again, incurring latency from long loops. Sometimes routing devices that do not work properly are on the path, causing transmission delays. Sometimes there is overloaded traffic on the path and introduces delay. And sometimes a failure occurs that completely prevents the user's ISP from being routed to a given destination. Thus, the general Internet usually has very reliable optimal routes and latencies that are largely determined by distance (especially with long-distance connections that cause routes out of the user's local area), Gives a connection from one point to another, but such reliability and latency are not guaranteed and often get from a user's home to a given destination on the general Internet I can't.

  In one embodiment, when a user client 415 first connects to the hosting service 210 to play a video game or use an application, the client can connect to the hosting service server centers HS1 to HS6 available at startup. Communicate with each (eg, using the techniques described above). If the latency is sufficiently short for a particular connection, that connection is used. In one embodiment, the client communicates with all or a subset of the hosting service server center from which the one with the lowest latency connection has been selected. The client may select a service center with the shortest latency connection, or identify what the service center has the shortest time connection and provide this information to the client (eg, in the form of an Internet address) Good.

  If a particular hosting service server center is overloaded and / or the user's game or application can tolerate latency to another lightly loaded hosting service server center, the client 415 may have other hosting services. Redirected to the server center. In such a situation, the game or application that the user is running is paused on the server 402 at the server center in the user's overload condition, and the game or application state data is stored on a server at another hosting service server center. 402. The game or application is then resumed. In one embodiment, the hosting service 210 waits until the game or application reaches a natural pause (eg, between levels in the game, or after the user initiates a “save” operation on the application) and transfers. To do. In yet another embodiment, the hosting service 210 waits for user activity to pause for a specified period of time (eg, 1 minute), and then initiates a transfer at that point.

  As described above, in one embodiment, hosting service 210 subscribes to Internet bypass service 440 of FIG. 14 to attempt to provide guaranteed latency to its clients. As used herein, an Internet bypass service is a service that provides a private network route from one point to another in the Internet with guaranteed characteristics (eg, latency, data rate, etc.). For example, if hosting service 210 receives a large amount of traffic from a user using AT & T's DSL service provided in San Francisco rather than routing to AT & T's San Francisco-based central office, hosting service 210 may Large private data connections can be leased from a service provider (possibly AT & T itself or another provider) between the base central office and one or more service centers for hosting service 210. If the route from all hosting service server centers HS1 to HS6 to San Francisco users using AT & T's DSL via the general Internet then causes a very long latency, a private data connection will be used instead. Can be used. A private data connection is generally more expensive than a route through the general Internet, but as long as it remains a small percentage of the hosting service 210 connection to the user, the impact on the total cost is low and the user Will experience a more consistent service.

  Service centers often have two layers of backup power sources during a power outage. The first layer is typically a backup power source from the battery (or from another readily available energy source such as a flywheel that is attached to the generator and kept running), where the main power source is Supply power immediately in the event of a power failure to keep the server center running. If the power failure is short and the main power recovers quickly (eg, within 1 minute), all of the battery is needed to keep the server center running. However, if the power outage is for a long time, typically a generator (eg, diesel energization) is started and can take over the battery and run as long as there is fuel. Such a generator is very expensive because it must be able to generate as much power as the server center normally obtains from the main power source.

  In one embodiment, each of the hosting services HS1 to HS5 shares user data with each other, suspends ongoing games and applications in the event of a power outage at one server center, and then stores game or application status data. Transfer from each server 402 to another server center server 402 and notify each user's client 415 to instruct it to communicate with the new server 402. If such a situation does not occur frequently, it can accept the user's transfer to a hosting service server center that cannot provide optimal latency (ie, the user simply waits long during a power outage period). This only allows time), which allows a very wide range of options for transferring users. For example, given a time zone difference across the United States, East Coast users go to bed at 11:30 PM, while West Coast users at 8:30 PM begin to peak in video game usage. If a power outage occurs at the west coast hosting service server center at that time, there may not be enough west coast servers 402 at other hosting service server centers to handle all users. In such a situation, users can be transferred to the east coast hosting service server center with available servers 402, resulting in long latency only for those users. When a user is transferred from a failed server center, the server center then initiates an orderly shutdown of the server and equipment, shutting down all equipment until the battery (or other immediate power backup) is exhausted. can do. In this way, the cost of the generator for the server center can be avoided.

  In one embodiment, during periods of heavy load on hosting service 210 (due to peak user load or because one or more server centers have lost power), users wait for the games or applications they are using. Forwarded to another server center based on time requirements. Thus, users using games or applications that require low latency are given priority to available low latency server connections when power is limited.

Hosting Service Features FIG. 15 illustrates an embodiment of a server center component for the hosting service 210 used in the following feature description. Similar to the hosting service 210 shown in FIG. 2a, server center components are controlled and coordinated by the control system 401 of the hosting service 210 unless otherwise indicated.

  Inbound internet traffic 1501 from user client 415 is directed to inbound routing 1502. Typically, inbound Internet traffic 1501 enters the server center via a high speed fiber optic connection to the Internet, but network connections with adequate bandwidth, reliability and low latency are sufficient. Inbound routing 1502 is a network switch (a network can be embodied as an Ethernet network, a fiber channel network, or via other transport means) and a system of routing servers that support the switch. Pick up and route each packet to the appropriate application / game (app / game) server 1521-1525. In one embodiment, packets delivered to a particular app / game server represent a subset of data received from a client and / or other components in the data center (eg, network components such as gateways and routers). ) May be converted / changed. In some cases, for example, if a game or application is executed once in parallel on multiple servers, the packet is routed to more than one server 1521-1525 at a time. The RAID arrays 1511-1512 are connected to the inbound routing network 1502 and allow the app / game servers 1521-1525 to read and write to the RAID arrays 1511-1512. Further, a RAID array 1515 (which can be implemented as a plurality of RAID arrays) is also connected to the inbound routing 1502, and data from the RAID array 1515 can be read from the app / game servers 1521-1525. Inbound routing 1502 is a tree structure of switches rooted in inbound Internet traffic 1501, a mesh structure that interconnects all the various devices, or concentrated traffic between intercommunication devices is separated from concentrated traffic between other devices. It can be implemented in a wide range of conventional network architectures, including a series of interconnected subnets. One form of network configuration is SAN, which is typically used as a storage device, but can also be used for general high-speed data transfer between devices. The app / game servers 1521-1525 each have a plurality of network connections to the inbound routing 1502. For example, the servers 1521-1525 can have a network connection to a subnet attached to the RAID array 1511-1512 and another network connection to a subnet attached to another device.

  The app / game servers 1521-1525 may be configured to be all the same, some different, or all different, as described above with respect to server 402 in the embodiment shown in FIG. 4a. In one embodiment, each user typically uses at least one app / game server 1521-1525 when utilizing a hosting service. For simplicity, assume that a given user uses an app / game server 1521, but one user can use multiple servers and multiple users can use a single app / game. Servers 1521-1525 can be shared. User control input sent from client 415 as described above is received as inbound internet traffic 1501 and routed to app / game server 1521 through inbound routing 1502. The app / game server 1521 uses the user's control input as a control input to a game or application running on the server, and calculates the next frame of video and associated audio. The app / game server 1521 then outputs the uncompressed video / audio 1529 to the shared video compression 1530. The app / game server can output uncompressed video via means including a 1 gigabit or higher Ethernet connection, but in one embodiment, the video is output via a DVI connection, and audio and other compression and communication channels. The status information is output via a universal serial bus (USB) connection.

  Shared video compression 1530 compresses uncompressed video and audio from app / game servers 1521-1525. The compression may be implemented entirely in hardware or may be implemented in hardware executing software. There may be a dedicated compressor for each app / game server 1521-1525, or, if the compressor is fast enough, use a given compressor to move from more than one app / game server 1521-1525. Video / audio can be compressed. For example, at 60 fps, the video frame time is 16.67 ms. If the compressor can compress a frame in 1 ms, it can be used to compress video / audio from as many as 16 app / game servers 1521-1525, which is the input from one server. Is taken after another server, the compressor saves the state of each video / audio compression process, and the context is switched as it cycles between the video / audio streams from the server. As a result, substantial cost savings are obtained in the compression hardware. Since different servers complete frames at different times, in one embodiment, compressor resources are in shared pool 1530 with shared storage means (eg, RAM, flash) for storing the state of each compression process, and server 1521 When the 1525 frame is complete and ready for compression, the control means determines which compression resources are available at that time, including the compression process status of the server and the uncompressed video / Give with audio frames.

  Part of the state of each server's compression process is information about the compression itself, eg decompressed frame buffer data of previous frames that can be used as a reference for P tiles, video output resolution; compression quality; tile structure; Note that bit allocation; quality of compression, audio format (eg, stereo, surround sound, Dolby®, AC-3). However, the state of the compression process depends on the communication channel state information regarding the peak data rate 941, whether the previous frame (shown in FIG. 9b) is currently output (thus whether the current frame should be ignored) and potential Also includes channel characteristics to be considered for compression, for example, whether there are excessive packet losses (eg, with respect to I tile frequency, etc.) that affect compression decisions. The app / game server 1521 when the peak data rate 941 or other channel characteristics change over time as determined by the app / game server 1521-1525 that supports each user monitoring the data sent from the client 415. -1525 sends the relevant information to the shared hardware compression 1530.

  Shared hardware compression 1530 may also use the means described above to packetize the compressed video / audio and, if appropriate, apply the FEC code, copy some data, or otherwise To ensure that the video / audio data stream is received by the client 415 and decompressed with the highest possible quality and reliability.

  Some applications, such as those described below, require that the video / audio output of a given app / game server 1521-1525 be obtained simultaneously in multiple resolutions (or in other formats). Thus, if the app / game server 1521-1525 notifies the resource of the shared hardware compression 1530, the uncompressed video / audio 1529 of that app / game server 1521-1525 may have a different format, different resource, and / or Compressed simultaneously with different packet / error correction structures. In some cases, some compression resources can be shared between multiple compression processes that compress the same video / audio (eg, many compression algorithms require multiple sizes of video before applying compression). There is a scaling step that can be used to accommodate multiple compression processes at once if it is required to output video of different sizes). In other cases, a separate compression resource is required for each format. In any case, all the various resolutions and formats of compressed video / audio 1539 required for a given app / game server 1521-1525 are outbound routing (whether one or many). Output to 1540 at a time. In one embodiment, the output of the compressed video / audio 1539 is in UDP format and is therefore a position-oriented stream of packets.

  The outbound routing network 1540 routes each compressed video / audio stream to the intended user (s) through an outbound internet traffic 1599 interface (which is typically connected to a fiber interface to the internet). Or a series of routing servers and switches directed to other destinations and / or back to delay buffer 1515 and / or back to inbound routing 1502 and / or directed through a private network (not shown) for video distribution. ing. Note that outbound routing 1540 (as described below) can output a given video / audio stream to multiple destinations at once. In one embodiment, this is an Internet protocol that broadcasts a given UDP stream intended to be streamed to multiple destinations at a time and repeats this broadcast by routing servers and switches in the outbound routing 1540. It is implemented using (IP) multicast. The multiple broadcast destinations are multiple user clients 415 via the Internet, multiple inbound routing 1502, multiple app / game servers 1521-1525, and / or one or more delay buffers 1515. Thus, the output of a given server 1521-1522 is compressed into one or more formats, and each compressed stream is directed to one or more destinations.

  Furthermore, in another embodiment, multiple app / game servers 1521-1525 are used simultaneously by a single user (eg, in a parallel processing configuration to generate 3D output of complex scenes) and each server is completed. When generating a portion of the video, the video output of multiple servers 1521-1525 can be combined into a composite frame by shared hardware compression 1530, and from that point on, it is a single app / game. It is handled as described above as if it had arrived from the server 1521-1525.

  In one embodiment, a copy of the entire video generated by the app / game server 1521-1525 (at least the resolution of the video viewed by the user or higher) is at least a few minutes (in one embodiment, 15 minutes), the delay buffer 1515 Note that it is recorded in This allows each user to “rewind” the video from each session to review previous work or performance (in the case of games). Thus, in one embodiment, each compressed stream of video / audio output 1539 routed to user client 415 is also multicast to delay buffer 1515. When video / audio is stored in the delay buffer 1515, the directory of the delay buffer 1515 finds the network address of the app / game server 1521-1525 that is the source of the delayed video / audio and the delayed video / audio. Provides a cross-reference between possible locations on the delay buffer 1515.

The live, instantly viewed, instantly playable game app / game server 1521-1525 can be used to run a given application or video game for the user, as well as navigation by the hosting service 210 and It can also be used to create user interface applications for hosting service 210 that support other features. A screen shot of one such user interface application is shown in FIG. 16, ie a “game finder” screen. This particular user interface screen allows the user to view 15 games that are being played live (or delayed) by other users. Each of the “thumbnail” video windows, such as 1600, is a live video window with motion, showing one video from one user's game. The field of view shown in the thumbnail may be the same field of view that the user is viewing or may be a delayed field of view (e.g., if the user plays a combat game, the user will see where others are hiding) And may choose to delay the view of the game play for a period of time, eg, 10 minutes). The field of view may be a camera field of view of a game different from the user's field of view. Through menu choices (not shown in this figure), the user can select a selection of games to watch at a time based on various criteria. As a small example of an exemplary selection, the user may select a random selection of games (as shown in FIG. 16), all games of one type (all played by different players), only the top ranked players of the game, A player at a given level, a low-ranking player (eg, if the player is learning the basics), a player who is a “friend” (or rival), a game with the most viewers, etc. can be selected.

  In general, each user will determine if the video from their game or application will be seen by others, and if so, what others, when others will see them, only if they will be seen late, Note the judgment.

  The app / game server 1521-1525 that generates the user interface screen shown in FIG. 16 obtains 15 video / audio feeds by sending a message to the app / game server 1521-1525 for each user for whom a game is requested. To do. This message is sent through inbound routing 1502 or another network. This message includes the requested video / audio size and format and identifies the user viewing the user interface screen. A given user chooses the “privacy” mode and chooses not to allow other users to watch his game's video / audio (from his point of view or another point of view) or in the paragraph above As stated, the user is allowed to watch video / audio from his game, but may choose to delay viewing video / audio. User app / game server 1521-1525 that receives and accepts the request to allow viewing video / audio sends confirmation to requesting server and appends to shared hardware compression 1530 in the requested format or screen size Also notify the request to generate a typical compressed video stream (assuming the format and screen size are different from those already generated) and the destination of the compressed video (ie requesting server) Also instruct. If the requested video / audio is only delayed, the requesting app / game server 1521-1525 is so notified and delayed with the location of the video / audio in the directory of the delay buffer 1515. The delayed video / audio is obtained from the delay buffer 1515 by looking up the network address of the app / game server 1521-1525 which is the source of the video / audio. Once all these requests have been generated and processed, up to 15 raw thumbnail-sized video streams are routed from outbound routing 1540 to inbound routing 1502 to app / game servers 1521-1525 to the user interface screen. And is decompressed and displayed by the server. The delayed video / audio stream may have a screen size that is too large, and if so, the app / game server 1521-1525 decompresses the stream and scales the video stream down to the thumbnail size. In one embodiment, the request for audio / video is sent (and managed by) a central “management” service (not shown in FIG. 15) similar to the hosting service control system of FIG. Then redirect the request to the appropriate app / game server 1521-1525. Further, in one embodiment, the request may not be issued because the thumbnail is “pushed” to the user's client that allows it.

  Audio from fifteen games all mixed at the same time can make an unpleasant sound. The user may choose to mix all sounds together in this way (perhaps just to get a sensation of noise caused by all the actions they are seeing), or the user can play one game at a time. You may choose to listen only to audio from. Single game selection is accomplished by moving the yellow selection box 1601 to the given game (moving the yellow box using the arrow keys on the keyboard, moving the mouse, joystick Or by pressing the direction button of another device, such as a mobile phone). When a single game is selected, only the audio from that game is played. Game information 1602 is also shown. In the case of this game, for example, the issuer logo (“EA”) and the game logo “Need for Speed Carbon” and the orange horizontal bar indicate the people who are playing or watching the game at that particular moment. Numbers are indicated in relative terms (in this case many, and thus the game is “hot”). In addition, “Stats” is also provided, which means that 145 players are actively playing 80 different instantiations of Need for Speed Game (ie, individual player games or multiplayer games). And 680 viewers (this user is one of them). These statistics (and other statistics) maintain a log of the hosting service 210 operations and are billed to the user as appropriate and paid to the publisher that provides the content. And stored in RAID arrays 1511-1512. Some statistics are recorded by actions by the service control system 401, and some are reported to the service control system 401 by individual app / game servers 1521-1525. For example, the app / game server 1521-1525 executing this “game finder” application sends a message to the hosting service control system 401 when watching the game (and finishing watching the game), and how many games Allows updating of statistics on whether or not Some of the statistics are used in user interface applications such as this “game finder” application.

  When the user clicks the activation button on the input device, the user can zoom in on the thumbnail video in the yellow box while keeping the raw state to full screen size. This effect is illustrated in the process of FIG. Note that the video window 1700 has increased in size. In order to embody this effect, the app / game server 1521-1525 receives the full screen size video stream of the game routed to it from the app / game server 1521-1525 running the selected game. Request to make a copy (at the resolution of the user's display device 422). The app / game server 1521-1525 running the game no longer needs a copy of the game thumbnail size from the shared hardware compressor 1530 (another app / game server 1521-1525 does not require such a thumbnail) Limit) and then command a full screen size copy of the video to be sent to the app / game server 1521-1525 that is zooming the video. The user who is playing the game may or may not have the display device 422 having the same resolution as the user who is zooming up the game. In addition, others viewing the game may or may not have the same resolution display device 422 as the user zooming in on the game (and different audio playback means such as stereo or surround sound). May be included). Thus, the shared hardware compressor 1530 determines whether a properly compressed video / audio stream has already been generated that satisfies the requirements of the user requesting the video / audio stream, and if it exists. Notify outbound routing 1540 to route a copy of the stream to the app / game server 1521-1525 zooming the video, and if not, compress another copy of the video appropriate for that user To instruct the outbound routing to send the stream back to the app / game server 1521-1525 and the inbound routing 1502 that are zooming the video. The server currently receiving a full screen version of the selected video will decompress it and gradually scale up to full size.

  FIG. 18 shows what the screen will look like after the game is fully zoomed up to full screen, and the game is displayed at the full resolution of the user's display device 422 as shown by the video pointed to by arrow 1800. It is shown in The app / game server 1521-1525 running the game finder application sends a message to the other app / game server 1521-1525 that has generated a thumbnail that is no longer needed, and no other games are seen. A message is transmitted to the service control server 401. In this regard, the generated display is an overlay 1801 that is at the top of the screen and provides information and menu control to the user. Note that as the game progressed, the audience increased to 2503 viewers. Thus, with a large number of viewers, a large number of viewers are tied to a display device 422 that has the same or nearly the same resolution (each app / game server 1521-1525 is capable of scaling the video to adjust the fit). Have).

  Since the game shown is a multiplayer game, the user can decide to join the game at some point. The hosting service 210 may or may not allow the user to join the game for various reasons. For example, a user may have to pay to play a game, or may choose not to play, and the user may not have enough ranking to participate in that particular game ( For example, the user's Internet connection may not have a short enough waiting time for the user to play (eg, a latency constraint for watching the game). Thus, a game played at a distance (effectively on another continent) can be viewed without latency issues, but for games to play, latency is (a) enjoying the game And (b) it must be short enough for the user to be on an equal footing with other players with short latency connections). If the user is allowed to play, the app / game server 1521-1525 that provided the user with a “game finder” user interface is suitable for the hosting service control server 401 to play a particular game. Start the configured app / game server 1521-1525 (ie, search and start) to load the game from the RAID array 1511-1512, and then the hosting service control server 401 is now hosting the game Instructs inbound routing 1502 to transfer control signals from the user to the current app / game game server, and compresses video / audio from the app / game server hosting the “Game Finder” application From, to request that the instruction to the shared hardware compressor 1530 to switch to it to compress the video / audio from the app / game server game currently host. The “game finder” app / game service and the vertical sync of the new app / game server hosting the game are not synchronized, resulting in a likely time difference between the two syncs. The shared video compression hardware 1530 starts compressing the video when the app / game server 1521-1525 completes the video frame, so the first frame from the new server is earlier than the full frame time of the old server. This is before the previous compressed frame completes its transmission (eg, considering transmission time 992 in FIG. 9b, uncompressed frame 3 963 is completed half an early frame time. The transmission time 992 is affected. In such a situation, the shared video compression hardware 1530 ignores the first frame from the new server (eg, as frame 4 964 is ignored (974)), and the client 415 The last frame from the server is held for a special frame time, and the shared video compression hardware 1530 begins to compress the next frame time video from the new app / game server hosting the game. Visually, the transition from one app / game server to another app / game server is seamless. The hosting service control server 401 then informs the app / game game server 1521-1525 hosting the “game finder” to switch to the idle state until it is needed again.

  The user can then play the game. And what is better is that the game is played perceptually immediately (since it is loaded from the Raid array 1511-1512 to the app / game game server 1521-1525 at gigabit / second speed) and the game A strictly configured operating system for the game with ideal drivers, registry configuration (for Windows), and no other applications running on the server that can compete with the game's operation Along with a server that is strictly suitable for the game.

  Also, as the user progresses through the game, each segment of the game is loaded from the RAID array 1511-1512 to the server at a gigabit / second rate (ie, 1 gigabyte load is 8 seconds) and the RAID array 1511- Because of the massive storage capacity of 1512 (which is a very large and cost effective because it is a shared resource between many users), geometric settings or other game segment settings are pre-calculated Can be stored in RAID arrays 1511-1512 and loaded very quickly. Furthermore, since the hardware configuration and calculation capability of each app / game server 1521-1525 are known, the pixel and vertex shaders can be pre-calculated.

  Thus, the game starts almost instantaneously, runs in an ideal environment, and subsequent segments are loaded almost instantaneously.

  However, beyond these effects, the user can see others playing the game (via the “game finder” and other means described above) and determine if he is interested in the game, and if so If so, you can learn secrets from seeing others. The user can then instantly demo the game without having to wait for large downloads and / or installations, and the user can probably play the game on a low cost trial basis or on a long term basis. You can play instantly. The user can then play the game on a Windows PC, Macintosh, home TV receiver, or even on a mobile phone with a wireless connection with a sufficiently short waiting time. Also, all this can be done without physically having a copy of the game.

  As mentioned above, users may not allow others to see their gameplay, allow their games to be viewed after a delay, or allow their users to see their games. You can decide whether to allow or allow your game to be viewed by all users. Nevertheless, video / audio is stored in the delay buffer 1515 for 15 minutes in one embodiment, and the user could have done when watching TV on a digital video recorder (DVR). Similarly, you can "rewind" to see your previous gameplay, pause, play in slow, fast forward, etc. In this example, the user plays a game, but can use the same “DVR” capability when the user uses an application. This is also useful for reviewing previous behavior and other applications described below. Furthermore, this “3D DVR” capability is also supported if the game is designed so that it can be rewound based on the use of game state information, such as changing the field of view of the camera. It is required to design the game to support it. The “DVR” capability using the delay buffer 1515 is of course limited to video that occurs when the game or application is used, but works with the game or application, but for games with 3D DVR capability, The user can control the “fly through” in 3D of a previously played segment and have the resulting video recorded in the delay buffer 1515 to record the game state of the game segment. it can. Thus, a particular “fly-through” is recorded as a compressed video, but the game state is also recorded, so at a later date, different fly-throughs will be considered for the same segment of the game.

  As described below, each user at the hosting service 210 has a “user page” on which information about them and other data can be posted. Among the things that users can post are video segments from gameplay that they saved. For example, if a user overcomes a particularly difficult challenge in the game, the user “rewinds” to just before the point where great achievements could have been made in the game, and then some time for other users to see. The hosting service 210 can be instructed to save a video segment of width (eg, 30 seconds) to the user's “user page”. To accomplish this, the user simply plays the video stored in the delay buffer 1515 into the RAID array 1511-1512 and uses it to index that video segment in the user's “user page”. This is a problem of the game server 1521-1525.

  When the game has the 3D DVR capability described above, the game state information required for the 3D DVR can also be recorded by the user and used for the user's “user page”.

  If the game is designed to have an “audience” (ie, a user who can navigate the 3D world and observe it without being involved in an action) in addition to an active player, a “game finder” application Allows users to participate in the game as spectators and players. From a realization point of view, there is no difference to the hosting system 210 even if the user is a spectator rather than an active player. The game is loaded into the app / game server 1521-1525, and the user controls the game (eg, controls a virtual camera that sees the world). The only difference is the user's gaming experience.

Another feature of multi-user collaborative hosting service 210 is the ability for multiple users to collaborate while watching live video, even if they use very different viewing devices. This is useful both when playing games and when using applications.

  Many PCs and mobile phones are equipped with video cameras and have the ability to perform real-time video compression, especially when the video is small. Small cameras that can be attached to televisions are also available, and real-time compression can be implemented in software or using one of a number of hardware compression devices for compressing video. It ’s not difficult. Also, many PCs and all mobile phones have a microphone, and a headset can be used with the microphone.

  When such a camera and / or microphone is combined with local video / audio compression capabilities (especially using the low latency video compression techniques described herein), the user can view video from the user's home 211. And / or enable audio to be sent to the hosting service 210 along with the control data of the input device. When such a technique is used, the capability shown in FIG. 19 can be achieved, i.e., the user can have his video and audio 1900 appear on a screen in another user's game or application. This example is a multiplayer game in which teammates cooperate in a car race. The user's video / audio can be selectively viewed / listened only by teammates. And since there is virtually no waiting time, using the techniques described above, players can speak and move in real time with each other without noticeable delay.

  This video / audio integration is accomplished by arriving the compressed video and / or audio from the user's camera / microphone as inbound internet traffic 1501. Inbound routing 1502 then routes the video and / or audio to the app / game game server 1521-1525 that is allowed to view / listen to it. Each app / game game server 1521-1525 user who chooses to use video and / or audio will then decompress it and, as indicated at 1900, in the game or application as needed. Integrate to appear in

  The example of FIG. 19 shows how such cooperation is used in the game, but such cooperation is a very powerful tool for the application. In New York City, Chicago architects have designed larger buildings for New York-based real estate developers, but that decision includes a financial investor who travels and hangs at an airport in Miami, and To satisfy both investors and property developers, consider a situation in which some design elements of a building need to be judged on how to harmonize with the surrounding building. The construction company has a high-resolution monitor with a camera attached to a PC in Chicago, the property developer has a laptop with a camera in New York, and the investor has a mobile phone with a camera in Miami. is doing. The construction company can use the hosting service 210 to host a powerful construction design application that can do very realistic 3D rendering, including a large database of buildings in New York City and A database can be used. The construction design application is executed on one of the app / game servers 1521-1525, or on some of them if it requires significant computing power. Each of the three users at different locations is connected to the hosting service 210, and each has a simultaneous view of the video output of the construction design application, which is a given device and network each user has. Appropriately sized with shared hardware compression 1530 for connection characteristics (eg, a construction company can view a 2560x1440 60 fps display through a 20 Mbps commercial Internet connection, and a New York real estate developer can A 1280x720 60fps video can be viewed via a 6Mbps DSL connection on a laptop, and an investor can view a 320x180 60fps video via a 250Kbps cellular data connection on his mobile phone. (Conference calls are handled by any of a number of conference call software packages widely available in the app / game server 1521-1525), and through the operation of buttons on the user input device, the user can You can make your own video appearances using a local camera, and as the meeting progresses, architects can see how the building rotates when it rotates and flies near other buildings in the area Or can be shown in very photorealistic 3D rendering, and the same video can be viewed by all parties at the resolution of each party's display device. Can't handle 3D animation with realism like this That's not a problem, but it goes without saying that you download or remember the huge database required to render buildings around New York City, even though they are far from each user's point of view. In spite of being a different local device, the user simply has a seamless experience with incredible realism, and one person can see his face and express his emotions. If you want to communicate well, you can do so, and either the property developer or investor gets control of the construction program and has their own input device (keyboard, mouse, keypad or touch screen). If you want to use it, you can and will respond with no perceptual latency (and their network connection is (Assuming no reasonable waiting time). For example, in the case of a mobile phone, if the mobile phone is connected to the WiFi network at the airport, the waiting time is very short. But if you use cellular data networks available in the US today, you will probably suffer from a noticeable delay. Furthermore, for most purposes of meetings that investors are looking at, cellular latencies must be acceptable in order for an architect to control the approaching flight of a building or speak in a video teleconference.

  Finally, at the end of the collaborative conference call, property developers and investors make and comment their comments from the hosting service, and the construction company “rewinds” the conference video recorded in the delay buffer 1515. And comments, facial expressions, and / or actions added to the 3D model of the building created during the meeting can be reviewed. If there are specific segments that you want to save, those segments of video / audio can be moved from the delay buffer 1515 to the RAID arrays 1511-1512 for recording storage and later playback.

  Also, in terms of cost, if an architect needs to use computational power and a large New York City database only for a 15 minute conference call, he has a high power workstation and an expensive copy of the large database. Instead of purchasing a resource, you only have to pay for the time spent using the resource.

Video rich community service hosting service 210 enables an unprecedented opportunity to establish video rich community services on the Internet. FIG. 20 shows an exemplary “user page” for a game player in hosting service 210. Similar to the “game finder” application, the “user page” is an application that is executed on one of the app / game servers 1521-1525. The thumbnails and video windows on this page all show a constantly moving video (or a loop if the segment is short).

  By using a video camera or uploading a video, the user (whose username is “KILLHAZARD”) can post his own video 2000 and other users can see it. . This video is stored in a RAID array 1511-1512. Also, when another user arrives at KILLHAZARD's “user page”, if KILLHAZARD is currently using hosting service 210 (the user will see his “user page” to see him). The live video 2001 is shown no matter what he does (assuming forgiveness). This is achieved by the app / game server 1521-1525 hosting the “user page” application requested by the server control system 401, and whether KILLHAZARD is active and if so, the app he uses. / Game server 1521-1525. Then, using the same method used by the “Game Finder” application, a compressed video stream of the appropriate resolution and format is sent to the app / game server 1521-1525 running the “User Page” application, And displayed. If the user selects a window in KILLHAZARD's live gameplay and then clicks on the input device appropriately, the window is zoomed up (again, using the same method as the “Game Finder” application) The live video then fills the screen with the resolution of the viewing user's display device 422 that is appropriate for the characteristics of the viewing user's Internet connection.

  An important advantage of this solution over the past is that the user viewing the “user page” can see the live play of games that he does not own and, very well, play the game There is no need to have a local computer or game console that can. This gives a good opportunity for the user to see the user shown on the “user page”, which is “in action” to play the game, as well as an opportunity for the viewing user to learn about the game they want to try or improve.

  Camera recordings or uploaded video clips from KILLHAZARD's companion 2002 are also shown on the “user page” and below each video clip is text indicating whether the companion is playing the game online (eg, (six_shot is playing the game “Eragon”, MrSnggles99 is offline, etc.). By clipping a menu item (not shown), the partner's video clip shows what was recorded or uploaded, and what the partner currently playing the game on the hosting service 210 then does in the game. Switch to a live video of what you are doing. This is therefore a game finder group for a companion. If a buddy game is selected and the user clicks on it, the entire screen is zoomed in and the user can see the game played full screen live.

  Again, the user watching the companion game does not have a copy of the game or a local computing / game console resource to play the game. Watching the game is practically instantaneous.

  As described above, when a user plays a game on the hosting service 210, the user "rewinds" the game, finds the video segment that he wants to save, and then saves the video segment to his "user page". These are called “Brag Clips”. The video segments 2003 are all “proud clips” 2003 saved by KILLHAZARD from previous games played. The number 2004 indicates how many times the “proud clip” has been viewed, and when the “proud clip” is viewed, the user has the opportunity to rate it and the orange keyhole shaped icon 2005 The number indicates how high the rating is. “Proud clip” 2003 continuously loops with the rest of the video on the page when the user views the “user page”. If the user selects and clicks on one of the “Proud Clips” 2003, it zooms in and plays the “Proud Clip” 2003 to play, pause, rewind, fast forward, step through, etc. Presented with DVR controls that allow

  The playback of “proud clip” 2003 is performed when the user records the “proud clip” and decompresses and plays back the compressed video segment stored in the RAID array 1511-1512 in the app / game server 1521-. 1525 is implemented by loading.

  “Proud clip” 2003 is also a “3D DVR” video segment (ie, a game state sequence from a game that can be played and allows the user to change the camera's viewpoint), which is such a capability. Is from games that support. In this case, game state information is stored in addition to the specific “fly-through” compressed video recording created by the user when the game segment was recorded. When a “user page” is seen and all thumbnails and video windows are constantly looping, 3D DVR “Proud Clip” 2003 records as compressed video when the user records a “fly-through” of the game segment The “Proud Clip” 2003 is continuously looped. However, when the user selects a 3D DVR “Proud Clip” 2003 and clips it, in addition to the DVR control that allows the “Proud Clip” of the compressed video to be played, the user can use the 3D DVR for the game segment. You can click a button that gives you the ability. They can control their camera “fly-through” during the game segment themselves, and if they want (and the user who owns the user page allows it) another “proud clip” "Fly-through" can be recorded in compressed video form, which is then made available to other viewers of the user page (immediately or by the owner of the user page "proud clip" After having the opportunity to review).

  This 3D DVR “Proud Clip” 2003 capability is enabled by running a game about to play the recorded game state information on another app / game server 1521-1525. Since the game can be run almost instantaneously (as described above), play is limited to the game state recorded by the “proud clip” and the user then plays the compressed video in the delay buffer 1515. It is not difficult to be able to perform “fly-through” with the camera while recording to the camera. When the user completes the “fly-through”, the game is deactivated.

  From the user's point of view, activating “fly-through” with the 3D DVR “Proud Clip” 2003 requires less effort than controlling the linear “Proud Clip” 2003 DVR control. They may not know anything about the game or how to play the game. They are just virtual camera operators staring at the 3D world during a game segment recorded by another person.

  Users can also overdub their own audio into “proud clips” that are recorded or uploaded from a microphone. In this way, "Proud Clips" can be used to generate custom animations using characters and actions from the game. This animation technique is commonly known as “machinima”.

  As the user progresses through the game, different skill levels are achieved. The played game reports the achievement level to the service control system 401, and their skill level is shown in the “user page”.

Interactive animated advertising Online advertising has transitioned from text to still video to video and now to interactive segments that are typically embodied using animation thin clients such as Adobe Flash. The reason for using animation thin clients is that users typically cannot tolerate delays in the benefits that products and services are recommended to them. Thin clients also run on very low performance PCs, so advertisers can have a high degree of confidence that interactive ads will function properly. Unfortunately, animation thin clients such as Adobe Flash are limited in their degree of interactivity and breadth of experience (to reduce download time).

  FIG. 21 shows an interactive advertisement where the user selects the exterior and interior colors of the car while rotating the car in the showroom and showing in real time ray tracing how the car will look. The user can select an avatar to drive the car, and then the user can remove the car to drive on a race track or in an exotic location such as Monaco. The user can select a larger engine or a better tire and then see how the modified configuration affects the acceleration performance of the car or the road biting.

  Of course, advertising is actually a sophisticated 3D video game. However, for advertisements that can be played on a PC or video game console, you will probably require a 100MB download, and in the case of a PC, you will need to install a special driver and the PC will have enough CPU or GPU computing power. If it is missing, it may not be executed at all. Therefore, such an advertisement is not possible with the conventional configuration.

  In the hosting service 210, such advertisements are launched almost instantaneously and are fully executed regardless of the capabilities of the user's client 415. Thus, they start faster than thin-client bi-directional ads, are significantly more experienced, and are very reliable.

Streaming geometry RAID arrays 1511-1512 and inbound routing 1502 during real-time animation ensure delivery of geometry on-the-fly during game play or application during real-time animation (eg, fly-through with complex databases) In order to design video games and applications that rely on RAID arrays 1511-1512 and inbound routing 1502 to provide high speed and low latency data rates.

  In a conventional system such as the video game system shown in FIG. 1, a mass storage device that can be used for an actual home device, in particular, has a geometric shape during game play, except in situations where the required geometric shape can be predicted to some extent. Is much slower to stream. For example, in a drive game where certain roads exist, the geometric shape of the building entering the field of view can be reasonably well predicted, and the mass storage device is located nearing the geometric shape You can search for places in advance.

  However, in complex scenes with unpredictable changes (for example, in battle scenes where there are complex characters all around), the PC or video game system's RAM is completely in the geometry for the object currently in view. If it is filled and then the user suddenly redirects them to the field of view behind them, there will be a delay before the geometry can be displayed if it is not preloaded in RAM It will be.

  In hosting service 210, RAID arrays 1511-1512 can stream data at gigabit Ethernet speeds and higher, and in SAN networks, achieve 10 gigabits per second over 10 gigabit Ethernet or other networks. Can do. 10 gigabits / second loads gigabytes of data in less than 1 second. With a frame time of 60 fps (16.67 ms), almost 170 megabits (21 MB) of data can be loaded. Of course, rotating media in a RAID configuration still suffers a latency greater than one frame time, but flash-based RAID storage is eventually as large as a rotating media RAID array, such as There is no long waiting time. In one embodiment, mass RAM write-through caching is used to provide very low latency access.

  Thus, with high enough network speed and large storage with sufficiently low latency, geometry can be transferred to the app / game game server 1521-1525 as fast as the CPU and / or GPU can process 3D data. Can be streamed. Thus, in the above example where the user suddenly turns the character and looks back, the geometry for all the characters behind can be loaded before the character completes the rotation, so for the user It seems as if you are in a real world that is as real as a live action.

As mentioned above, one of the last pioneers in photoreal computer animation is the human face, which is sensitive to imperfections, so there are some errors from the photoreal face. Negative reaction may be invited from the viewer. FIG. 22 shows Contour ^™ “Reality Capture Technology” (a co-pending patent application each assigned to the assignee of this CIP application, ie, 10 / 942,609 filed on September 15, 2004, “Apparatus”. and method for capturing the motion of a performer ", No. 10 / 942,413 filed on September 15, 2004," Apparatus and method for capturing the expression of a performer ", filed February 25, 2005. No. 11 / 066,954 “Apparatus and method for improving marker identification within a motion capture system”, No. 11 / 077,628 “Apparatus and method for performing motion capture using shutter” filed on Mar. 10, 2005. synchronization ", No. 11 / 255,854," Apparatus and method for performing motion capture usi "filed on October 20, 2005 ng a random pattern on capture surfaces ", No. 11 / 449,131" System and method for performing motion capture using phosphor application techniques "filed June 7, 2006, filed June 7, 2006. No. 11 / 449,043 “System and method for performing motion capture by strobing a fluorescent lamp”, No. 11 / 449,127 “System and method for three dimensional capture of stop-” filed on June 7, 2006 The live performance captured using “motion animated characters” follows a very smooth capture surface, followed by a high polygon number tracking surface (ie, the polygonal motion accurately follows the facial motion). Shows how this occurs. Finally, photorealistic results are obtained when a video of a live performance is mapped to a tracking surface to form a textured surface.

  Current GPU technology can render the number and texture of polygons on the tracking surface and illuminate the surface in real time, but the polygons and texture change from frame time to frame time (this is the most photorealistic result). If so, it will quickly consume all available RAM on a modern PC or video game console.

  Using the streaming geometry technology described above, the geometry is continuously supplied to the app / game game server 1521-1525, and the photoreal faces are continuously animated, making them almost indistinguishable from the faces of live motion. It's practical to be able to create a video game with a frightening face.

Integrated video, television programs and audio material (collectively “linear content”) of linear content and interactive features can be widely used in many forms by home and office users. Linear content can be obtained on physical media such as CD, DVD, HD-DVD and Blu-ray media. It can also be recorded by DVR from satellite and cable TV broadcasts. It can also be used as pay-per-view (PPV) content through satellite and cable TV and as video-on-demand (VOD) in cable TV.

  More and more linear content is available as downloaded content and streaming content over the Internet. Today, there is not really one place to experience all the features associated with linear media. For example, DVDs and other video optical media typically have interactive features that are not available elsewhere, such as director commentary, “making of” features, and the like. Online music sites have cover technology and song information that is not generally available on CDs, but not all CDs are available online. Also, websites related to television programs often have special features, blogs, and sometimes comments from actors or creative staff.

  In addition, many videos or sports events are often released with linear media (for videos) or closely tied to real-world events (for example, player trades). There are often games.

  Hosting service 210 is well suited for distributing linear content linked with different forms of related content. Indeed, video distribution is no longer a challenge to delivering highly interactive video games, and hosting service 210 can deliver linear content to a wide range of home or office devices or mobile devices. FIG. 23 shows an example user interface page for hosting service 210 showing the selection of linear content.

  However, unlike most linear content delivery systems, the hosting service 210 has associated interactive components (eg, menus and features on DVDs, interactive overlays on HD-DVDs, and Adobe Flash animations on websites (described below)). )) Can also be distributed. Thus, the client device 415 restrictions no longer introduce restrictions on which features are available.

  Further, the hosting service 210 can link linear content with video game content dynamically and in real time. For example, if a user watches a Quidditch game in a Harry Potter movie and wants to try playing Quidditch, the user can click the button to pause the movie and immediately go to the Quidditch segment of the Harry Potter video game. it can. After playing the Quidditch game, another click on the button will immediately resume the movie.

  In photorealistic graphics and production techniques where photo-captured video is indistinguishable from live-motion characters, users can use video games in the hosting services described here, from quiditch games in live-motion movies. When transitioning to a Quidditch game, the two scenes are virtually indistinguishable. This provides a whole new creation option for directors of both linear and interactive (eg video game) content since the line between the two worlds is indistinguishable.

  Using the hosting service architecture shown in FIG. 14, the viewer can be given control of the 3D movie virtual camera. For example, in a scene that occurs in a train car, the viewer can control the virtual camera to see the surroundings of the vehicle as the story progresses. This assumes that all 3D objects (assets) in the vehicle are available and that the original movie has a sufficient level of computational power to render the scene in real time.

  And even non-computer generated entertainment can provide very exciting interactive features. For example, the 2005 movie “Pride and Prejudice” has many scenes in a brilliant old English mansion. For some scenes of the mansion, the user can pause the video and then control the camera to film the mansion tower, or perhaps the surrounding area. To realize this, the camera can be transported through the mansion with a fish-eye lens so as not to lose sight of the position, as in the case of the conventional Apple QuickTime VR. The various frames are then converted, so that the video is stored in the RAID array 1511-1512 with the movie without distortion and can be played back when the user chooses to go to the virtual tower.

  In sports events, live sports events, such as basketball games, can be streamed through the hosting service 210 for the user to watch on a normal TV. After the user sees a particular play, the video game of the game (eventually with a basket player that looks as photoreal as the real player) can be started with the player starting at the same location, and the user (Probably each with control of one player) can redo the play and see if it is better than the player.

  The hosting service 210 described here is very well suited to support this futuristic world. Because it can retain computing power and mass storage resources that are impossible to install in a home or most office setting, and the home setting always has an older generation of PCs and video games. However, the computational resources are the latest computational hardware available and always up-to-date. And in the hosting service 210, all of this computational complexity is hidden from the user, so it is as simple as switching TV channels from the user's point of view, even using a very sophisticated system. In addition, the user has access to all computing power and access to experience where computing power is obtained from the client 415.

The multiplayer game can be communicated with the app / game game server 1521-1525 through the network of inbound routing 1502 in that the multiplayer game is a multiplayer game, and executed in the hosting service 210 with a network bridge to the Internet (not shown). Can communicate with servers or gaming machines that are not. When playing a multiplayer game on a computer on the general Internet, the app / game game server 1521-1525 is said to have very fast access to the Internet (compared to when the game is run on a home server). The lowest common denominator, which has advantages but is limited by the ability of other computers to play games over a slow connection, and the game server on the Internet is a home computer on a relatively slow consumer Internet connection It is also potentially limited by being designed to accept.

  However, when a multiplayer game is played entirely within the server center of the hosting service 210, a world of difference can be achieved. Each app / game game server 1521-1525 hosting games for the user is very fast, very low latency connections and a huge amount of very fast memory with other app / game game servers 1521-1525. Interconnected with a server that hosts centralized control for multiplayer games with arrays. For example, if gigabit Ethernet is used for the inbound routing 1502 network, the app / game game servers 1521-1525 communicate with each other at a gigabit / second rate, potentially with a latency of 1 ms or less, It also communicates with a server that hosts centralized control for a multiplayer game. In addition, RAID arrays 1511-1512 can respond very quickly and transfer data at gigabit / second rates. For example, in a traditional system limited to game clients running on PCs or game consoles at home, the user customizes the character with respect to appearance and clothing so that the character has a large amount of geometry and unique behavior If the character enters another user's field of view, the user waits until the long and slow download is complete and all geometry and behavior data is loaded into the computer. Will have to. Within the hosting service 210, the same download can be done at a rate of gigabits per second over the corresponding gigabit Ethernet from the RAID arrays 1511-1512. Even if home users have an Internet connection of 8 Mbps (which is very fast with today's standards), Gigabit Ethernet is 100 times faster. Therefore, what takes 1 minute for a high-speed Internet connection is less than 1 second for Gigabit Ethernet.

Top player group and tournament hosting service 210 is very well suited for tournaments. Since the game is not run on the local client, there is no opportunity for the user to cheat. Also, because output routing 1540 can multicast UDP streams, hosting service 210 can broadcast major tournaments to thousands of people in the audience at once.

  In fact, when there are several video streams that are popular to allow thousands of users to receive the same stream (eg, showing a view of a major tournament), Akamai or It is more efficient to send the video stream to a “content delivery network” (CDN) such as Timelight.

  Similar efficiency levels can be obtained when using the CDN to indicate the “game finder” page of the top player group.

  In the case of major tournaments, live broadcasters can be broadcast during a game using famous live announcers. Many users watch major tournaments, but relatively few play in tournaments. The audio from the famous announcer can be routed to the app / game game server 1521-1525 hosting the user playing in the tournament and hosting a spectator mode copy of the game in the tournament, and the audio is game audio Can be overdubbed on top. Famous announcer footage can also be overlaid on the game and possibly on the audience's view.

Web page load acceleration World Wide Web, its primary transport protocol, Hypertext Transfer Protocol (HTTP), was the only business with a high-speed Internet connection and online consumers using dial-up modems or ISDN It was thought and defined in the times. At that time, the “gold standard” for high speed connections was a T1 line that provided a data rate of 1.5 Mbps symmetrically (ie, with an equal data rate in both directions).

  Today, the situation is completely different. In many of the developed worlds, the average home connection speed through a DSL or cable modem connection has a much higher downstream data rate than the T1 line. In fact, in some parts of the world, fiber-to-the-curb carries data rates as high as 50-100 Mbps to the home.

  Unfortunately, HTTP has not been configured (and not implemented) to effectively take advantage of these rapid speed improvements. A website is a collection of files on a remote server. Very simply, HTTP requests a first file, waits for the file to be downloaded, then requests a second file, waits for the file to be downloaded, And so on. In fact, HTTP allows more than one "open connection", i.e. allows more than one file to be requested at a time, but the agreed standard (and the desire to prevent overloading the web server). ), Very few open connections are allowed. In addition, because of the way web pages are constructed, browsers often do not notice multiple simultaneous pages that are immediately available for download (ie, only after parsing the page, new videos, as well as video). It becomes clear that you need to download the file). Thus, the website files are essentially loaded one by one. And because of the request and response protocol used by HTTP, there is approximately 100 ms latency associated with each file loaded (when accessing a typical web server in the United States).

  For relatively low speed connections, this does not introduce major problems. This is because the download time of the file itself dominates the latency of the web page. But as the connection speed increases, problems begin to arise, especially with complex web pages.

  In the example shown in FIG. 24, a typical commercial website is shown (this particular website is from a brand of first-class athlete shoes). There are 54 files on this website. These files include HTML, CSS, JPEG, PHP, Java (registered trademark) Script, and Flash files as well as video content. A total of 1.5 Mbytes must be loaded before the page is live (ie, the user clicks on it and starts using it). There are many reasons for many files. One is a complex and sophisticated web page, and another is dynamically based on information about the user accessing the page (eg, the user's home country, language, whether it was previously purchased, etc.) A web page that is assembled based on, and different files are downloaded based on all these factors. This is still a very typical commercial web page.

  FIG. 24 shows the amount of time that elapses before a web page becomes live as the connection speed increases. At a 1.5Mbps connection speed 2401, using a conventional web server with a conventional web browser, it takes 13.5 seconds for the web page to be live. With a 12 Mbps connection speed 2402, the load time is reduced to 6.5 seconds, or about twice as fast. However, at a 96 Mbps connection speed 2403, the load time can only be reduced to about 5.5 seconds. The reason for this is that at this high download speed, the time to download the file itself is minimal, but there is still about 100 ms of latency per file, resulting in 54 files * 100 ms = 5. The waiting time is 4 seconds. Thus, no matter how fast the connection to the home is, this website always takes at least 5.4 seconds to live. Another factor is server-side queuing, i.e. each HTTP request is added to the back of the queue, so this has a big impact on busy servers. This is because every time a small item is obtained from the web server, the HTTP request needs to wait for its turn.

  One way to solve these problems is to discard or redefine HTTP. Or perhaps the website owner has successfully merged the file into a single file (eg, in Adobe Flash format). But as a practical matter, this company and many others are investing heavily in its website architecture. In addition, some homes have 12-100 Mbps connections, but the majority of homes are still at slow speeds and HTTP also works at low speeds.

  One alternative is to host the web browser for the app / game server 1521-1525 and host the files for the web server of the RAID array 1511-1512 (or the app / potentially hosting the web browser). in the RAM or local storage of the game server 1521-1525). Because there is a very fast interconnect through inbound routing 1502 (or to local storage), there is a minimum latency per file using HTTP, rather than a 100 ms latency per file using HTTP. It will be time. Thus, instead of a user at home accessing the web page through HTTP, the user can access the web page through client 415. Then, with a 1.5 Mbps connection (since this web page does not require significant bandwidth for the video), the web page will be live in less than 1 second per line 2400. In essence, there is no waiting time for the web browser running on the app / game server 1521-1525 to display the raw page, and a detectable wait until the client 415 displays the video output from the web browser. no time. As the user manipulates and / or types on the web page, the user's input information is sent to the web browser running on the app / game server 1521-1525, and the web browser responds accordingly.

  One drawback of this solution is that bandwidth is used even if the web page is static when the compressor always sends video data. This can be improved by configuring the compressor to send data only when (and only when) the web page changes, and then send data only to the portion of the page that has changed. There are some web pages with constantly changing flashing banners, etc., but such web pages tend to be annoying and usually web pages are quiet unless there is a reason for something to move (eg, a video clip). It will be something like that. For such a web page, perhaps less data is transmitted using the hosting service 210 than a conventional web server. This is because only the video that is actually displayed is transmitted, and neither the thin client execution code nor large objects that are never seen, such as rollover video, are transmitted.

  Thus, using the hosting service 210 to host a legacy web page, the load time of the web page can be reduced to the point that opening a web page is the same as switching a television channel, ie Web pages are actually live instantly.

Ease of Debugging Games and Applications As mentioned above, video games and applications with real-time graphics are very complex applications and typically include bugs when released to the market. Software developers have some means to get user feedback on bugs and reduce machine state after a crash, but strictly identify what happened to the game or real-time application that caused it to crash or function improperly It is very difficult to do.

  When a game or application is executed on the hosting service 210, the video / audio output of the game or application is constantly recorded in the delay buffer 1515. In addition, the watchdog process operates each app / game server 1521-1525 and regularly reports to the hosting service control system 401 that the app / game server 1521-1525 operates smoothly. If the watchdog process fails to report, the server control system 401 attempts to communicate with the app / game server 1521-1525 and collects whatever machine state is available if it succeeds. . Whatever information is obtained, it is sent to the software developer along with the video / audio recorded in the delay buffer 1515.

  Thus, when a game or application software developer gets a crash notification from the hosting service 210, he gets a frame-by-frame record of what caused the crash. This information is very valuable for tracking and fixing bugs.

  Also, if the app / game server 1521-1525 crashes, the server is restarted at the most recent restartable point and a message is given to the user apologizing for technical difficulties.

Resource Sharing and Cost Savings The system shown in FIGS. 4a and 4b provides various benefits to both end users and game and application developers. For example, home and office client systems (eg, PCs or game consoles) are typically used only a small amount of time per week. October 5, 2006 by Nielsen Entertainment “Active Gamer Benchmark Study” (http://www.prnewswire.com/cgibin/stories.pl?ACCT=104&STORY=/www/story/10-05-2006/0004446115&EDATE=) According to the press release, aggressive gamers spend an average of 14 hours a week playing video game consoles and handheld about 17 hours a week. The report also states that active gamers spend an average of 13 hours per week for all gameplay activities (including console, handheld and PC gameplay). Considering that console video game play time is a high number, a week is 24 * 7 = 168 hours, which means that in an aggressive gamer's house, the video game console is It means that only 17/168 = 10% is used. That is, the video game console is idle for 90% of the time. If it is a high-cost video game console and the manufacturer subsidizes such a device, it is a very inefficient use of expensive resources. PCs in the company are also typically used for a fraction of the week, especially non-portable desktop PCs that are often required for high-end applications such as Autodesk Maya. Some companies also operate all hours and holidays, and some PCs (eg portables carried to work at home at night) are used for all hours and holidays, but most activities of the company Tends to be centered around Mondays to Fridays at approximately 9am to 5pm in the designated business hours, rarely on holidays and rest times (like lunch), and most PC use Occurs while a user is actively engaged with a PC, so the use of a desktop PC tends to follow these business hours. Assuming that the PC is permanently used for 5 days a week from 9:00 AM to 5:00 PM, this means that the PC is used 40/168 = 24% of the time of the week. High performance desktop PCs are a very expensive investment for a company, reflecting a very low usage level. Schools that teach with desktop computers use computers with less time of the week and vary based on class hours, but most classes take place during the daytime from Monday to Friday. Thus, in general, PCs and video game consoles are only utilized for a small portion of the time of the week.

  In particular, many people are at work or school during the weekday Monday-Friday daytime, so these people generally do not play video games during these hours, and therefore do not play video games. Playing is typically during other times such as nights, weekends or holidays.

  Given the configuration of the hosting service shown in FIG. 4a, the usage patterns described in the two paragraphs above result in a very efficient use of resources. As will be apparent, the user who can be serviced by the hosting service 210 at a given time, especially if the user requests a real-time response to a complex application such as a sophisticated 3D video game. There is a limit to the number. However, unlike home video game consoles or office PCs that are typically idle most of the time, the server 402 can be reused by different users at different times. For example, a high-performance server 402 with a high-performance dual CPU and dual GPU and large-capacity RAM can be used by companies and schools from 9 am to 5 pm on weekdays, but at night, weekends and holidays It can be used by gamers who play sophisticated video games. Similarly, low-performance applications may have a Celeron CPU, no GPU (or very low-end GPU), and low-performance servers 402 with limited RAM by companies and schools during the company's time. A low performance game can be utilized and the low performance server 402 can be utilized when it is not during company time.

  Furthermore, the hosting service configuration described herein efficiently shares resources among thousands of users, even if not millions. In general, online services have only a small percentage of the total user base that uses the services at a given time. Given the above Nielsen video game usage statistics, it's easy to see why. If an aggressive gamer plays a console game for only 17 hours per week, the peak usage time of the game is at night (5-AM12, 7 * 5 days = 35 hours / week) and weekends (AM8-AM12, 16 * 2 = 32 hours / week) Assuming typical non-working non-business hours, there are 35 + 32 = 65 peak hours per week for 17 hours of game play. The exact peak user load on the system is difficult to estimate for a number of reasons: some users play during off-peak hours, users have collective peaks at certain times of the day, and peak times play Influenced by the type of game (eg, children's games are probably played early in the night), and so on. However, if the average number of hours a gamer plays is much less than the number of hours a day the gamer probably plays the game, only a fraction of the number of users of the hosting service 210 will use it at any given time. is there. For this analysis, it must be assumed that the peak load is 12.5%. Thus, 12.5% of the computational, compression and bandwidth resources are only used at a given time, so that a given performance level is played by reusing resources to play a given performance level game. Supporting users is only 12.5% of the hardware cost.

  In addition, given that certain games and applications require more computing power than others, resources can be dynamically allocated based on the game played by the user or the application being executed. Thus, a user selecting a low performance game or application is assigned a low performance (cheap) server 402 and a user selecting a high performance game or application is assigned a high performance (more expensive) server 402. It is done. In fact, a given game or application has a low-performance segment and a high-performance segment of the game or application, and the user retains the user's behavior on the lowest cost server 402 that satisfies the game or application needs. As such, it is possible to switch from one server 402 to another between game or application segments. A RAID array 405 that is much faster than a single disk can also be used with a low performance server 402 to benefit from high disk transfer rates. Thus, the average cost per server 402 across all games played or applications used is significantly lower than the cost of the most expensive server 402 playing the highest performance game or application, and the low performance server 402 The disk performance benefits are derived from the RAID array 405.

  Furthermore, the server 402 of the hosting service 210 is not more than a PC motherboard with no disk or peripheral interface other than the network interface, but will eventually be integrated into a single chip with only a high-speed network interface to the SAN 403. . Also, the RAID array 405 is probably shared among many more users than disks, so the disk cost per active user is much lower than a single disk drive. All of this equipment is probably in a rack in an environmentally controlled server room environment. If the server 402 fails, the hosting service 210 can easily repair or replace it. In contrast, a PC or game console in a home or office must be able to survive moderate wear and tear by being struck or dropped, requires a housing and has at least one disk drive. Sold by retailers that must survive bad environmental conditions (eg, packed into overhead AV cabinets with other gear), require service guarantees, must be packaged and shipped, and possibly collect retail margins Must be stubborn stand-alone equipment. In addition, a PC or game console is expected to be the most computationally strong game or application used at some point in the future, even if a low performance game or application (or game or application segment) is played most of the time. Must be configured to satisfy the peak performance of And when a PC or console fails, repairing it is an expensive and time consuming process that adversely affects manufacturers, users, and software developers.

  Thus, if the system shown in FIG. 4a gives the user an experience comparable to local computing resources, the architecture shown in FIG. 4a is required for a home, office or school user to experience a given level of computing power. It is quite inexpensive to give its computing power through.

Eliminating the need for upgrade In addition, users no longer have to worry about upgrading their PCs and / or consoles to play new games or handle new applications with higher performance. A game or application at the hosting service 210 is available to the user regardless of what type of server 402 is required for that game or application, and all games and applications are almost instantaneous (ie, , Loaded quickly from the RAID array 405 or local storage on the server 402) and properly executed with the latest updates and bug fixes (ie, the software developer runs the server 402 running a given game or application) The server configuration with the optimal driver can then be configured, and over time, the developer can update all copies of the game or application in the hosting service 210, It can give grayed repair, or the like at a time). In fact, after the user starts using the hosting service 210, the user will probably find games and applications that will continue to provide excellent experience (eg, through updates and / or bug fixes), and the user will be a year ago Discovers that a new game or application is available for a service 210 that uses a computing technology (eg, high performance GPU) that did not exist in 1 year, and thus plays the game or runs the application after 1 year You will discover that it was impossible for users to buy technology a year ago. The computing resource that plays the game or runs the application is invisible to the user (i.e., from the user's perspective, the user starts the game almost immediately, like switching TV channels) The user's hardware is “upgraded” without the user being constantly aware of upgrades.

Eliminating the Need for Backup Another major problem for users in companies, schools and homes is backup. Information stored on the local PC or video game console (eg, in the case of a console, the user's game performance and ranking) is lost if the disc fails or is accidentally erased. Many applications that provide manual or automatic backups for PCs are available and the game console state can be uploaded to an online server for backup, but local backups are typically separate local disks (or other non-volatile). Must be stored and organized somewhere safe, and backups to online services are available slowly over a typical low-cost Internet connection. Because of speed, there are often limitations. In the hosting service 210 of FIG. 4a, the data stored in the RAID array 405 is not lost when the disk fails using conventional RAID configuration techniques well known to those skilled in the art. The server center specialist who has received the disk is notified and can be configured to replace the disk and automatically update the RAID array to fail again. In addition, since all the disk drives are close to each other and there is a high speed local network between them through the SAN 402, all disk systems are regularly backed up at the server center with secondary storage, and this secondary storage. It is not difficult to store the device at the server center or move it away. From the user's point of view of the hosting service 210, the data is simply always secure and there is no need to think about backups.

Demo access users often want to try a game or application before purchasing it. As mentioned above, there are traditional means of demonstrating games and applications (the verb form of “demo” means to try a demonstration version, which is also called “demo” as a noun), Each of them is plagued with limitations and / or inconvenience. Using the hosting service 210 makes it easy and convenient for users to try out demos. In fact, all users try the demo by selecting a mode through the user interface (as described below). The demo loads almost instantly to the server 402 suitable for the demo and only runs like any other game or application. Regardless of whether the demo requires a very high performance server 402 or a low performance server 402, the user can use any form of home or office client 415 from the user's perspective. Works properly. Game or application demo software publishers have precise control over what demos are allowed and how long users can try, and of course, the demos have the opportunity to access the full version of the game or application being demonstrated. User interface elements that provide the user with

  Demos are probably offered below or at no cost, so some users try to use demos (especially game demos that are fun to play repeatedly). The hosting service 210 can use various techniques that limit the use of demos for a given user. The simplest solution is to establish a user ID for each user and limit the number of times a given user ID is allowed to play the demo. However, the user can set a plurality of user IDs when the user ID is free. One technique that addresses this problem is to limit the number of times a given client 415 is allowed to play a demo. If the client is a stand-alone device, the device has a serial number and the hosting service 210 can limit the number of times a demo can be accessed by a client with that serial number. When the client 415 is executed as software of a PC or other device, a serial number can be specified by the hosting service 210 and stored in the PC, and this can be used to restrict the use of the demo. A record of the PC network adapter "Media Access Control (MAC)" address (and / or other machine-specific identifier, eg, hard drive serial number, etc.), if it can be reprogrammed and the serial number can be erased or changed Another option is provided for the hosting service 210 to keep and limit the use of demos for it. However, if the MAC address of the network adapter can be changed, this is not a foolproof method. Another solution is to limit the number of times a demo can be played for a given IP address. The IP address can be periodically reassigned by the cable modem and DSL provider, but in practice it is not done very often and if the IP can be determined to be in a block of IP addresses for residential DSL or cable modem access A small number of demos can typically be established for a given home (eg, by contacting an ISP). There may also be multiple devices behind NAT routers that share the same IP address at home, but typically there are only a limited number of such devices in a residential setting. Multiple demos can be established for one company if the IP address is in a block serving the company. But in the end, the best way to limit the number of demos on a PC is a combination of all the solutions described above. There is no foolproof way for determined technically skilled users to limit the number of demos that are played repeatedly, but by creating a large number of barriers, troublesome PC users can abuse the demo system. Is not worth it, but rather can create enough deterrent so that demos can be used as they are intended to try out new games and applications.

Benefits for schools, companies and other facilities, particularly for companies, schools and other facilities that utilize the system shown in FIG. 4a. Companies and schools have significant costs associated with installing, maintaining and upgrading PCs, especially when it comes to PCs for running high performance applications such as Maya. . As mentioned above, PCs are generally used for only a small portion of the time of the week, and as in the home, the cost of a PC with a given level of performance capability is in an office or school environment. Is much higher than the server center environment.

  For large companies or schools (eg, large universities), it is practical for the IT department of such an entity to set up a server center and maintain computers that are accessed remotely over a LAN grade connection. is there. There are many solutions for remotely accessing a computer via a LAN or through a private high bandwidth connection between offices. For example, on a Microsoft Windows terminal server, or through a virtual network computing application such as VNC from RealVNC, or through a thin client means from Sun Microsystems, the user can use a PC with a range of quality in graphic response time and user experience. You can get remote access to the server. In addition, such self-managed server centers are typically dedicated to a single company or school, so heterogeneous applications (eg, entertainment and company applications) use the same computing resources at different times of the week. It cannot take advantage of the duplication of use that can be considered when using it. Thus, many companies and schools lack the scale, resources or expertise to set up a server center with a LAN speed network connection to each user. In fact, most schools and businesses have the same Internet connection (eg, DSL, cable modem) as the home.

  Such organizations still have a need for very high performance calculations on either a steady or periodic basis. For example, a small building company has a small number of architects and needs relatively little computation when doing design work, but periodically requires very high performance 3D calculations (eg, , When creating new architectural design 3D flythroughs for clients). The system shown in FIG. 4a is very well suited for such tissues. Organizations do not need more than the same type of network connection (eg, DSL, cable modem) that is typically very inexpensive and provided to the home. They either use an inexpensive PC as the client 415 or use an inexpensive dedicated device that simply omits the PC and easily implements the control signal logic 413 and the short latency video decompression 412. . These features are particularly attractive for schools that have problems with PC theft and damage to dedicated components in the PC.

  Such a configuration solves a number of problems in such organizations (and many of these effects are also shared by home users performing common calculations). For one, the operating costs (which ultimately must be returned to the user in some form to obtain a feasible transaction) can be very low. This is because (a) compute resources are shared with other applications that have different peak usage times during the week, and (b) organizations access (and incur costs of) high performance compute resources only when needed. (C) the organization does not need to provide resources to back up or maintain high performance computing resources.

Eliminating piracy Furthermore, games, applications, interactive movies, etc. are no longer subject to piracy as they are today. Since the game is run at the service center, the user is not given access to the basic program code, and therefore nothing is pirated. Even if the user copies the source code, the user cannot execute the code on a standard game console or home computer. This opens the market in world places like China where standard video games are not available. Also, second-hand games cannot be resold.

  For game developers, there are few discontinuities in the market today. Hosting service 210 is a game requirement, as opposed to the current situation where a completely new generation of technology forces users and developers to upgrade and game developers rely on timely delivery of hardware platforms. As time changes, it can be gradually updated over time.

Streaming interactive video The discussion above is a new basic concept of low latency streaming interactive video based on the general Internet (which also includes the audio used here implicitly with the video) Describes the wide range of applications that are possible. Conventional systems that provide streaming video over the Internet only allow applications that can be implemented with long latency interactivity. For example, playback controls for linear video (eg pause, rewind, fast forward) work well with long latency and can be selected from linear video feeds. And as noted above, the nature of certain video games allows them to be played with a long latency. However, the long latency (or low compression ratio) of conventional solutions for streaming video severely limits the potential applications of streaming video or narrows their development to specialized network environments, and Even in such an environment, the prior art introduces a substantial burden on the network. The technology described here opens the door to a wide range of applications that can be considered for low-latency streaming interactive video through the Internet, especially those enabled through consumer-grade Internet connections.

  A client as small as the client 465 of FIG. 4c, sufficient to provide an improved user experience with virtually any amount of computing power, any amount of fast storage, and a very fast network in a powerful server The device actually enables a new era of computation. Furthermore, since bandwidth requirements do not increase as the computing power of the system increases (ie, bandwidth requirements are only tied to display resolution, quality and frame rate), broadband Internet connections are ubiquitous ( Typical consumer and corporate applications when reliable and wide enough to meet the needs of all users' display devices 422 (eg, through widespread, low latency wireless coverage) On the other hand, the question is whether a thick client (such as a PC or mobile phone running Windows, Linux, OSX, etc.) is necessary or a thin client (such as Adobe Flash or Java) may be used.

  The advent of streaming interactive video has led to rethinking assumptions about the structure of the computational architecture. One example is the server center embodiment of the hosting service 210 shown in FIG. The video path for the delay buffer and / or group video 1550 can be selected in such a way that the multicast streaming bi-directional video output of the app / game server 1521-1525 is selectable in real time via path 1552 or via path 1551. Later, it is fed back to the app / game server 1521-1525. This allows a wide range of practical applications (such as those shown in FIGS. 16, 17 and 20) that are not possible or feasible with conventional servers or local computing architectures. However, as a more general architectural feature, what feedback loop 1550 provides is iteration at the streaming interactive video level. This is because the video can be looped back indefinitely when requested by the application. This makes it possible to realize a wide range of application possibilities not previously available.

  Another important architectural feature is that the video stream is a unidirectional UDP stream. This effectively allows for streaming bi-directional video multicasting to any degree (in contrast, bi-directional streams of TCP / IP streams are increasingly congested from front-to-back communication to the network as the number of users increases. Will be generated). Multicasting is an important capability within a server center. This is because the system enables one-to-many or many-to-many communication in response to the increasing needs of Internet users (and indeed the world population). Again, the example described for FIG. 16, which shows the use of both streaming interactive video repetition and multicasting, is just the top of a very large iceberg for possibilities.

  In one embodiment, the various functional modules shown here and associated steps are performed by a specific hardware component, such as an application specific integrated circuit (ASIC), that includes fixed wiring logic to perform the steps. Or performed by a combination of programmed computer components and custom hardware components.

  In one embodiment, these modules are embodied in a programmable digital signal processor (DSP) of Texas Instruments' TMS320x architecture (eg, TMS320C6000, TMS320C5000, etc.). A variety of different DSPs that conform to these basic principles can be used.

  These embodiments can include the various steps described above. These steps can be implemented in machine-executable instructions that cause a general purpose or special purpose processor to perform several steps. Various elements not related to these basic principles, such as computer memory, hard drives, and input devices, have been omitted from the drawings in order not to obscure the appropriate aspects.

  The gist elements disclosed herein may be provided as a machine-readable medium for storing machine-executable instructions. Machine readable media can be flash memory, optical disc, CD-ROM, DVD ROM, RAM, EPROM, EEPROM, magnetic or optical card, propagation media suitable for storing electronic instructions, or other types of machines Including but not limited to readable media. For example, the present invention transfers from a remote computer (eg, server) to a requesting computer (eg, client) via a data signal implemented on a carrier wave or other propagation medium over a communication link (eg, a modem or network connection). Can be downloaded as a computer program.

  Elements of the subject matter disclosed herein also include machine-readable media that store instructions used to program a computer (eg, a processor or other electronic device) to perform a series of operations. It should be understood that it may be provided as a computer program product. Alternatively, the operation may be performed by a combination of hardware and software. Machine-readable media suitable for storing floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROM, RAM, EPROM, EEPROM, magnetic or optical cards, electronic instructions Including, but not limited to, propagation media or other types of media / machine readable media. For example, an element of the presently disclosed subject matter is that a program is transferred from a remote computer or electronic device to a requesting computer by a data signal implemented on a carrier wave or other propagation medium over a communication link (eg, a modem or network connection). Can be downloaded as a computer program product.

  Furthermore, while the subject matter disclosed herein has been described with reference to specific embodiments, numerous changes and modifications can be made within the scope of the present disclosure. Accordingly, the present invention and the accompanying drawings are merely illustrative and are not limiting.

100: CPU / GPU
101: RAM
102: Display device 103: Hard drive 104: Optical media drive 105: Network connection 106: Game controller 205: Client device 206: Internet 210: Hosting service 211: User's house 220: Software developer 221: Input device 222: Monitor Or TV receiver 301: Maximum data rate 302: Maximum data rate that can be actually used 303: Requested data rate 401: Hosting service control system 402: Server 403: SAN
404: Low latency video compression 405: RAID array 406: Control signal 410: Internet 412: Low latency decompression 413: Control signal logic 415: Home or office client 421: Input device 422: Monitor or HDTV
441: Central office, head end, cell tower, etc. 442: WAN interface 443: Firewall / Router / NAT
444: WAN interface 451: Control signal 452: User's house routing 453: User ISP
454: Internet 455: Server center routing 456: Frame calculation 457: Video compression 458: Video decompression 462: Power over Ethernet 463: HDMI output 464: Display capability 465: Ethernet vs. HDMI client 466: Glasses with shutters 468: Monitor or SD / HDTV
469: Bluetooth input device 476: Flash 480: Bus 481: Ethernet interface 483: Control CPU
484: Bluetooth 486: Video decompressor 487: Video output 488: Audio decompressor 489: Audio output 490: HDMI
497: Ethernet 499: Power

Claims (3)

Subdividing the program code and / or data used to execute the online application into a first format and a second format;
Storing the first form of program code and data in a first form of memory that provides relatively low latency memory access;
Storing the second form of program code and data in a second form of memory that provides a relatively long latency memory access compared to the first form of memory;
Retrieving program code and data from the first memory and the second memory in response to a client request to execute an online application;
Sending a streaming interactive video stream representing the video generated by the application to the client;
A computer-implemented method comprising:   The method of claim 1, wherein the first type of memory is a flash memory and the second type of memory is a hard drive.   The method of claim 1, wherein the application is a video game.