CN102989174B - Obtain the input being used for controlling the operation of games - Google Patents

Abstract

The method disclosing the input obtaining the operation for controlling game.In an embodiment of the present invention, before the analysis of gesture recognition, the controller routing information from inertia, picture catching and sound source can be mixed.

Description

Obtain the input being used for controlling the operation of games

Priority request

This application claims the rights and interests of following patent: U.S. Patent application No.11/381729, authorize XiaoDongMao, title is " microminiature microphone array ", (attorney docket SCEA05062US00), and on May 4th, 2006 submits to；Application number 11/381728, authorizes XiaoDongMao, and title is " echo and noise eliminate ", and (attorney docket SCEA05064US00), on May 4th, 2006 submits to；U.S. Patent application No.11/381725, authorizes XiaoDongMao, and title is " method and apparatus of target sound detection ", and (attorney docket SCEA05072US00), on May 4th, 2006 submits to；U.S. Patent Application No. 11/381727, authorizes XiaoDongMao, and title is " noise remove on control station with the electronic installation of far field microphone ", (attorney docket SCEA05073US00), and on May 4th, 2006 submits to；U.S. Patent application No.11/381724, authorizes XiaoDongMao, and title is " target sound detection and the method and apparatus characterized ", and (attorney docket SCEA05079US00), on May 4th, 2006 submits to；U.S. Patent application No.11/381721, authorizes XiaoDongMao, and title is " selective sound source listening in conjunction with computer interaction process ", (attorney docket SCEA04005JUMBOUS), and on May 4th, 2006 submits to；By reference all of which is incorporated into herein.

This application claims the rights and interests of following patent: CO-PENDING application number 11/418988, authorize XiaoDongMao, title is " adjusting the method and apparatus being used for catching the audit area of sound ", (attorney docket SCEA-00300), and on May 4th, 2006 submits to；CO-PENDING application number 11/418989, authorizes XiaoDongMao, and title is " method and apparatus for catching audio signal according to visual image ", and (attorney docket SCEA-00400), on May 4th, 2006 submits to；CO-PENDING application number 11/429047, authorizes XiaoDongMao, and title is " according to the method and apparatus that the position of signal catches audio signal ", (attorney docket SCEA-00500), and on May 4th, 2006 submits to；CO-PENDING application number 11/429133, authorizes RichardMarks et al., and title is " selective sound source listening in conjunction with computer interaction process ", (attorney docket SCEA04005US01-SONYP045), and on May 4th, 2006 submits to；And CO-PENDING application number 11/429414, authorize RichardMarks et al., title is " with the computer picture of the intensity of computer program interface and input equipment and Audio Processing ", (attorney docket SONYP052), and on May 4th, 2006 submits to；By reference their whole complete disclosure are incorporated herein in.

The application also requires the rights and interests of following patent: U.S. Patent application No.11/382031, and title is " multi-input game control mixer ", (attorney docket SCEA06MXR1), and on May 6th, 2006 submits to；U.S. Patent application No.11/382032, title is " for the system that the user in tracking environmental handles ", and (attorney docket SCEA06MXR2), on May 6th, 2006 submits to；U.S. Patent application No.11/382033, title is " for the system of three-dimensional input control, method and apparatus " ", (attorney docket SCEA06INRT1), on May 6th, 2006 submits to；U.S. Patent application No.11/382035, title is " inertia can follow the tracks of hand held controller ", (attorney docket SCEA06INRT2), and on May 6th, 2006 submits to；U.S. Patent application No.11/382036, title is " method and system for visual tracking application connected effect ", and (attorney docket SONYP058A), on May 6th, 2006 submits to；U.S. Patent application No.11/382041, title is " for inertia is followed the tracks of the method and system of application connected effect ", and (attorney docket SONYP058B), on May 7th, 2006 submits to；U.S. Patent application No.11/382038, title is " being used for the method and system to acoustic tracking application connected effect ", and (attorney docket SONYP058C), on May 6th, 2006 submits to；U.S. Patent application No.11/382040, title is " being used for the method and system to multichannel Mixed design application connected effect ", and (attorney docket SONYP058D), on May 7th, 2006 submits to；U.S. Patent application No.11/382034, title is " scheme that the user for detecting and follow the tracks of game controller body handles ", and (attorney docket 86321SCEA05082US00), on May 6th, 2006 submits to；U.S. Patent application No.11/382037, title is " for the movement of hand held controller converts to the scheme of the input of system ", and (attorney docket 86324), on May 6th, 2006 submits to；U.S. Patent application No.11/382043, title is " can detect and can follow the tracks of hand held controller ", (attorney docket 86325), and on May 7th, 2006 submits to；U.S. Patent application No.11/382039, title is " method for the movement of hand held controller is mapped to game commands ", and (attorney docket 86326), on May 7th, 2006 submits to；S Design Patent application No.29/259349, title is " controller with infrared port ", (attorney docket SCEA06007US00), and on May 6th, 2006 submits to；S Design Patent application No.29/259350, title is " controller with tracking transducer ", (attorney docket SCEA06008US00), and on May 6th, 2006 submits to；U.S. Patent application No.60/798031, title is " dynamic object interface ", (attorney docket SCEA06009US00), and on May 6th, 2006 submits to；And S Design Patent application No.29/259348, title is " tracked control device ", (attorney docket SCEA06010US00), and on May 6th, 2006 submits to；U.S. Patent application No.11/382250, title is " obtaining the input being used for controlling the operation of games ", and (attorney docket SCEA06018US00), on May 8th, 2006 submits to；By reference all of which is intactly incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/430594, authorize GarzZalewski and RileyR.Russel, title is " audio visual environment of use user selects the system and method for advertisement ", (attorney docket SCEA05059US00), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/430593, authorize GarzZalewski and RileyR.Russel, title is " using audio visual environment to select advertisement on gaming platform ", (attorney docket SCEAUS3.0-011), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/382259, authorize GarzZalewski et al., title is " for determining not relative to the method and apparatus of the User Activity of system ", (attorney docket 86327), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/382258, authorize GarzZalewski et al., title is " method and apparatus for determining the User Activity grade relative to system ", (attorney docket 86328), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/382251, authorize GarzZalewski et al., title is " having can the hand held controller of detecting element for what follow the tracks of ", and (attorney docket 86329), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/382252, title is " for obtaining the tracking device controlling the information that games run ", (attorney docket SCEA06INRT3), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: co-pending U.S. Patent application number 11/382256, title is " having the tracking device for obtaining the acoustic emitter controlling the information that games run ", (attorney docket SCEA06ACRA2), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246744, and title is " PlayStation 3 videogame console/PS3 front ", (attorney docket SCEACTR-D3), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246743, and title is " PlayStation 3 videogame console/PS3 ", (attorney docket SCEACTRL-D2), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246767, and title is " PlayStation 3 videogame console/PS3 ", (attorney docket SONYP059A), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246768, and title is " PlayStation 3 videogame console/PS3 ", (attorney docket SONYP059B), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246763, title is " having the ergonomics game controller apparatus of LED and optical port ", (attorney docket PA3760US), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246759, and title is " game controller apparatus with LED and optical port ", (attorney docket PA3761US), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246765, and title is " design of optics game controller interface ", (attorney docket PA3762US), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246766, title is " having the dual-handle game control device of LED and optical port ", (attorney docket PA3763US), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246764, and title is " the game interface device with LED and optical port ", (attorney docket PA3764US), and on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The application also requires the rights and interests of following patent: copending United States design patent application number 29/246762, title is " having the ergonomics game interface device of LED and optical port ", (attorney docket PA3765US), on May 8th, 2006 submits to；By reference its complete disclosure is incorporated herein in.

The cross reference of related application

The application relates to the U.S. Provisional Patent Application No.60/718145 that title is " audio frequency, video, simulation and user interface example " of JIUYUE in 2005 submission on the 15th, it is incorporated into by reference herein.

The application relates to following patent: U.S. Patent application No.10/207677, and title is " man-machine interface using deformable device ", and on July 27th, 2002 submits to；U.S. Patent application No.10/650409, title is " audio input system ", and on August 27th, 2003 submits to；U.S. Patent application No.10/663236, title is " method and apparatus for adjusting shown picture view according to tracked head movement ", and within 2003, JIUYUE is submitted on the 15th；U.S. Patent application No.10/759782, title is " method and apparatus for light Line Input Devices ", and on January 16th, 2004 submits to；U.S. Patent application 10/820469, title is " method and apparatus of detection and removal audio frequency disturbance ", and on April 7th, 2004 submits to；And U.S. Patent application No.11/301673, title is " following the tracks of the method using opposing headers and hand position to realize instruction interface via camera ", and within 2005, December is submitted on the 12nd；U.S. Patent application No.11/165473, title is " delay matching of audio-frequency/video frequency system ", and on June 22nd, 2005 submits to；By reference all of which is hereby incorporated by.

The application further relates to following patent: co-pending U.S. Patent application No.11/400997, and on April 10th, 2006 submits to, and title is " being used for the system and method from phonetic acquisition user profile ", (attorney docket SCEA05040US00)；By reference its complete disclosure is incorporated herein in.

Technical field

In general, the present invention relates to man-machine interface, it particularly relates to process the multichannel input that the user for following the tracks of one or more controller handles.

Background technology

Computer entertainment system generally includes hand held controller, game console or other controller.User or player use controller to send order or other instruction to entertainment systems, in order to control video-game or other simulation played.Such as, controller can be equipped with by the manipulator of user operation, such as stick.The variable that is manipulated by of stick converts digital value to from the analogue value, and this digital value is sent to game host.Controller also can be equipped with can by the button of user operation.

The present invention has been developed just for these and other background information factors.

Accompanying drawing explanation

In conjunction with accompanying drawing by with reference to described in detail below, it can be readily appreciated that the theory of the present invention, accompanying drawing includes:

Fig. 1 is the pictorial diagram illustrating the video game system being operated according to one embodiment of present invention；

Fig. 2 is the perspective view of the controller made according to one embodiment of present invention；

Fig. 3 is the schematic three dimensional views of the accelerometer illustrating according to one embodiment of present invention, can be used for controller；

Fig. 4 is according to one embodiment of the invention, for mixing the block diagram of the various system controlling input；

Fig. 5 A is the block diagram of a part for the video game system of Fig. 1；

Fig. 5 B be according to one embodiment of present invention, for following the tracks of the flow chart of the method for the controller of video game system；

Fig. 5 C is the flow chart illustrating according to one embodiment of present invention, utilizing during the game on video game system carries out the method for position and/or orientation information；

Fig. 6 is the block diagram illustrating video game system according to an embodiment of the invention；And

Fig. 7 is the block diagram of the Cell processor realization of video game system according to an embodiment of the invention.

Specific embodiment describes

Although for the ease of illustrating, described in detail below comprising many details, however it will be understood by those of ordinary skill in the art that, many changes and change to details below are within the scope of the present invention.It is therefore proposed that the example embodiment of the following description of the present invention, and do not lose the generality of the present invention requiring rights and interests and the present invention requiring rights and interests is not applied restriction.

The various embodiments of method described herein, equipment, scheme and system provide the user's detection of movement, motion and/or manipulation, seizure and tracking to whole controller main body itself.User is mobile to being detected of whole controller main body, move and/or handle the various aspects that can be used for controlling as additional command game or other simulation carried out.

The step of the manipulation of game controller body can be realized by detection and tracking user by different modes.Such as, for instance the image capture unit such as the inertial sensor such as accelerometer or gyroscope, such as digital camera can with computer entertainment system with the use of, in order to detection hand held controller main body motion, and be converted into game in action.Such as follow the tracks of the example of the controller with inertial sensor at title described in the U.S. Patent application 11/382033 (attorney docket SCEA06INRT1) of " system that three-dimensional input controls, method and apparatus ", be incorporated into by reference herein.Such as use picture catching to carry out the example of tracking control unit at title described in the U.S. Patent application 11/382034 (attorney docket SCEA05082US00) of " scheme that the user for detecting and follow the tracks of game controller body handles ", be incorporated into by reference herein.It addition, it be also possible to use microphone array and suitable signal processing acoustically tracking control unit and/or user.In the example of this acoustic tracking described in U.S. Patent application 11/381721, it is incorporated into by reference herein.

Phonoreception is surveyed, inertia sensing and picture catching can individually or with any combination for detecting the many different types of motion of controller, for instance moves up and down, reverse and move, move left and right, jerk movement, bar type motion, underriding campaign etc..This type games may correspond to various order so that motion is converted into the action in game.The manipulation of game controller body be can be used to realize many different types of game, simulation etc. by detection and tracking user; this allows user such as to participate in daggers and swords or the fight of light sword; use the rod and follow the tracks of the shape of article; participate in many different types of competitive sports, participate in the fight on screen or other antagonism etc..Games can be configured to the motion of tracking control unit, and identifies some pre-recorded posture from tracked motion.One or more identification in these postures can trigger the change of game state.

In an embodiment of the present invention, can mix, before for the analysis of gesture recognition, the controller routing information obtained from these separate sources.The tracking data from separate sources (such as sound, inertia and picture catching) can be mixed by the mode of the probability of improvement identification posture.

With reference to Fig. 1, it is shown that the system 100 being operated according to one embodiment of present invention.As it can be seen, computer entertainment control station 102 can couple with TV or other video display units 104, in order to show the image of video-game or other simulation wherein.Game or other simulation are storable in inserting on the DVD of control station 102, CD, flash memory, USB storage or other storage medium 106.User or player 108 direct game controller 110 control video-game or other simulation.In fig. 2, it is seen that game console 110 includes inertial sensor 112, it responds the change of position of game console 110, motion, orientation or orientation and produces signal.Except inertial sensor, game console 110 may also include conventional control input equipment, for instance stick 111, button 113, R1, L1 etc..

In operation, user 108 mobile controller 110 for physically.Such as, controller 110 can be moved towards any direction by user 108, for instance upper and lower, to side, to opposite side, torsion, roll, rock, jerk, underriding etc..These of controller 110 itself move can by camera 112 by via analyze be tracked from the signal of inertial sensor 112, the mode that is described below detects and catches.

Referring again to Fig. 1, system 100 can include camera or other video image trap setting 114 alternatively, and it can be positioned such that controller 110 is within the visual field 116 of camera.Analysis from the image of image capture device 114 can be combined use with the analysis of the data from inertial sensor 112.As in figure 2 it is shown, controller 110 can be optionally equipped with the light sources such as such as light emitting diode (LED) 202,204,206,208, to help to be tracked by video analysis.They may be mounted in the main body of controller 110.Term as used herein " main body " is used for describing in game console 110 part that will grasp (or wearing when it is wearable game console).

Such as authorizing inventor GaryM.Zalewski, title analysis to this kind of video image for tracking control unit 110 described in the U.S. Patent Application No. 11/382034 (attorney docket SCEA05082US00) of " scheme that the user for detecting and follow the tracks of game controller body handles ", it is incorporated into by reference herein.Control station 102 can include sonic transducer, for instance microphone array 118.Controller 110 may also include acoustical signal maker 210 (such as loudspeaker), thus providing sound source to help to have the acoustic tracking of the controller 110 of microphone array 118 and suitable Underwater Acoustic channels, as described in U.S. Patent application 11/381724, it is incorporated into by reference herein.

In general, position and the orientation data of formation controller 110 it is used for from the signal of inertial sensor 112.This data can be used to many physics aspects of the movement of computing controller 110, for instance any telemetry station of its acceleration along any axle and speed, its inclination, pitching, driftage, rolling and controller 110." remote measurement " used herein is generally referred to as and remotely measures information of interest and to system or to the designer of system or operator's report.

The ability of the movement of detection and tracking control unit 110 enables a determination of whether to perform any predetermined movement of controller 110.It is to say, some Move Mode of controller 110 or posture can pre-define and with playing games or the input order of other simulation.Such as, the downward underriding posture of controller 110 may be defined as an order, and the torsion posture of controller 110 may be defined as another order, and the posture of rocking of controller 110 may be defined as another order, and the rest may be inferred.So, the mode of user 108 mobile controller 110 for physically controls another input of game with acting on, and it provides the user more stimulates happier experience.

Exemplarily rather than restriction, inertial sensor 112 can be accelerometer.Fig. 3 illustrate take such as to be determined by the simple quality of spring 306,308,310,312 and framework 304 Elastic Coupling at four points 302 the example of accelerometer 300 of form.Pitch axis and roll axis (being represented by X and Y respectively) are arranged in the plane with frame intersection.Yaw axis Z is orientated vertical with the plane comprising pitch axis X and roll axis Y.Framework 304 can be installed to controller 110 by any appropriate ways.When framework 304 (and game console 110) accelerates and/or rotates, quality is determined 302 can relative to framework 304 displacement, and spring 306,308,310,312 can extend in the following manner or compress, translation which depends on pitching and/or rolling and/or driftage and/or the quantity spun up and direction and/or angle.The displacement of mass 302 and/or the compression of spring 306,308,310,312 or elongation can adopt such as suitable sensor 314,316,318,320 to sense, and are converted into known or that predetermined way is relevant to the acceleration amount of pitching and/or rolling signal.

There is many different modes come tracking quality position certainly and/or be applied to power thereon, including strain ga(u)ge material, photon sensor, Magnetic Sensor, Hall effect device, piezo-electric device, capacitance sensor etc..Embodiments of the invention can include the sensor of any quantity and type or the combination of sensor type.By example rather than restriction, sensor 314,316,318,320 can be provided in the gap close induction type electrode on mass 302.Electric capacity between mass and each electrode changes with the mass position relative to each electrode.Each electrode may be connected to circuit, and this circuit produces the signal relevant relative to the electric capacity of electrode (therefore with mass relative to the nearness of electrode) to mass 302.It addition, spring 306,308,310,

312 can include resistance-strain flowmeter sensor, and they produce the signal relevant to the compression of spring and elongation.

In certain embodiments, framework 304 can be installed to controller 110 with gimbal so that accelerometer 300 keeps fixed orientation relative to pitching and/or rolling and/or yaw axis.So, controller shaft X, YZ can map directly to the respective shaft in real space, without considering the controller shaft inclination relative to real space coordinate axes.

As it has been described above, the data from inertia, picture catching and sound source can be analyzed, the path of position and/or orientation to generate tracking control unit 110.As illustrated in the diagram in figure 4, system 400 according to an embodiment of the invention can include tracking information 402, image dissector 404 and acoustic analyser 406.Each reception in these analyzers carrys out the signal of self-inductance measurement environment 401.Analyzer 402,404,406 can pass through hardware, software (or firmware) or two of which or more certain combination realize.The tracking information that each generation in analyzer is relevant to the position of concerned object and/or orientation.Exemplarily, concerned object can be controller noted above 110.Image dissector 404 is operated in combinations with the method described in U.S. Patent application 11/382034 (attorney docket SCEA05082US00), form field according to it and it is operated relatively.Tracking information 402 is operated for the method described in the U.S. Patent application 11/382033 (attorney docket SCEA06INRT1) of " system that three-dimensional input controls, method and apparatus " in combinations with title, form field according to it and it is operated relatively.Acoustic analyser 406 is operated in combinations with the method described in U.S. Patent application 11/381,724, form field according to it and it is operated relatively.

The different passages being seen as from the input of position and/or orientation information associate analyzer 402,404 with 406.Blender 408 can accept multiple input channel, and this kind of passage can comprise the sample data characterizing sensing environment 401, generally from the angle of passage.The position of tracking information 402, image dissector 404 and acoustic analyser 406 generation and/or orientation information can be coupled to the input of blender 408.Blender 408 and analyzer 402,404,406 can be inquired about by game software program 410, and can be configured to response events and interrupt Games Software.Event can include gesture recognition event, linkage change, configuration variation, arranges noise grade, arranges sampling rate, change mapping chain etc., and its example is discussed below.Blender 408 is operated in combinations with method described herein, form field according to it and it is operated relatively.

As mentioned above, signal from the such as different input channels of inertial sensor, video image and/or acoustic sensor etc. can be analyzed by tracking information 402, image dissector 404 and acoustic analyser 406 respectively, in order to determine motion and/or the orientation of controller 110 during carrying out video-game according to the inventive method.A series of (aseriesof) processor executable program code instruction that is that this method can be embodied as in processor readable medium storage and that run on digital processing unit.Such as, as shown in Figure 5A, video game system 100 can include the control station 102 with the tracking information 402, image dissector 404 and the acoustic analyser 406 that are realized by hardware or software.Exemplarily, analyzer 402,404,406 can be embodied as the software instruction run on appropriate processor unit 502.Exemplarily, processor unit 502 can be digital processing unit, for instance the microprocessor of common type in video game console.A part for instruction is storable in memorizer 506.Alternatively, tracking information 402, image dissector 404 and acoustic analyser 406 can pass through hardware, such as special IC (ASIC) realizes.This analyzer hardware may be provided on controller 110 or control station 102, or can be remotely located in other position.In hardware realizes, analyzer 402,404,406 can be in response to such as from processor 502 or such as programmable by USB cable, wireless connections or the external signal in other certain remotely located source that connected by network.

Tracking information 402 can include or realize analyzing the signal that inertial sensor 112 generates and the instruction utilizing the information relevant with the position of controller 110 and/or orientation.Similarly, image dissector 404 can realize analyzing the instruction of the image that image capture unit 114 catches.It addition, acoustic analyser can realize analyzing the instruction of the image that microphone array 118 catches.As shown in the flow chart 510 of Fig. 5 B, these signals and/or image can be received by analyzer 402,404,406, as shown in frame 512.Signal and/or image can be analyzed by analyzer 402,404,406, to determine that the inertia relevant with the position of controller 110 and/or orientation is followed the tracks of information 403, image trace information 405 and acoustics and followed the tracks of information 407, as depicted at block 514.Tracking information 403,405,407 can be relevant to one or more degree of freedom.Preferably tracking six degrees of freedom, with the manipulation of characterization control device 110 or other tracked object.This kind of degree of freedom can tilt to the controller along x, y and z axes, go off course, roll and position, speed or acceleration are relevant.

As depicted at block 516, blender 408 mixes Inertia information 403, image information 405 and acoustic information 407, to generate accurate position and/or orientation information (orientationinformation) 409.Exemplarily, blender 408 can inertia, image and acoustics are followed the tracks of, according to game or environmental condition, the weight that information 403,405,407 application is different, and takes weighted average.It addition, blender 408 can include the blender analyzer 412 of its own, analyzer 412 analyzes the position/orientation information of combination, and generates gained " blender " information of its own of the combination of the information comprising other parser generation.

In one embodiment of the invention, Distribution Value can be given the tracking information 403,405,407 from analyzer 402,404,406 by blender 408.As it has been described above, some set that input can control data is averaging.But, in the present embodiment, before input control data are averaging, give certain value to it, thus, the input control data from some analyzer control data than the input from other analyzer and have bigger analysis importance.

Blender 408 can undertake several functions in the context of native system, including observation, correction, stable, derivation, combination, Route Selection, mixing, report, buffering, interrupts other process and analysis.This can perform relative to one or more the received tracking information 403,405,407 from analyzer 402,404,406.Though each some tracking information that receives and/or derive of analyzer 402,404,406, blender 408 can be implemented as the use of the tracking information 403,405,407 that optimization receives, and generate accurate tracking information 409.

Analyzer 402,404,406 is preferably configured with blender 408 becomes the output format that tracking information provides similar.Information parameter of following the tracks of from any analyzer element 402,404,406 maps to the single parameter in analyzer.Alternatively, by processing the one or more one or more tracking informations parameter from analyzer 402,404,406, blender 408 can form the tracking information of any one of analyzer 402,404,406.Blender can be combined two or more elements of the tracking information of the identical parameters type taking from analyzer 402,404,406, and/or the multiple parameters for the tracking information of parser generation perform function, to create the synthesis set of the output with the beneficial effect that the multiple passages from input generate.

Accurate tracking information 409 can use during employing system 100 carries out video-game, as indicated at block 518.In certain embodiments, the posture made of period can be carried out to use position and/or orientation information relative to user 108 in game.In certain embodiments, blender 408 is operated in combinations with gesture recognizers 505, in order at least one action in game environment associated with the one or more user actions (manipulation of the controller in such as space) from user.

As shown in the flow chart 520 of Fig. 5 C, position and/or orientation information can be used to come the path of tracking control unit 110, as shown in frame 522.Exemplarily rather than restriction, this path can include representing the set relative to the point of the position of certain coordinate system of the quality of the controller center certainly.Each position point can be represented by X, the Y in one or more coordinates, such as Cartesian coordinates and Z coordinate.Time can associate with each point on path so that can monitor the shape in path and the controller progress along path.It addition, each point in set can be associated with the data representing the orientation of controller, such as controller around one or more angles of the central rotation of its mass.The each point on path rotates around the angle at the center of its mass and the value of speed of angular acceleration additionally, can be associated with the speed at the center of the mass of controller and acceleration and controller.

As shown in frame 524, it is possible to compared in the path in tracked path with the one or more storages corresponding to known and/or pre-recorded posture 508, these known and/or pre-recorded postures 508 are context-sensitive with the video-game carried out.Evaluator 505 can be configured to identify that user or process audio frequency differentiate posture etc..Such as, user can be identified by posture by evaluator 505, and posture can be that user is specific.This given pose can be recorded and be included among the stored pre-recorded posture 508 of memorizer 506.Recording process can be optionally stored on the audio frequency generated during the record of posture.Sensing environment is sampled in multichannel analyzer and processes.Processor is referred to gesture model with according to voice or sonogram, determine with high accuracy and performance and differentiate and/or identify user or object.

As shown in Figure 5A, represent that the data 508 of posture are storable in memorizer 506.The example of posture includes but not limited to: object-throwing, for instance ball；Swing object, for instance bat or golf club；Suction hand pump；On or off door or window；Steering wheel rotation or other wagon control；Wushu movement, for instance boxing；Sand papering action；Waxing and wax removing；Paint house；Shake hands；Send sound of laughing sound；Rolling；Throw rugby；Swing handle moves；3D mouse moves；Roll mobile；The movement of major profile；Any recordable movement；Along moving back and forth of any vector, i.e. to tyre inflating, but carry out with certain arbitrary orientation in space；Movement along path；There is the movement accurately stopped with the time started；In noise floor, batten, recordable, tracking and the user based on any time repeated handle；Etc..Each in these postures can be pre-recorded from path data and store as time-based model.The comparison of the posture of path and storage can from supposing stable state, if path deviation stable state, then path can be compared by the posture of elimination process Yu storage.At frame 526, without coupling, then at frame 522, analyzer can continue the path of tracking control unit 110.Fully mate if existed between path (or its part) and the posture of storage, then the state played can change, as shown in 528.The change of game state may include but be not limited to interrupt, send control signal, change variable etc..

Here it is it may happen that an example of this situation.When determining that controller 110 has been moved off stable state, the movement of analyzer 402,404,406 or 412 tracking control unit 110.As long as the path of controller 110 meets path defined in the gesture model 508 of storage, then those postures are possible " hits ".If the path of controller 110 (in noise tolerance sets) deviates any gesture model 508, then from hit list, delete that gesture model.Each posture reference model includes the time base of record posture.The posture 508 of controller path data Yu storage is compared by analyzer 402,404,406 or 412 at reasonable time index.The appearance of limit resets clock.When deviateing stable state (that is, when following the tracks of mobile outside noise threshold), hit list is loaded all possible gesture model.Start clock, and the movement of controller and hit list are compared.It is Walkthrough (walkthrough) time more equally.Terminate if any posture in hit list arrives posture, be then hit at first time.

In certain embodiments, blender 408 and/or each analyzer 402,404,406,412 can notify that games are about the time that some event occurs.The example of this kind of event includes the following:

The 1 acceleration point (X and/or Y and/or Z axis) interrupting reaching is in some game situation, and when the acceleration of controller changes in flex point, analyzer can notify or interrupt the routine in games.Such as, user 108 can use controller 110 to carry out the game scapegoat of the quarter back in control representation rugby simulation game.Analyzer can carry out tracking control unit (expression rugby) via the path generated according to the signal from inertial sensor 112.The specific change of the acceleration of controller 110 can signal service.At this moment, analyzer can another routine in trigger (such as physical modeling bag), simulate the track of rugby according to the position of penalty mark place controller and/or speed and/or orientation.

Interrupt the new posture identified

It addition, analyzer can be configured by one or more inputs.The example of this kind of input includes but not limited to:

The reference tolerance used when noise grade (X, Y or Z axis) noise grade can be the shake analyzing the hands of user in game is set.

Sampling rate is set." sampling rate " used herein can refer to the analyzer frequency for being sampled from the signal of inertial sensor.Sampling rate can be set to signal over sampling or is averaging.

Linkage (gearing) is set." linkage " used herein refer generally to controller move with play in the ratio of movement that occurs.The example controlling this " linkage " in the context of video-game is found in the U.S. Patent Application No. 11/382040 (attorney docket No.:SONYP058D) submitted on May 7th, 2006, it is incorporated herein in by reference.

Mapping chain is set." mapping chain " used herein refers to the figure of gesture model.Gesture model figure can be made to be suitable for specific input channel (path data such as only generated) or the hybrid channel formed mixer unit from inertial sensor signal.

Three input channels can be served by two or more different analyzers similar from tracking information 402.Specifically, they comprise the steps that tracking information 402 as described herein, such as authorizing the U.S. Patent application 11/382034 of inventor GaryM.Zalewski, title is " scheme that the user for detecting and follow the tracks of game controller body handles " video analyzer described in (attorney docket SCEA05082US00), it is incorporated herein by reference, and the acoustic analyser described in U.S. Patent application 11/381721 such as incorporated herein by reference.Analyzer can configure with mapping chain.Map chain to be swapped out by game during game carries out, for instance analyzer or blender can be arranged.

Referring again to the frame 512 of Fig. 5 B, it will be appreciated by those skilled in the art that and there is many ways in which and generate signal from inertial sensor 112.This document describes several example therein.With reference to frame 514, there is many ways in which that the sensor signal generated in analysis block 512 is to obtain the tracking information relevant to the position of controller 110 and/or orientation.Exemplarily rather than restriction, tracking information may include but be not limited to individually or information relevant with following parameters in any combination:

Controller orientation.The orientation of controller 110 can according to relative to certain with reference to be orientated pitching (pitch), rolling (roll) or driftage (yaw) angle, such as represent with radian.The rate of change (such as angular velocity or angular acceleration) of controller orientation can be additionally included in position and/or orientation information.Such as, when inertial sensor 112 includes gyrosensor, the controller orientation information of the form of the one or more output valves proportional to the angle of pitching, rolling or driftage can be directly obtained.

Location of controls (Cartesian coordinate X, Y, Z of such as certain referential middle controller 110)

Controller X-axis speed

Controller Y-axis speed

Controller Z axis speed

Controller X-axis acceleration

Controller Y-axis acceleration

Controller Z axis acceleration

It should be noted that can represent according to the coordinate system different from cartesian relative to position, speed and acceleration, position and/or orientation information.Such as, cylinder or spherical coordinate can be used for position, speed and acceleration.Can directly obtain from accelerometer type sensor relative to the acceleration information of X, Y and Z axis, as described herein.X, Y and Z acceleration can for being integrated from the time of certain initial time, to determine the change of X, Y and Z speed.Can pass through to be added the given value of velocity variations with X, Y and Z speed of initial time, calculate these speed.X, Y and Z speed can be integrated for the time, to determine X, Y and the Z displacement of controller.Can pass through to be added displacement known X, Y with initial time and Z location, determine X, Y and Z location.

This customizing messages of stable state Y/N-represents whether controller is in stable state, and it may be defined as any position, it is possible to through change.In a preferred embodiment, stable position can be that controller is maintained at the position of the height substantially flushed with user's waist with about horizontal alignment.

The data that " time from last stable state " is generally referred to as to through how long, section is relevant since stable state (as mentioned above) being detected for the last time.As previously described, the determination of time can in real time, calculate by processor cycle or sampling period.For relative to initial point come reset controller follow the tracks of to guarantee the personage or the accuracy of object that map in game environment, " time from last stable state " can be important.For determining the actions available/posture (foreclose or be included) being likely to run in game environment subsequently, these data also can be important.

" the last posture of identification " is generally referred to as the last posture identified by gesture recognizers 505 (can be realized) by hardware or software.For previous posture can in discernible possible posture subsequently or game environment occur other certain action is relevant the fact, the mark of the last posture of identification can be important.

The time of the last posture identified

By games or software, above-mentioned output can be sampled at any time.

In one embodiment of the invention, Distribution Value can be given the tracking information 403,405,407 from analyzer 402,404,406 by blender 408.As it has been described above, some set that input can control data is averaging.But, in the present embodiment, at the forward direction that input control data are averaging, it gives certain value, and thus, the input control data from some analyzer control data than the input from other analyzer and have bigger analysis importance.

Such as, blender 408 can need the tracking information relevant to acceleration and stable state.Then, blender 408 is by reception tracking information 403,405,407, as mentioned above.Tracking information can include the parameter relevant to acceleration and stable state, for instance the above.Before the data representing this information are averaging, Distribution Value can be given and follow the tracks of message data set 403,405,407 by blender 408.Such as, can be weighted for x and the y acceleration parameter from tracking information 402 with the value of 90%.But, can only be weighted for x and the y acceleration parameter from image dissector 406 with 10%.Acoustic analyser is followed the tracks of information 407 and can be weighted with 0% when relating to acceleration parameter, i.e. this data void value.

Similarly, the Z axis from tracking information 402 is followed the tracks of information parameter and can be weighted with 10%, and image dissector Z axis is followed the tracks of information and can be weighted with 90%.Acoustic analyser is followed the tracks of information 407 and can be weighted with 0% value equally, but the steady track information originating from acoustic analyser 406 can be weighted with 100%, and wherein all the other analyzers tracking information can be weighted with 0%.

After giving suitable distribution of weights, in combinations with that weight, input being controlled data and be averaging, to show that weighted average input controls data set, this data set is analyzed by gesture recognizers 505 subsequently, and associates with the specific action in game environment.The value of association can pre-define by blender 408 or by particular game title.These values can also be the result dynamically adjusted that blender 408 identifies that the particular data quality from each analyzer thus carry out is further discussed below.Adjustment can also be the result building the historical knowledge base when particular data has the characteristic of particular value and/or the given game title of response in specific environment.

Blender 408 can be configured to carry out period dynamic operation in game.Such as, when blender 408 receive various input control data time, it can recognize that certain data all the time outside acceptable scope of data or quality or reflection may indicate that related input device process mistake damage data.

It addition, some condition of real world environments can change.Such as, the natural light in the family game environment of user is likely to forward in the morning and is continuously increased the lower period of the day from 11 a.m. to 1 p.m, thus the problem causing image data capture.Additionally, neighbours or household are likely to become more noisy with the passage of time in a day, thus going wrong when causing audio data capture.Equally, if user has be carried out a few hours game, then their respond becomes less sharp, the problem thus resulting in the explanation of inertial data.

In these cases, or in other situation any that the quality of the input control data of particular form becomes problem, distribution of weights (weight) dynamically can be given the specific collection of the data from specific device by blender 408 again, make to give specific input and control the more or less importance of data, as mentioned above.Similarly, game environment can with particular game need change game process and change so that again assignment or need specific input control data.

Similarly, blender 408 can according to processing mistake or the feedback data that can be generated by gesture recognizers 505 recognize certain data being delivered to gesture recognizers 505 and be processed incorrectly, processes lentamente or entirely without process.Respond this feedback or recognize these difficult treatment (such as, while image analysis data is within tolerance interval, mistake is produced by gesture recognizers 505 when being associated), if which input is blender 408 adjustable seek from which analyzer controls data and the time when having.Before input control data are delivered to blender 408, blender 408 also can need suitable analyzer input controls some analysis of data and processes, it can process data (such as data being averaging) again so that constitutes another layer about efficiently and properly processing the data passing to gesture recognizers 505 and ensures.

In certain embodiments, blender 408 can recognize that certain data damage, invalid or beyond outside particular variables, and the specific input relevant to those data can be needed to control data or variable, make it may replace incorrect data, or suitably analyze relative to necessary variable and calculate certain data.

Can realize according to mode shown in Fig. 6 according to embodiments of the invention, the video game system of the above-mentioned type and method.Video game system 600 can include processor 601 and memorizer 602 (such as RAM, DRAM, ROM etc.).If it addition, to realize parallel processing, then video game system 600 can have multiple processor 601.Memorizer 602 includes data and games code 604, and it can include part configured as described above.Specifically, memorizer 602 can include inertial signal data 606, and these inertial signal data 606 can include storage control routing information as above.Memorizer 602 also can comprise the gesture data 608 stored, for instance represents the data of the one or more postures relevant to games 604.The coded command running on processor 602 can realize multi input blender 605, and it can carry out configuring and working as described above.

System 600 may also include well-known support function 610, for instance input/output (I/O) element 611, power supply (P/S) 612, clock (CLK) 613 and high-speed cache 614.Equipment 600 can include the mass storage device 615 of storage program and/or data alternatively, for instance disc driver, CD-ROM drive, tape drive etc..Controller can also include display unit 616 and user interface section 618 alternatively, in order to mutual between controller 600 and user.Display unit 616 can take the cathode ray tube (CRT) of display text, numeral, graphical symbol or image or the form of flat screens.User interface 618 can include keyboard, mouse, stick, light pen or other device.It addition, user interface 618 can include microphone, video camera or other chromacoder, to provide the direct of signal to be analyzed to catch.The processor 601 of system 600, memorizer 602 and other assembly can via system bus 620 phase interchangeable signal (such as code command and data), as shown in Figure 6.

Microphone array 622 can pass through I/0 function 611 and couple with system 600.Microphone array can include about 2 to about 8 microphones, it is preferable that about 4 microphones, the distance that wherein adjacent microphone is separated less than about 4 centimetres, is preferably between about 1 centimetre and about 2 centimetres.Preferably, the microphone in array 622 is omni-directional microphone.Optional image capture unit 623 (such as video camera) can be passed through I/O function 611 and be coupled with equipment 600.One or more sensing actuators 625 with camera mechanical couplings can exchange signal via I/O function 611 and processor 601.

Term as used herein " I/0 " is generally directed to system 600 and device transmission data or transmission to the periphery from any program of data of system 600 and peripheral unit, operation or device.The transmission of each data is considered as the output from a device and the input to another device.Peripheral unit includes the output device merely entering device, such as printer etc. of such as keyboard and mouse etc. and such as to serve as the device such as writable cd-ROM of input and output device.Term " peripheral unit " including: the external device (ED) of such as mouse, keyboard, printer, monitor, microphone, game console, camera, external Zip drive or scanner etc. and other peripheral hardware of the such as interior arrangement of CD-ROM drive, CD-R drive or internal modems etc. or such as flash memory reader/write device, hard disk drive etc..

In certain embodiments of the present invention, equipment 600 can be video gaming units, and it can include the controller 630 via I/0 function 611 with processor wired (such as USB cable) or wireless coupling.Controller 630 can have analog joystick control 631 and conventional button 633, and they provide control signal conventional during carrying out video-game.This kind of video-game can be embodied as processor readable data and/or the instruction of the program 604 from being storable in memorizer 602 or other processor readable medium of such as associating etc. with mass storage device 615.In certain embodiments, blender 605 can receive the input from analog joystick control 631 and button 633.

Stick control 631 is generally configured to control bar send signalisation along the movement of X-axis so that moving to the left or to the right, will control bar and (upwards) or backward move and then signal the movement along Y-axis (downwards) forward.In configuration in the stick of three-dimensional movement, (clockwise) torsion stick can signal the movement along Z axis (counterclockwise) or to the right to the left.These three axle-X, Y and Z-are generally called rolling, pitching and driftage, especially with respect to aircraft.

Game console 630 can include operable with in processor 602, game console 630 at least one or both carry out the communication interface of digital communication.Communication interface can include universal asynchronous receiver emitter (" UART ").UART can be operable to receive the control signal of the operation for controlling to follow the tracks of device or the signal for communicating with another device from tracking device transmission.Alternatively, communication interface includes USB (universal serial bus) (" USB ") controller.USB controller can be operable to receive the control signal of the operation for controlling to follow the tracks of device or the signal for communicating with another device from tracking device transmission.

It addition, controller 630 can include one or more inertial sensor 632, it can provide position and/or orientation information via inertial signal to processor 601.Orientation information can include angle information, for instance the inclination of controller 630, rolling or driftage.Exemplarily, inertial sensor 632 can include any amount of accelerometer, gyroscope or inclination sensor or their any combination.In a preferred embodiment, inertial sensor 632 includes: inclination sensor, is suitable for sensing game console 630 relative to the orientation tilted with roll axis；First accelerometer, is suitable for sensing the acceleration along yaw axis；And second accelerometer, it is suitable for sensing the angular acceleration relative to yaw axis.Accelerometer can be embodied as such as MEMS device, including the mass installed by one or more springs, wherein has for sensing the mass sensor relative to the displacement in one or more directions.The signal carrying out the displacement depending on mass of sensor may be used to determine the acceleration of game console 630.This kind of technology can by realizing from the instruction of the games 604 being storable in memorizer 602 and being run by processor 601.

Exemplarily, the simple quality that the accelerometer being suitable as inertial sensor 632 can for example be by spring, couple with frame elastic on three or four points is certainly.Pitching and roll axis are arranged in and the plane of the frame intersection being installed to game material controlling device 630.When framework (with game console 630) rotates around pitching and roll axis, quality is certainly by displacement under the influence of gravity, and spring will extend or compresses in the way of depending on the angle of pitching and/or rolling.The displacement of mass can be sensed and convert the signal depending on pitching and/or rolling amount to.Angular acceleration around yaw axis or the linear acceleration along yaw axis also can produce the compression of spring and/or the motion characteristics figure of elongation or mass, and they can be sensed and convert the signal of the amount depending on angle or linear acceleration to.This accelerometer means by the compression of tracking quality movement certainly or spring and expansive force, can be measured around the inclination of yaw axis, roll angle acceleration and the linear acceleration along yaw axis.There is many different modes come the position of tracking quality block and/or be applied to power thereon, including strain ga(u)ge material, photon sensor, Magnetic Sensor, Hall effect device, piezo-electric device, capacitance sensor etc..

It addition, game console 630 can include one or more light source 634, for instance light emitting diode (LED).Light source 634 can be used to distinguish a controller and another controller.Such as, one or more LED or can keep realizing this aspect by making the flicker of LED mode code.Exemplarily, 5 LED can be arranged on game console 630 with linear or two-dimensional model.Although the linear array of LED is preferred, but, LED can alternatively be arranged to rectangular pattern or arch pattern, in order to determine the plane of delineation of LED array when analyzing the image of the obtained LED mode of image capture unit 623.Additionally, LED mode code can be additionally used in the location determining game console 630 during game carries out.Such as, LED can help the inclination of identification controller, driftage and rolling.This detection pattern can help in game, such as better user/sensations of middle offer such as aircraft flying games.Image capture unit 623 can catch the image comprising game console 630 and light source 634.The analysis of this kind of image can determine that position and/or the orientation of game console.This analysis can by store in the memory 602 and the code instructions 604 run by processor 601 realize.For the ease of being caught the image of light source 634 by image capture unit 623, light source 634 may be provided on two or more different sides of game console 630, for instance is arranged on the front and back (as shown in shade).This layout allows image capture unit 623 to keep the mode of game console 630 to obtain the image of light source 634 for the different orientation of game console 630 according to user.

It addition, light source 634 can pass through such as pulse code, amplitude modulation(PAM) or frequency modulation(PFM) form provides telemetered signal to processor 601.This kind of telemetered signal can indicate whether press which stick button and/or press the dynamics of this kind of button.Such as by pulse code, pulsewidth modulation, frequency modulation(PFM) or light intensity (amplitude) modulation, telemetered signal can be encoded into optical signal.Telemetered signal from optical signal can be decoded by processor 601, and responds the telemetered signal of decoding and running game order.From the graphical analysis of image capture unit 623 obtained game console 630, telemetered signal can be decoded.Alternatively, equipment 600 can include being exclusively used in the Individual optical sensor receiving the telemetered signal from light source 634.The U.S. Patent Application No. 11/429414 authorizing RichardL.Marks et al., the title combination for " with the computer picture of the intensity of computer program interface and input equipment and Audio Processing " (attorney docket No.SONYP052) described in such as submitted on May 4th, 2006 determine that the intensive quantity with computer program interface is to use LED, is intactly incorporated herein in it by reference.It addition, the analysis of the image comprising light source 634 can be used for remote measurement and determines position and/or the orientation of game console 630.This kind of technology can realize in the instruction of program 604 by being storable in memorizer 602 and being run by processor 601.

Processor 601 can be combined with the sound source position of the optical signalling of the light source 634 detected from image capture unit 623 and/or the acoustical signal detected from microphone array 622 and characterization information and use the inertial signal from inertial sensor 632, in order to the information of the derive position about controller 630 and/or its user and/or orientation.Such as, " acoustic radar " sound source position and sign can be combined with microphone array 622 and be used for following the tracks of mobile voice, and the motion (by inertial sensor 632 and/or light source 634) of game console simultaneously is individually tracked.In acoustic radar, operationally select precalibrated audit area, and filter the sound that the source outside precalibrated audit area sends.Precalibrated audit area can include audit area corresponding to a large amount of focuses with image capture unit 623 or the visual field.The U.S. Patent Application No. 11/381724 that the title authorizing XiadongMao is " method and apparatus for target sound detection and sign " of submission on May 4th, 2006 is described in detail the example of acoustic radar, by reference it is incorporated herein in.Any amount of various combination providing the different mode of control signal to processor 601 can be combined use with embodiments of the invention.This kind of technology can be realized by the code instructions 604 being storable in memorizer 602 and run by processor 601, and can including one or more instruction alternatively, the one or more processor of these commands direct operationally selects precalibrated audit area and filters the sound sent from the source outside precalibrated audit area.Precalibrated audit area can include audit area corresponding to a large amount of focuses with image capture unit 623 or the visual field.

Program 604 can include one or more instruction alternatively, and the one or more processor of these commands direct is from the microphone M of microphone array 622₀...M_MMiddle generation discrete time-domain input signal x_m(t), it is determined that monitor sector (sector), and use monitoring sector to select finite impulse response filter coefficient in half-blindness source separates, in order to separate from input signal x_mThe different sound sources of (t).Program 604 may also include and is applied to by one or more fractional delays and carrys out self-reference microphone M₀Input signal x₀T selected input signal x that () is different_mThe instruction of (t).Each fractional delay may be selected to the signal to noise ratio optimizing discrete time-domain output signal y (t) from microphone array.Fractional delay is selected such that self-reference microphone M₀Signal be first relative to the signal of other microphone from array in time.Program 604 may also include the instruction of output signal y (t) that mark time delay Δ introduces microphone array so that: y (t+ Δ)=x (t+ Δ) * b₀+x(t-1+Δ)*b₁+x(t-2+Δ)*b₂+...+x(t-N+Δ)*b_N, wherein Δ is between 0 and ± 1.The U.S. Patent Application No. 11/381729 that the title authorizing XiadongMao is " microminiature microphone array " that on May 4th, 2006 submits to is described in detail the example of this kind of technology, is incorporated by reference its complete disclosure.

Program 604 can include one or more instruction, and these instructions operationally make system 600 select the precalibrated monitoring sector comprising sound source.This kind of instruction can make equipment determine whether sound source is arranged in initial sector or is positioned at the specific side of initial sector.If sound source is not in default sector, then instruction can operationally select the different sectors of the specific side of default sector.The feature of this difference sector may be in the decay of the input signal closest to optimum.These instructions operationally can calculate the decay of the input signal from microphone array 622 and the decay to optimum.Instruction can operationally make equipment 600 determine the pad value inputting signal of one or more sector, and selects the sector decayed closest to optimum.The title the authorizing XiadongMao example of this technology described in the U.S. Patent application 11/381725 of " for the method and apparatus that target sound detects " such as submitted on May 4th, 2006, is openly incorporated herein in it by reference.

Signal from inertial sensor 632 can provide part tracking information to input, and the signal generated by following the tracks of one or more light sources 634 from image capture unit 623 can provide another part to follow the tracks of information input.Exemplarily rather than restriction, this kind of " mixed model " signal can be used in the game of rugby type video, and wherein quarter back's head to the left is met and discussed and delivered to the right after feinting deception.Specifically, the game player holding controller 630 labour contractor can turn to the left side, and carry out by controller seem rugby be equally flapped toward right throw action be simultaneously emitted by sound.The microphone array 622 being combined with " acoustic radar " program code can follow the tracks of the voice of user.Image capture unit 623 can be followed the tracks of the motion of user's head or follow the tracks of other order not needing sound or use controller.Sensor 632 can follow the tracks of the motion of game console (expression rugby).Image capture unit 623 also can light source 634 on tracking control unit 630.Can when reaching a certain amount of and/or direction of acceleration of game console 630, or when passing through to press key commands that the button on controller 630 triggers, user can unclamp " ball ".

In certain embodiments of the present invention, for instance the inertial signal from accelerometer or gyroscope may be used to determine the position of controller 630.Specifically, the acceleration signal from accelerometer can relative to time integral once, and to determine the change of speed, and speed can be integrated relative to the time, to determine the change of position.If the value of the initial position of certain time and speed be it is known that, the change of these values and speed and position can be used to determine absolute position.The position of inertial sensor is used to determine ratio use image capture unit 623 and light source 634 faster although can make, but, inertial sensor 632 is likely to a type of mistake through being called " drift ", and it is inconsistent that the mistake wherein accumulated in time may result in from the physical location between position (with shadow representation) and the game console 630 of the computed stick 631 of inertial signal.Embodiments of the invention allow various ways to process this kind of mistake.

Such as, by being reset to by the initial position of controller 630 equal to the current position calculated, drift can manually be offset.User can use the one or more of the button on controller 630 to trigger the order resetting initial position.Alternatively, by being reset to current location according to the determined position as reference of image obtained from image capture unit 623, the drift based on image can be realized.Such as when user triggers the button on game console 630 one or more, can the manually implemented this drift compensation based on image.Alternatively, for instance at regular intervals or response game carry out and be automatically obtained the drift compensation based on image.This kind of technology can be realized by the code instructions 604 being storable in memorizer 602 and run by processor 601.

In some embodiments, it may be desirable to the spurious signals compensated in inertial sensor signal.Such as, for carrying out over sampling from the signal of inertial sensor 632, and sliding average can be calculated from over sampling signal, in order to from inertial sensor signal, remove spurious signals.In some cases, it can be possible to wish signal is carried out over sampling, and from certain subset of data point, get rid of high and/or low value, and calculate sliding average from remainder data point.Additionally, other data sampling and manipulation technology can be used for adjusting the signal from inertial sensor, in order to remove or reduce the importance of spurious signals.The selection of technology can be depending on the character of signal, the calculating that signal is performed, the character carried out of playing or their certain combinations two or more.This kind of technology can realize in the instruction of program 604 by being storable in memorizer 602 and being run by processor 601.

Processor 601 can respond the code instructions of the data 606 and program 604 being stored by memorizer 602 and being retrieved and run by processor module 601 and perform the analysis of inertial signal data 606 as above.The code section of program 604 may conform to any one of multiple different programming language, for instance compilation, C++, JAVA be many other Languages perhaps.Processor module 601 forms general purpose computer, and it becomes special-purpose computer when running the programs such as such as program code 604.Although program code 604 is described herein as being realized by software and running on general purpose computer, but, it will be appreciated by those skilled in the art that the method for task management alternatively can use the hardware of such as special IC (ASIC) or other hardware circuit etc. to realize.It will thus be appreciated that embodiments of the invention can be realized by software, hardware or the combination of both in whole or in part.

In one embodiment, program code 604 wherein may also include processor readable instruction sets, and this instruction set realizes method 520 or their certain methods combining the same feature two or more with method 510 and Fig. 5 C with Fig. 5 B.Program code 604 generally can include one or more instruction, and the one or more processor analysis of these commands direct is from the signal of inertial sensor 632, in order to generates position and/or orientation information, and utilizes this information during carrying out video-game.

Program code 604 can include processor executable alternatively, including one or more instructions, they operationally make image capture unit 623 monitor the visual field before image capture unit 623, what identify in the light source 634 in the visual field is one or more, the change of the light that detection sends from light source 634；And response detects that change triggers the input order to processor 601.The title authorizing RichardL.Marks such as submitted on January 16th, 2004 is combined with image capture device for the U.S. Patent Application No. 10/759782 of " method and apparatus for optical input device " described in and uses LED to the action triggering in game console, it is intactly incorporated herein in by reference.

Program code 604 can include processor executable alternatively, including one or more instructions, they operationally use from the signal of inertial sensor and from image capture unit by following the tracks of the signal that generates of one or more light sources as the input to games system, as mentioned above.Program code 604 can include processor executable alternatively, including one or more instructions of the drift operationally compensated in inertial sensor 632.

It addition, program code 604 can include processor executable alternatively, including the one or more instructions operationally adjusting the controller manipulation linkage to game environment and mapping.This feature allows user to change the manipulation of game console 630 " linkage " to game state.Such as, 45 degree of game console 630 rotate and can rotate linkage with the 45 of game object degree.But, this 1: 1 linkage ratio can be changed into and converts the Y rotation (or tilt or go off course or " manipulation ") of game object to so that the X degree of controller rotates (or tilt or go off course or " manipulation ").Linkage can be 1: 1 ratio, 1: 2 ratio, 1: X ratio or X: Y ratio, the wherein desirable arbitrary value of X and Y.It addition, the mapping that input channel controls to game also can be revised in time or immediately.Amendment can include the threshold value etc. changing posture locus model, location revision, scale, posture.This mapping may be programmed, at random, overlapping, staggered etc., in order to provide the user the manipulation of dynamic range.Map, the amendment of linkage or ratio can be carried out according to game by games 604, game state, by user's modifier button (keypad etc.) of being arranged on game console 630 or widely response input channel adjust.What input channel may include but be not limited to audio user, controller generates audio frequency, controller generated follow the tracks of audio frequency, controller buttons state, video camera output, include accelerometer data, inclination, driftage, rolling, position, the controller telemetry of acceleration and carry out sensor can follow the tracks of user or other data any that the user for object handles.

In certain embodiments, games 604 can change mapping or linkage in time respectively by predetermined time correlation mode from a kind of scheme or ratio to another kind of scheme.Linkage and mapping change can be applied to game environment by various modes.In one example, when personage's health, video game character can control according to the one scheme of linking, and when the health of personage worsens, the whole control order of system adjustable, therefore forces user to aggravate the movement of controller to illustrate order to personage.Adjusting input so that when regaining the control of personage under new mappings when such as may call for user, the video game character becoming disorientation can force the mapping changing input channel.Amendment input channel also can carry out period change in game to the mapping scheme of the conversion of game commands.This conversion can be responded the amendment order sent under one or more elements of game state or response input channel and be undertaken by various modes.Linkage and mapping may be additionally configured to affect configuration and/or the process of one or more elements of input channel.

It addition, such as the acoustic emitter 636 of loudspeaker, buzzer, bugle, bagpipe etc. may be mounted to Joystick controller 630.In certain embodiments, acoustic emitter can be installed to " main body (body) " of Joystick controller 630 in a detachable manner.Positioning and characterize at program code 604 in " acoustic radar " embodiment of the sound adopting microphone array 622 to detect, acoustic emitter 636 can provide and can be detected by microphone array 622 and by program code 604 for following the tracks of the audio signal of the position of game console 630.Acoustic emitter 636 can be additionally used in and from game console 630, additional " input channel " is supplied to processor 601.Can regularly with pulsing from the audio signal of acoustic emitter 636, in order to the beacon making acoustic radar tracing positional is provided.Audio signal (with pulsing or alternate manner) can be audible or hyperacoustic.Acoustic radar can be followed the tracks of the user of game console 630 and handle, and wherein this manipulation tracking can include the information relevant with the position of game console 630 and orientation (such as pitching, rolling or yaw angle).Pulse can trigger with the suitable working cycle, and this is that those skilled in the art can apply.Pulse can be initiated according to from the control signal of system arbitrament.The distribution of the control signal between two or more Joystick controllers 630 that system 600 (by program code 604) tunable couples with processor 601, to guarantee to follow the tracks of multiple controller.

In certain embodiments, blender 605 can be configured to obtain for using the input received from routine controls such as the such as analog joystick control 631 game console 630 and buttons 633 to control the input of the operation of games 604.Specifically, receive blender 605 and can receive the controller input information of self-controller 630.Controller input information can include following at least one: a) identify that the user of game console may move the information of current location controlling bar relative to the resting position controlling bar, or b) identify that whether the switch that comprises in game console is movable information.Blender 605 also can receive the supplementary input information from the environment just using controller 630.Exemplarily rather than restriction, supplement input information can include following one or more: the i) information that the image capture device from environment (such as image capture unit 623) is obtained；And/or ii) from the information with at least one inertial sensor associated (such as inertial sensor 632) of game console or user；And/or iii) the obtained acoustic intelligence (such as from microphone array 622, it is possible to the acoustical signal generated with acoustic transmitter 636 is combined) of sonic transducer from environment.

Controller input information may also include and identifies that whether pressure sensitive buttons is movable information.Inputting information and supplementary input information to produce combination input by processing controller, blender 605 is available for controlling the combination input of the operation of games 604.

Combination input can include each merging input for controlling each function corresponding at the run duration of games 604.Can pass through to merge the controller about specific independent function and input information and the supplementary input information about specific independent function, obtain each at least some merging input.Combine input and can include the merging input for controlling certain function at the run duration of games 604, and can pass through to merge about the controller input information of this function and the supplementary input information about this function, obtain at least some merging input.In such cases, by asking the average of the value of the value representing controller input information and the supplementary input information of expression, merging can be performed.Exemplarily, the value of controller input information and the average of the value of supplementary input information can be asked according to the ratio of to.Alternatively, controller input information and supplementary input information all can be endowed different weights, and can, according to the weighted average of composed weight, the value inputting information and supplementary input information as controller, perform to average.

In certain embodiments, the value of controller input information or first of supplementary input information can be used as the amendment to games and input, for revising the control of at least one the still function of activity activated for input information or supplementary input information according to controller second.Supplement input information and can include being operated by the orientation information of the orientation of the obtained inertial sensor information of inertial sensor 632 and/or expression user's movable objects.Alternatively, supplement input information and include the position of instruction user's movable objects or the information of at least one of orientation." user's movable objects " used herein above can refer to controller 630 or be installed to the product of main body of controller 630, and supplementary input information includes the information of orientation of instruction user's movable objects.Exemplarily, this orientation information can include the information of at least one in instruction pitching, driftage or rolling.

In certain embodiments, can pass through to would indicate that the value of the controller input information of the position controlling bar (such as analog joystick 631 one of them) merges with the value of the supplementary input information of the orientation representing user's movable objects, obtain to combine and input.As mentioned above, user's movable objects can include the object and/or the game console 630 that are installed to game console 630, and when control bar be moved rearwards by, simultaneously pitching just increasing to (high head (nose-up)) value time, combination input can reflect the input of facing upward of enhancing.Similarly, when control bar move forward, simultaneously pitching reduce to negative (undershoot) value time, combination input can reflect that the underriding of enhancing inputs.

The value specifying the controller representing the position the controlling bar input information value as thick control information and the supplementary input information specifying the orientation representing user's movable objects can be passed through as carefully controlling information, obtain combination input.Alternatively, can pass through to specify the switch identifying game console be whether the value of movable controller input information as the value of thick control information and the supplementary input information specifying the orientation representing user's movable objects as thin control information, obtain combination input.It addition, the value that can pass through the supplementary input information of the orientation of appointment expression user's movable objects as thick control information and specifies the controller representing the position controlling bar to input the value of information as carefully controlling information, obtain and combine input.In addition, also by specify the switch identifying game console be whether the value of movable controller input information as the value of thin control information and the supplementary input information specifying the orientation representing user's movable objects as thick control information, obtain combination input.In all these situations or any one situation therein, combination input can represent the thick value controlling information adjusting relatively small number according to thin control information.

In certain embodiments, can by controller being inputted the value represented by information and the value additive combination represented by supplementary input information, make combination input provide the signal with any one the higher or lower value inputting the value that information or supplementary input information individually take than controller to games 604, obtain combination input.Alternatively, combination input can provide the signal with smooth value to games 604, and smooth value signal passes through any one the slower change of the value individually taken than controller input information or supplementary input information in time.Combination input also can provide the high-definition signal of the signal content with increase to games.High-definition signal can change through inputting any one of value that information or supplementary input information individually takes than controller in time more rapidly.

Although describing embodiments of the invention according to the example relevant to the game of PlayStation 3 videogame console/PS3 630, but, the embodiments of the invention including system 600 can handle use on main body, molded object, knob, structure etc. any user, wherein has inertia sensing ability and wireless or alternate manner inertial sensor signal transmission capabilities.

Exemplarily, embodiments of the invention can realize on parallel processing system (PPS).This kind of parallel processing system (PPS) generally includes two or more processor elements, and they are configured to use some parts of independent processor parallel running program.Exemplarily rather than restriction, Fig. 7 illustrates a type of cell processor 700 according to an embodiment of the invention.Cell processor 700 can be used as the processor 601 of Fig. 6 or the processor 502 of Fig. 5 A.In the example depicted in fig. 7, cell processor 700 includes main storage 702, power processor element (PPE) 704 and multiple coprocessor element (SPE) 706.In the example depicted in fig. 7, cell processor 700 includes single PPE704 and eight SPE706.In this configuration, seven in SPE706 can be used for parallel processing, and standby when one can be retained as in other seven is out of order.Alternatively, cell processor can include organizing PPE (PPE group) more and organizing SPE (SPE group) more.In this case, hardware resource can be shared between unit in a group.But, software must be shown as independent component by SPE and PPE.Therefore, embodiments of the invention are not limited to use with the compound and cooperation shown in Fig. 7.

Main storage 702 generally include general and Nonvolatile memory devices and configure for such as system, data transmission synchronizes, memorizer maps specialized hardware depositor or the array of the functions such as I/O and I/O subsystem.In an embodiment of the present invention, video game program 703 can be resided in main storage 702.Memorizer 702 also can comprise signal data 709.Video program 703 can include inertia, image and above in relation to described in Fig. 4, Fig. 5 A, Fig. 5 B or Fig. 5 C or their certain acoustic analyser of configuring of combination and blender.Program 703 can be run on PPE.Program 703 can be divided into the multiple signal processing tasks that can run on SPE and/or PPE.

Exemplarily, PPE704 can be 64 the Power PC Processor unit (PPU) having related L1 and L2 high-speed cache.PPE704 is General Porcess Unit, and it may have access to system administration resources (such as memory protection table).Hardware resource can clearly be mapped to the PPE actual address space seen.Therefore, PPE can pass through to use suitable effective address value directly any one of these resources to be addressed.The major function of PPE704 is the task of the SPE706 in management and distribution cell processor 700.

Although Fig. 7 only illustrates single PPE, but realize at some cell processors, in cell Broadband Engine Architecture (CBEA), cell processor 700 can have to be organized in multiple PPE, PPE groups of PPE group and can there is more than one PPE.These PPE groups can share the access to main storage 702.Additionally, cell processor 700 can include two or more groups SPE.SPE group also can share the access to main storage 702.This kind of configuration is within the scope of the invention.

Each SPE706 includes the local storage LS of coprocessor unit (SPU) and its own.Local storage LS can include one or more independent memory storage area, and each associates with specific SPU.Each SPU can be configured to the instruction (including data to load and data storage operations) only running in the locally stored territory of the association from its own.In this configuration, can pass through to send direct memory access (DMA) (DMA) order from memory stream controller (MFC) to transmit data or the transmission data from locally stored territory to (independent SPE's) locally stored territory, perform the data transmission between other position of local storage LS and system 700.Compared with PPE704, SPU is less complicated computing unit, because they do not perform any system management function.SPU generally has single-instruction multiple-data (SIMD) ability, and generally process Data Concurrent plays any desired data transmission (obeying the PPE access attribute set up), in order to perform its distribution task.The purpose of SPU is to realize needing the application of higher computing unit density, and can be efficiently used the instruction set of offer.A large amount of SPE in the system that PPE704 manages allow the cost-effective process for widespread adoption.

Each SPE706 can include private memory stream controller (MFC), and it includes the associative storage administrative unit that can keep and process memory protection and access grant information.MFC provides the data transmission between main storage means and the local storage of SPE of cell processor, protection and Tong Bu elemental method.MFC command describes pending transmission.The order of transmission data orders (or MFC DMA command) sometimes referred to as MFC direct memory access (DMA) (DMA).

Each MFC can support that multiple DMA transmits simultaneously, and can keep and process multiple MFC command.The request of each MFCDMA data transfer command can comprise locally stored address (LSA) and effective address (EA).It only can be associated the local storage direct addressin of SPE by locally stored address.Effective address can have and is more normally applied, for instance, it can quote main storage means, including all SPE local storages, if they are aliased into actual address space.

In order to help the communication between SPE706 and/or between SPE706 and PPE704, SPE706 and PPE704 can include the signalisation depositor relying on signaling event.PPE704 and SPE706 can be coupled by star topology, and wherein PPE704 serves as the router transmitting message to SPE706.Alternatively, each SPE706 and PPE704 can have the one way signal notice depositor being called mailbox.Mailbox can be used for presiding over operating system (0S) by SPE706 and synchronize.

Cell processor 700 can include input/output (I/O) function 708, cell processor 700 can pass through the peripheral device interface of this function and such as microphone array 712 and optional image capture unit 713 and game console 730 etc..Game console unit can include inertial sensor 732 and light source 734.It addition, Component Interconnect bus 710 can connect above-mentioned various assembly.Each SPE and PPE can access bus 710 by Bus Interface Unit BIU.Cell processor 700 may also include two controllers being typically found in processor: the bus interface controller BIC of the data stream between memory interface controller MIC and control I/O708 and the bus 710 of the data stream between control bus 710 and main storage 702.Although the requirement of MIC, BIC, BIU and bus 710 is likely to greatly change for different realizations, but those skilled in the art can be familiar with its function and for realizing their circuit.

Cell processor 700 may also include internal interrupt controller IIC.IIC assembly management is supplied to the priority of the interruption of PPE.IIC permission processes the interruption of other assembly from cell processor 700, without using main system interrupt control unit.IIC is regarded as second level controller.Main system interrupt control unit can process the interruption originated outside cell processor

In an embodiment of the present invention, some calculating of one or more executed in parallel of PPE704 and/or SPE706 can be used, such as above-mentioned fractional delay.Each fractional delay calculates and can run as one or more independent tasks, and when they become available, difference SPE706 can carry out these tasks.

Although being above the complete description to the preferred embodiments of the present invention, but it is able to use various alternative, amendment and equivalents.Therefore, the scope of the present invention should do not determined with reference to above description, but jointly should determine with reference to claims and complete equivalent scope thereof.It is as herein described regardless of whether preferred any feature all can with as herein described regardless of whether preferred any further feature be combined.In claims below, " one " refers to the one or more quantity after this word, unless otherwise noted.Appended claims is not meant to be interpreted as comprise means-plus-function restriction, unless use in given claim word " be used for ... parts " be expressly recited this restriction.

Claims (27)

1. for controlling a controller for the operation of program, including:

From user can the source of controller input information of steering controller, described controller input information includes for identifying that the user on described controller may move switch or controls the information of current state of bar；

From the source of the supplementary input information of described controller, wherein said supplementary input information includes the information indicating the three-dimensional motion of described controller；And

Wherein, described controller input information becomes with described supplementary input information configuration by process described controller input information and described supplementary input information with obtain combination input and combined, thus obtain for control described program operation described combination input, otherwise wherein by specify described controller input information value as thick control information and specify represent described user's movable objects orientation described supplementary input information value as thin control information or obtain described combination input.

2. controller as claimed in claim 1, wherein, described controller input information becomes with described supplementary input information configuration so that described combination input includes the merging input for controlling certain function at the run duration of described program, and obtains the described at least some merging input by merging about the described controller input information of described function and the described supplementary input information about described function.

3. controller as claimed in claim 2, wherein, by seeking the value and the average of the value representing described supplementary input information representing described controller input information, performs described merging.

4. controller as claimed in claim 3, wherein, asks the value of described controller input information and the average of the value of described supplementary input information according to the ratio of to.

5. controller as claimed in claim 3, wherein, described controller input information and described supplementary input information are all endowed different weights, and according to the weighted average of composed weight, the described value inputting information and supplementary input information as controller, step of averaging described in performing.

6. controller as claimed in claim 1, wherein, the value of described controller input information or first of described supplementary input information is configured for use as the amendment to described program and input, for the amendment control at least one still function of activity activated of input information or described supplementary input information according to described controller second.

7. controller as claimed in claim 1, wherein, the source of described side information includes the LED light source on described controller and diffusing globe, wherein said diffusing globe is configured to make the light generation diffusion from described light source, and wherein supplements input information from described in described LED light source diffusion Image Acquisition in the image obtained from image capture device.

8. controller as claimed in claim 1, wherein, described supplementary input information also includes being operated by the obtained inertial sensor information of inertial sensor or represents at least one of orientation information of orientation of user's movable objects.

9. controller as claimed in claim 8, wherein, described inertial sensor is installed to described controller, and described inertial sensor includes at least one of accelerometer or gyroscope.

10. controller as claimed in claim 1, wherein, described supplementary input information includes the position of instruction user's movable objects or the information of at least one of orientation.

11. controller as claimed in claim 10, wherein, described user's movable objects includes described controller or is installed at least one of product of main body of described controller, and described supplementary input information includes the information that indicates the orientation of described user's movable objects.

12. controller as claimed in claim 10, wherein, described supplementary input information includes the information of at least one of instruction pitching, driftage or rolling.

13. controller as claimed in claim 12, wherein, described supplementary input information includes the information of instruction pitching, driftage or rolling.

14. controller as claimed in claim 10, wherein, by would indicate that the value of controller input information of described switch or the state that controls bar merges with the value of described supplementary input information of the orientation representing described user's movable objects, obtaining described combination and inputting.

15. controller as claimed in claim 14, wherein, when described control bar be moved rearwards by, pitching simultaneously just increasing to (high head) value time, the input of facing upward that described combination input reflection strengthens.

16. controller as claimed in claim 15, wherein, when described control bar move forward, pitching simultaneously be reduced to negative (undershoot) value time, the underriding input that described combination input reflection strengthens.

17. controller as claimed in claim 14, wherein, described combination input represents the value of the described thick control information adjusting relatively small number according to described thin control information.

18. controller as claimed in claim 14, wherein, by specifying the described switch identified on described controller or controlling the value whether bar is movable described controller input information and obtain described combination input as the value of thick control information and the described supplementary input information specifying the orientation representing described user's movable objects as thin control information, wherein said combination input represents the value of the described thick control information adjusting relatively small number according to described thin control information.

19. controller as claimed in claim 14, wherein, described controller input information and described side information are arranged so that wherein said combination input represents the value of the described thick control information adjusting relatively small number according to described thin control information by specifying the described switch identified on described controller or controlling the value whether bar is movable described controller input information and obtain described combination input as the value of thin control information and the described supplementary input information specifying the orientation representing described user's movable objects as thick control information.

20. controller as claimed in claim 1, wherein, described controller input information and described side information be arranged so that by by the value represented by described controller input information with the value additive combination represented by described supplementary input information so that described combination input provides the signal with any one the higher value inputting the value that information or described supplementary input information individually take than described controller to obtain described combination input to described program.

21. controller as claimed in claim 1, wherein, described controller input information and described side information be arranged so that by by the value represented by described controller input information with the value subtractive combination represented by described supplementary input information so that described combination input provides the signal with any one the less value inputting the value that information or described supplementary input information individually take than described controller to obtain described combination input to described program.

22. controller as claimed in claim 1, wherein, described controller inputs information and described side information is arranged so that described combination input provides the signal with smooth value to described program, and described smooth value signal passes through any one the slower change of the value individually taken than described controller input information or described supplementary input information in time.

23. controller as claimed in claim 1, wherein, described controller input information and described side information are arranged so that described combination input provides the high-definition signal of signal content with increase to described program, and described high-definition signal is in time through inputting information than described controller or any one of value that described supplementary input information individually takes changes more rapidly.

24. controller as claimed in claim 1, wherein, described supplementary input information includes responding from the obtained acoustic intelligence of the sound of the source emission described controller sonic transducer from environment.

25. controller as claimed in claim 1, wherein, described controller input information includes identifying that whether pressure sensitive buttons is movable information.

26. controller as claimed in claim 1, wherein, described supplementary input information include following at least one: the i) information that the image capture device from environment is obtained, ii) from the information with at least one inertial sensor associated of described controller or user, or iii) from the information of the sonic transducer in environment.

27. controller as claimed in claim 1, wherein, described supplementary input information include the obtained information of the image capture device from environment, from the information of at least one inertial sensor associated of described controller or user and from the information of the sonic transducer in environment.