JP2002083302A - Information processing device, action recognition processing method, and program storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately recognize an input action by a user and to execute the prescribed process according to the recognized input action. SOLUTION: The image data of a gesture recognition display area 106 in an image obtained by photographing the hand of the user with a CCD(charge coupled device) camera 8 are specified, and the action of the hand of the user is recognized based on the image data of the gesture recognition display area 106. Since the action of the hand of the unser can be recognized by using only the image data of the gesture recognition display area 106 required in recognizing the action of the hand of the user, erroneous recognition is reduced, and recognition precision can be further improved.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus, an operation recognition processing method, and a program storage medium, and is suitably applied to, for example, a notebook personal computer (hereinafter referred to as a notebook personal computer).

[0002]

2. Description of the Related Art Conventionally, a notebook personal computer comprises a main body, display means such as a liquid crystal display, and input means such as a keyboard or mouse for inputting predetermined commands and characters. A predetermined process is executed in response to the command, and the execution result is displayed on a display unit.

In recent notebook personal computers, a so-called jog dial having a predetermined shape is used as input means other than a keyboard and a mouse, for example, so as to slightly protrude from the side of the housing of the notebook personal computer. In response to a rotating operation and a pressing operation on the jog dial, a command such as selection of a menu item and determination of a command is input.

[0004]

In a notebook personal computer having such a configuration, input means such as a keyboard, a mouse, and a jog dial must be directly operated by an operation method for each input means. There is a problem in that it is necessary to memorize an operation method for each input means and a complicated operation is required.

[0005] In addition, if the input operation by the user is incorrect, the notebook personal computer may execute an incorrect process, which is insufficient in usability.

The present invention has been made in view of the above points, and it is an object of the present invention to propose an information processing apparatus, an operation recognition processing method, and a program storage medium which can more accurately recognize an input operation by a user.

[0007]

According to the present invention, in order to solve the above-mentioned problems, image data of a predetermined area in an image obtained by imaging an object to be recognized by an imaging means is specified, and image data of the predetermined area is specified. By recognizing the operation of the recognition target based only on the recognition target operation, the operation of the recognition target can be recognized using only image data of a predetermined area necessary for recognizing the operation of the recognition target. Recognition can be reduced and recognition accuracy can be further improved.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below in detail with reference to the drawings.

(1) First Embodiment (1-1) External Configuration of Notebook Personal Computer In FIG. 1, reference numeral 1 denotes a notebook personal computer (hereinafter, referred to as an information processing apparatus) to which the present invention is applied as a whole. , This is called a notebook computer).
And a display unit 3 attached to the main body 2 so as to be openable and closable.

The main body 2 has a plurality of operation keys 4 for inputting various characters, symbols, numbers, and the like on a top surface thereof, and a stick type pointing device (hereinafter, simply referred to as a stick) 5 used for moving a mouse cursor. Left click buttons 5A and 5B corresponding to a left button and a right button of a normal mouse, a center button 5C for operating a scroll bar without moving the mouse cursor to the scroll buttons, built-in speakers 6A and 6B,
Push-type power switch 7, CCD provided on display unit 3
(Charge Coupled Device) A shutter button 9 for a camera 8, a power lamp PL, a battery lamp BL, and a message lamp M each composed of an LED (Light Emitting Diode).
L and the like are provided.

The display section 3 has a TFT (Thin Fil) corresponding to, for example, an 8.9 type (1024 × 480 pixels) on its front surface.
m Transisitor) Liquid crystal display 1 consisting of color liquid crystal
0 is provided, and an image pickup section 11 provided with a CCD camera 8 as an image pickup means is provided at the upper end of the front center so as to be rotatable with respect to the display section 3.

In the image pickup section 11, the CCD camera 8 can be rotated at an angle of about 180 degrees from the front direction to the rear direction of the display section 3 to position the display section 3 at an arbitrary angle. Focus adjustment when imaging a desired imaging target can be easily performed by rotating the adjustment ring 12 provided at the upper end of the imaging unit 11.

The display unit 3 is provided with microphones 13 on the front side and the back side near the left end of the imaging unit 11, and covers a wide area from the front side to the back side of the display unit 3 via the microphone 13. It is designed to be able to collect sound across.

The display unit 3 further includes a liquid crystal display 10
The pawls 14 and 15 are provided near the left end and the right end, respectively, and holes 16 and 17 are provided at predetermined positions of the main body 2 corresponding to the pawls 14 and 15, respectively. In this state, the claws 14 and 15 are fitted into the corresponding holes 16 and 17, respectively.

On the other hand, when the front side of the display unit 3 closed by the main body 2 is lifted, the fitted state of the holes 16 and 17 and the claws 14 and 15 is released. As a result, the display unit 3 can be developed from the main body 2.

The main body 2 has an IrDA (In)
frared Data Association) compliant infrared port 18,
Headphone terminal 19, microphone input terminal 2
0, USB (Universal Serial Bus) terminal 21, external power supply connector 22, external display output connector 2
3. A jog dial 24 and a modem terminal 25 for a modular jack are provided for inputting a command for executing a predetermined process by a rotation operation and a pressing operation of the rotary operation element.

On the other hand, as shown in FIG. 2, the main body 2 has an exhaust hole 26 on its left side and a PCMCIA (Personal Computer).
Memory Card International Association) standard PC
(Personal Computer) card compatible with PC card slot 27 and 4-pin IEEE (Institute of E)
Electrical and Electronics Engineers) 1394 terminal 28 is provided.

Further, as shown in FIG. 3, the main body 2 is provided with a battery connector 29 on a rear side surface thereof, and a slide type removal lever 31 for removing the battery pack 30 (FIG. 1) on the bottom surface thereof, and the slide type removal lever 31. Lock lever 32 for locking the sliding operation of lever 31
And a reset switch 33 for interrupting the operation of the main body 2 and reconstructing the environment at power-on. The battery pack 30 is detachably connected to the battery connector 29.

(1-2) Circuit Configuration of Notebook Type Personal Computer Next, the circuit configuration of the notebook personal computer 1 will be described in detail with reference to FIG. In the main body 2 of the notebook personal computer 1, a CPU for controlling various functions of the main body 2 in an integrated manner
(Central Processing Unit) 50 is connected to the host bus 52, and the CPU 50 controls the RAM (Rand Rand).
om Access Memory) 53 is executed at a predetermined operating speed based on a system clock provided from the clock generator 60, thereby realizing various functions. I have.

A cache memory 51 is connected to the host bus 52 so as to cache data used by the CPU 50 and realize high-speed access.

The host bus 52 is connected to a PCI (Peripher
al Component Interconnect) bus 55 and host-PC
The PCI bus 55 is connected via a video controller 56 and an IEEE 1349
Interface 57, video capture processing chip 8
3 and the PC card interface 58 are connected.

Here, the host-PCI bridge 54
It controls the exchange of various data between the PU 50 and the video controller 56, the video capture processing chip 83, the IEEE 1349 interface 57, and the PC card interface 58, and controls the memory bus 5
9 for controlling the memory of the RAM 53 connected thereto.

The host-PCI bridge 54 is connected to a video controller 56 and an AGP (Accelerated Graphics Processor).
(ort), so that image data can be transferred between the host-PCI bridge 54 and the video controller 56 at a high speed.

The video capture processing chip 83 includes an I ² C bus 82 (generally a SM (System
Management) bus, and when image data captured by the CCD camera 8 is supplied via the I ² C bus 82, the image data is stored in a built-in frame memory (not shown). And store it in JPEG (Joint
Photographic Experts Group) After generating JPEG image data by performing image compression processing according to the standard,
The JPEG image data is stored in the frame memory again.

Then, the video capture processing chip 83
Transmits the JPEG image data stored in the frame memory to the RAM 53 using the bus master function in response to a request from the CPU 50, and then transfers the JPEG image (still image) data or the Motion JPEG image (moving image) data to the hard disk drive ( HDD) 67.

The video controller 56 outputs image data based on various kinds of application software supplied at appropriate times and image data captured by the CCD camera 8 to the liquid crystal display 10 of the display unit 3,
A plurality of window screens can be displayed.

IEEE 1349 interface 57
Is directly connected to the IEEE 1394 terminal 28, and can be connected to another computer device or an external device such as a digital video camera via the IEEE 1394 terminal 28.

The PC card interface 58 is connected to the PC card slot 2 when adding optional functions.
7 is connected to a PC card (not shown) mounted on the PC 7 and, for example, a CD-ROM (Compact Dimm
sc-Read Only Memory) drive and DVD (Digital Ve
(rsatile Disc) drive so that it can be connected to an external device.

The PCI bus 55 is an ISA (Industrial S)
and a PCI-ISA bridge 66 and a PCI-ISA bridge 66.
The HDD 67 and the USB terminal 21 are connected to the bridge 66.

Here, the PCI-ISA bridge 66
DE (Integrated Drive Electronics) interface, configuration register, RTC (Real-T
imeClock) circuit, a USB interface, and the like, and controls the HDD 67 via the IDE interface based on a system clock supplied from the clock generator 60.

The hard disk of the HDD 67 has Windo
OS (Operating System) such as ws98 (trademark), e-mail program, auto pilot program, jog dial server program, jog dial driver,
Capture software, digital map software, and various other application software are stored, and are transferred to and loaded on the RAM 53 as needed in the course of startup processing.

The PCI-ISA bridge 66 is a U.S.A.
External devices such as a floppy (registered trademark) disk drive, a printer, and a USB mouse (not shown) connected via the B terminal 21 are controlled via a USB interface, and a modem 69 and a sound controller 70 connected to the ISA bus 65. Control.

The modem 69 is connected to an Internet service provider (hereinafter, referred to as a provider) from a modem terminal 25 via a public telephone line (not shown), and is connected to the Internet via a dial-up IP connection via the provider. ing.

The sound controller 70 generates sound data by converting a sound signal collected by the microphone 13 into digital data, outputs the sound data to the CPU 50, and converts the sound data supplied from the CPU 50 into an analog signal. , And outputs it to the outside via the built-in speaker 6.

The ISA bus 65 has an I / O (In / Ou).
t) The controller 73 is connected, receives power from an external power supply from the external power supply connector 22 via the power supply charging control circuit 85, and supplies power to each circuit when the power switch 7 is turned on. Here, the I / O controller 73 operates based on the system clock supplied from the clock generator 60 also.

The power supply / charge control circuit 85 includes an I / O
The controller 73 controls charging of the battery pack 30 connected to the battery connector 29 (FIG. 3).

The I / O controller 73 includes a microcontroller, an I / O interface, a CPU, a ROM,
The flash memory 7 comprises a RAM or the like.
9 (Basic Input / Output Sys.)
tem) to control the input and output of data between the OS and application software and various peripheral devices such as the liquid crystal display 10 and the HDD 67.

The I / O controller 73 is connected to the infrared port 18 and can execute infrared communication with, for example, another computer.

Further, the I / O controller 73 is connected to an inversion switch 77, and when the imaging unit 11 is rotated by 180 degrees toward the back side of the liquid crystal display 10, the inversion switch 77 is turned on. -ISA
The CPU 50 is notified via the bridge 66 and the host-PCI bridge 54.

In addition, the I / O controller 73
It is connected to the full-press / half-press switch 78 and the main body 2
When the shutter button 9 provided on the upper surface of the camera is half-pressed, the full-press / half-press switch 78 is turned on to the half-pressed state, the CPU 50 is notified of this, and the shutter button 9 is fully pressed. The full-press / half-press switch 78 is turned on to the full-press state, and the CPU 5
Notify 0.

That is, the CPU 50 transfers the capture software from the hard disk of the HDD 67 to the RAM 53.
When the shutter button 9 is half-pressed by the user in a state where the camera is raised up, the still image mode is entered and the CC
The D camera 8 is controlled to freeze the still image. When the D camera 8 is fully pressed, the frozen still image data is captured and sent to the video controller 56.

On the other hand, when the shutter button 9 is fully pressed by the user without starting the capture software, the CPU 50 enters the moving image mode, captures a moving image of up to about 60 seconds, and outputs the video. The data is sent to the controller 56.

By the way, the RO of the I / O controller 73
M stores a wakeup program, a key input monitoring program, an LED control program, a jog dial state monitoring program, and various other control programs.

Here, the jog dial state monitoring program is a program used in conjunction with a jog dial server program stored in the hard disk of the HDD 67, and monitors whether the jog dial 24 is rotated or pressed. It is.

The wake-up program is a PCI-I
When the current time supplied from the RTC circuit in the SA bridge 66 coincides with a preset start time, the CPU 50 is a program controlled to execute a predetermined process. Is a program for monitoring inputs from various key switches. The LED control program includes a power lamp PL,
This is a program for controlling lighting of various lamps such as a battery lamp BL and a message lamp ML (FIG. 1).

The RAM of the I / O controller 73 has an I / O register for a jog dial state monitoring program, a set time register for a wake-up program, a key input monitoring register for a key input monitoring program, and a key input monitoring register for an LED control program. An LED control register and other registers for various programs are provided.

The set time register stores the time information of the start time arbitrarily set by the user in advance for use in the wake-up program. Accordingly, the I / O controller 73 determines whether or not the current time supplied from the RTC circuit matches the arbitrarily set start time based on the wake-up program. Notify.

Thus, the CPU 50 starts up predetermined application software set in advance at the start time, and executes predetermined processing according to the application software.

The key input monitoring register stores operation key flags corresponding to input operations of the operation keys 4, the stick 5, the left click button 5A, the right click button 5B, the center button 5C and the like.

Therefore, the I / O controller 73 operates based on the key input monitoring program to determine whether the pointing operation with the stick 5 or the clicking operation of the left click button 5A, right click button 5B and center button 5C has been performed. The determination is made based on the state of the key flag, and when a pointing operation or a click operation is performed, the fact is notified to the CPU 50.

Here, the pointing operation is an operation of moving the mouse cursor to a desired position on the screen by pressing the stick 5 up, down, left and right with a finger, and the clicking operation is the left click button 5A or the right button. This is an operation to quickly press and release the click button 5B with a finger.

As a result, the CPU 50 executes a predetermined process according to the movement of the mouse cursor and the click operation by the pointing operation.

The LED control register includes a power lamp P
A lighting flag indicating the lighting state of various lamps such as L, battery lamp BL, and message lamp ML is stored.

Therefore, the I / O controller 73 causes the CPU 50 to operate the HD
The e-mail program is started from the hard disk at D67, and when an e-mail is received according to the e-mail program, a lighting flag is stored, and the message lamp ML is turned on by controlling the LED 81 based on the lighting flag.

The I / O register for the jog dial state monitoring program stores a rotation operation flag and a pressing operation flag corresponding to a rotation operation and a pressing operation on the jog dial 24.

Therefore, when a user-desired menu item is selected from a plurality of menu items by rotating and pressing the jog dial 24 connected via the rotation detecting unit 88, the I / O controller 73 performs I / O control. The rotation operation flag and the pressing operation flag stored in the O register are set, and the CPU 50 is notified to that effect.

In accordance with the jog dial server program read out from the HDD 67 and launched on the RAM 53, the CPU 50 executes a predetermined process corresponding to the menu item determined by the rotation operation and the press operation of the jog dial 24, or executes the menu item. Start the application software corresponding to.

The I / O controller 73 operates constantly under the control of the power supply / charge control circuit 85 even when the power switch 7 is off and the OS is not running. The user can start application software or a script file desired by the user by pressing the jog dial 24 in the power saving state or when the power is off.

It should be noted the I / O controller 73 is also connected to the I ² C bus 82, supplies the various setting parameters for the CCD camera 8 set by operating keys 4 or the jog dial 24 through the I ² C bus 82 Thus, the brightness and contrast of the CCD camera 8 are adjusted.

(1-3) Gesture Recognition Processing In addition to the above configuration, the notebook computer 1 uses the HDD 67 to execute application software called a cyber gesture program for recognizing a user's hand movement (gesture) captured by the CCD camera 8. It starts up from the hard disk, recognizes the movement of the user's hand captured by the CCD camera 8 based on the cyber gesture program, and executes a predetermined process according to the recognition result on the active window screen based on the application software. It has been done.

That is, the notebook computer 1 starts, for example, an image editing program capable of processing a captured still image, and selects a plurality of images stored in the hard disk of the HDD 67 in order to select a still image to be processed. Are sequentially displayed on the liquid crystal display 10, an image feed operation such as advancing or returning the still images displayed on the liquid crystal display 10 one by one in accordance with the turning operation of the jog dial 24 by the user is usually performed. But
In the present invention, the movement of the user's hand imaged by the CCD camera 8 is represented by C based on a cyber gesture program.
By causing the PU 50 to recognize, the image feed operation described above can be executed without touching the jog dial 24 under the control of the CPU 50.

Incidentally, in the notebook computer 1, the still image displayed on the liquid crystal display 10 when the jog dial 24 is rotated by a predetermined angle or more to the depth side as viewed from the user is displayed.
When the jog dial 24 is rotated by a predetermined angle or more toward the user as viewed from the user, only one still image displayed on the liquid crystal display 10 is fed back.

In practice, the CPU 50 of the notebook computer 1
The gesture recognition processing procedure is entered from the start step of the routine RT1 in FIG. 5, and the process proceeds to the next step SP1.

In step SP1, the CPU 50 starts a cyber gesture program from the hard disk of the HDD 67 in response to a user operation, generates a gesture recognition screen 100 as shown in FIG. 6 according to the cyber gesture program, and responds to the image editing program. After displaying on the active window screen,
Move to the next step SP2.

Here, as shown in FIG. 7, the gesture recognition screen 100 has a screen size of 164 × 136 pixels (pixels).
The title character portion 101 of "CYBERGESTURE" (trademark of Sony Corporation) indicating that it is a cyber gesture program, an option button 102 for performing detailed setting of functions, a help button 103, and a minimize button 1
04 and a close button 105 are provided. The input image actually input from the CCD camera 8 is displayed with a screen size of 160 × 120 pixels in the gesture recognition screen 100.

The gesture recognition screen 100 is formed with an extremely small screen size as compared with the screen size of the liquid crystal display 10 (1024 × 480 pixels), so that the active state displayed on the background of the gesture recognition screen 100 is reduced. The concealment area of a window screen from a still image is minimized.

The CPU 50 of the notebook personal computer 1 displays a cross-shaped gesture recognition display area 106 substantially at the center of the gesture recognition screen 100 so that the gesture recognition display area 106 is displayed in 256 gray scales. It has been done.

In this case, the gesture recognition display area 10
In FIG. 6, a plurality of substantially square targets 107A to 107I are arranged at equal intervals vertically and horizontally along a cross-shaped display area, and the targets 107A to 107I are in a search state in which no motion is recognized. Target 107A-10
The internal area of 7I is alternately displayed in red along the top, bottom, left and right in order, so that the user can easily recognize that the notebook computer 1 recognizes the movement of the user's hand in the left, right, up and down directions. While you can imagine,
The user can easily recognize that the cyber gesture program is running and is currently in the search state.

The targets 107A to 107I of the gesture recognition display area 106 have a size of 8 pixels × 8 pixels in the vertical and horizontal directions as shown in FIG.
Frame portions 107AF to 107IF each having a width of one pixel
And the frame portions 107AF-1
07IF is displayed in red, so that the target 1
07A to 107I are easily visible.

In the gesture recognition display area 106 of the gesture recognition screen 100, a black line display (not shown) is provided for every two horizontal lines of the scanning lines, thereby providing a screen for displaying a normal image. In contrast, the gesture recognition screen 100 is made to be easily recognized by the user.

Further, the gesture recognition display area 106
In response to "JOG DIAL" on the mode display unit 109, a push operation display unit 108A indicating the direction of movement of the hand movement corresponding to the push operation of the jog dial 24, and a cancel operation for canceling the push operation of the jog dial 24 A cancel operation display unit 108B indicating the corresponding hand movement direction, and a rotation operation display indicating the hand movement direction corresponding to the rotation operation when the jog dial 24 is rotated to the depth side as viewed from the user. A rotation operation display section 108D indicating the direction of movement of the hand corresponding to the rotation operation when the jog dial 24 is rotated to the front side when viewed from the user is displayed, and the hand in the jog dial mode is displayed. Is notified to the user that the moving direction is recognized.

In step SP2, the CPU 50 takes an image of the user present in front of the display unit 3 (FIG. 1) with the CCD camera 8 of the image pickup unit 11, and recognizes the input image obtained as a result of the movement of the user's hand. And proceeds to the next step SP3.

At this time, the CPU 50 captures the input image and displays the input image on the gesture recognition screen 1.
00 is displayed so that the user can recognize that the image is being captured in order to recognize the moving direction of the hand.

At step SP3, when the option button 102 of the gesture recognition screen 100 is pressed, the CPU 50 sets the detailed setting screen 14 as shown in FIG.
0 is superimposed on the active window screen to perform various detailed setting processing and mask selection setting processing according to the user's selection operation.

Actually, on the detailed setting screen 140, the “gesture setting” display section 142 is displayed as the forefront as a default, and the “arrow key” for making the movement of the user's hand correspond to the arrow key of the operation key 4 (Curso
r) "item 143," Jog Dial "item 144 corresponding to jog dial 24, or" Internet (Internet) "item 145 corresponding to page turning operation of a Web page on the Internet," Jog dial "item 144 is included. Clicked,
When the “mask” item 146 of the “mask” item 146 or the “not mask” item 147 is clicked, the CPU 50 sets the user's gesture to the jog dial mode corresponding to the movement of the jog dial 24 and A mask mode for masking an image area other than the gesture recognition display area 106 to acquire only image data in the gesture recognition display area 106, and determining the moving direction of the movement of the user's hand based on the acquired image data. Set to.

Further, the CPU 50 "does not mask".
When the item 147 is clicked, the image area other than the gesture recognition display area 106 is not masked, image data is acquired from all input images, and the movement of the user's hand is determined based on the acquired image data. A no-mask mode for determining the moving direction is set.

By the way, the CPU 50 executes the “arrow key” item 1
When the button 43 is clicked, a gesture mode corresponding to the cursor mode as shown in FIG. 10 is set in place of the gesture recognition screen 100 in the jog dial mode while the gesture mode of the user is set to correspond to the arrow key of the operation key 4. A recognition screen 130 is displayed.

On the gesture recognition screen 130, corresponding to "CURSOR" of the mode display unit 109, a press operation display unit 108A, a cancel operation display unit 108B, and a rotation operation display unit 108C in the jog dial mode.
And direction display units 131A to 131A indicating arrow keys corresponding to left, right, up and down instead of the rotation operation display unit 108D.
By displaying 1D, the user is notified that the movement direction of the user's hand movement is associated with the arrow keys (cursor).

Similarly, the CPU 50 executes “Internet”
When the item 145 is clicked, the gesture of the user is set to the Internet mode corresponding to the “forward” and “return” buttons on the Web page, and the gesture recognition screen 135 corresponding to the Internet mode as shown in FIG. 11 is displayed. I do.

On the gesture recognition screen 135, corresponding to "INTERNET" of the mode display section 109,
The page operation display sections 136A and 136B displayed by "BACK" and "NEXT" are displayed, and the scroll direction display section 13 corresponding to the up and down scroll directions is displayed.
By displaying 7A and 137B, the user is notified that the moving direction of the user's hand is associated with a page turning operation or a scroll operation of a Web page.

When the "effect setting" display section 149 is clicked on the detailed setting screen 140, the CPU 50 displays the "effect setting" display section 149 at the forefront as shown in FIG.

Then, the CPU 50 displays “animation” on the “effect setting” display section 149 for displaying an animation using the targets 107A to 107I of the gesture recognition display area 106 when the user recognizes the gesture. Item 153,
When a check mark is added to the “Enable sound effect” item 154 for generating a sound effect when a gesture is recognized, an animation is displayed when the movement direction of the user's hand movement is recognized, At the same time, it is set so as to generate a predetermined sound effect.

Subsequently, when the "recognition level" display section 150 is clicked by the user, the CPU 50 displays the "recognition level" display section 150 at the forefront as shown in FIG. The CPU 50 uses the adjustment bar 157 in the adjustment frame 156 to set the degree of hand movement in the range from “small” to “large” on the “recognition level” display section 150 by using the adjustment bar 157 in the adjustment frame 156. Set within.

In this case, as the degree of hand movement is set to “small”, the direction of movement can be determined with a small movement of the hand, but the probability of erroneous recognition increases. The larger the "larger" setting, the less it is possible to determine the moving direction unless the hand is moved greatly, but the lower the probability of erroneous recognition. Therefore, the user can set the recognition level according to his / her preference, so that the operability can be improved and the usability can be further improved.

Also, a “window display mode” display unit 1
When the user clicks on the button 48, the CPU 50 causes the “window display mode” display unit 1 as shown in FIG.
48 is displayed first.

When the check mark is added to the item “Always display window first” 159 in the “window display mode” display section 148, the CPU 50 may hide the gesture recognition screen 100 in another window. It is set to always be displayed first so that there is no display.

When the “camera mirror surface setting” display section 151 is clicked by the user, the CPU 50 displays the state shown in FIG.
As shown in the figure, the "camera mirror setting" display section 151 is displayed at the forefront.

Then, the CPU 50 sets “camera mirror surface setting”.
In the display unit 151, the check mark is not added to the “mirror display release” item 161 so that the input image displayed on the gesture recognition screen 100 when the CCD camera 8 captures the image itself is displayed as a mirror display. It is set to be set.

In this way, the user can perform various detailed setting processing and mask selection setting processing according to his / her preference.

In step SP4, the CPU 50 performs a mask process on the input image fetched from the CCD camera 8 in step SP2 using a mask image having a predetermined mask shape corresponding to the jog dial mode, and proceeds to the next subroutine SRT2. .

In this case, the CPU 50 actually corresponds to the gesture recognition display area 106 on the gesture recognition screen 100 (FIG. 7) as shown in FIG. 16, and the shielding area 164A and the shield area 164A are left so that only the gesture recognition display area 106 is left. Using a mask image 164 having a mask shape in which the non-shielding region 164B is formed, the mask image 16
By superimposing 4 on the input image and synthesizing it, image data relating to the movement of the user's hand can be obtained from only the area corresponding to the gesture recognition display area 106 in the input image.

As shown in FIG. 17, the subroutine SRT
In step SP11 of the second step, the CPU 50 divides the fetched input image into a plurality of macroblocks each consisting of 16 pixels × 16 pixels, and proceeds to the next step SP12.

At step SP12, the CPU 50
A motion vector for each macroblock is calculated based on a change in the luminance level of each macroblock from the previous frame, and the flow advances to next step SP13.

At step SP13, the CPU 50
A motion area is calculated by collecting macroblocks having motion vectors in the same direction, and the routine goes to the next step SP14.

In this case, since the CPU 50 performs the masking process on the input image using the mask image 164, only the macro blocks obtained from only the region corresponding to the gesture recognition display area 106 in the input image are subjected to data processing. In addition, the data processing amount is set to be a necessary minimum.

That is, the CPU 50 performs data processing on all the macroblocks of the input image as in the case of the no-mask mode. However, since the mask processing is actually performed, the image data of the macroblock is obtained only from the area corresponding to the gesture recognition display area 106. As a result, even when the mask image 164 is used, the data processing amount can be reduced by the same data processing method as in the no-mask mode.

At step SP14, the CPU 50
It is determined whether or not the motion area calculated in step SP13 exists in the input image. If a negative result is obtained here, this means that a motion area, which is a set of macroblocks having motion vectors in the same direction, does not exist in the input image.
Moves to step SP17 and repeats the processing from step SP11 onward for the next frame.

On the other hand, if an affirmative result is obtained in step SP14, this means that a motion area which is an aggregate of macroblocks having motion vectors in the same direction exists in the input image. At this time, CPU5
"0" moves to the next step SP15.

At step SP15, the CPU 50
It is determined whether or not the motion area is equal to or larger than a predetermined size.
If a negative result is obtained here, it indicates that the motion area is not equal to or larger than the predetermined size, that is, for example, it cannot be determined to be a palm area. At this time, the CPU 50 proceeds to step SP17 and proceeds to step SP17. SP1
The processing after 1 is repeated.

On the other hand, if a positive result is obtained in step SP15, this indicates that the motion area can be determined to be a predetermined area or more, that is, a palm area. Step SP
Move to 16.

In step SP16, the CPU 50
After calculating the position of the center of gravity of the moving region based on the coordinate values of the entire moving region, the upper end coordinate position of the moving region corresponding to the vertically upper portion with respect to the position of the center of gravity is detected and corresponds to the fingertip of the hand. By obtaining the position as the upper position of the center of gravity, the obtaining processing procedure regarding the position information of the hand in the subroutine SRT2 is ended, and the routine proceeds to the subroutine SRT3 following the routine RT1.

For example, the CPU 50 displays a pointer 110 having a predetermined shape at the position above the center of gravity acquired as shown in FIG. 18, and gesture-recognizes a recognition frame 109 including the pointer 110 and surrounding the entire palm area of the user. The display is overlapped in the display area 106.

In this case, the gesture recognition display area 1
Although the targets 06A to 107I, etc., 06 are not shown in the drawing, they are not shown for convenience of explanation, but are all actually displayed.

Here, the CPU 50 displays the recognition frame 109 in white with a width of one pixel, and sets each of the targets 107A to 107A.
The pointer 110 formed in the same shape and size as I is displayed in white in a pointer frame 110F having a width of one pixel, and the inside thereof is displayed in red.

As a result, the CPU 50
Frame portions 107AF to 107I in 07A to 107I
F red display and pointer frame 1 of the pointer 110
Each target 107A-
107I and the pointer 110 are clearly distinguished for the user.

The CPU 50 moves the recognition frame 109 and the pointer 110 according to the movement of the user's hand shown in the direction of arrow D.
Are displayed while being moved upward from the bottom in conjunction with each other, so that the user can visually recognize that the user is following the movement of the hand in the recognition process.

As shown in FIG. 19, the subroutine SRT
In step SP21 of No. 3, the CPU 50
3, the moving distance is calculated based on the difference between the coordinate values of the position above the center of gravity between the current frame stored in the ring buffer shape and the previous frame adjacent to the current frame, and the next step SP22 Move on to

In step SP22, the CPU 50
It is determined whether the moving distance between adjacent frames calculated in step SP21 is equal to or less than a predetermined upper threshold. If a negative result is obtained here, this means that the distance from the position indicating the fingertip of the previous frame to the position indicating the fingertip of the current frame is extremely far away, and as data for recognizing hand movement. This indicates that the moving distance is inappropriate, and the CPU 50 moves to the next step SP23.

At step SP23, the CPU 50
Since it is inappropriate to use the movement distance between adjacent frames as data, the detection of the movement direction performed after step SP24 is stopped, and the process returns to step SP2 of the routine RT1 (FIG. 5) to repeat the above processing.

On the other hand, if a positive result is obtained in step SP22, this means that the distance from the position indicating the fingertip of the previous frame to the position indicating the fingertip of the current frame is not extremely large, and the Indicates that the moving distance is appropriate as data for recognizing
The PU 50 moves to the next step SP24.

At step SP24, the CPU 50
As shown in FIG. 20, the coordinate value of the position of the upper center of gravity indicating the fingertip of the current frame sequentially stored in a ring buffer shape, and the upper part of the center of gravity indicating the fingertip of an arbitrary past frame selected from past several frames in a predetermined range. A maximum difference between the coordinate value of the position and the maximum value is calculated as the maximum movement distance, and it is determined whether the maximum movement distance is greater than a predetermined lower threshold.

Here, if a negative result is obtained, it means that the maximum movement distance based on the state transition of the input image over a plurality of frames is smaller than a predetermined lower threshold, that is, it is not worth recognizing hand movement. In this case, the CPU 50 excludes the maximum moving distance from the recognition processing target, returns to step SP21, and repeats the above processing.

On the other hand, if an affirmative result is obtained in step SP24, this means that the maximum movement distance is larger than the predetermined lower threshold value, and it is recognized that the hand has certainly moved left and right or up and down. At this time, the CPU 50 moves to the next step SP25.

At step SP25, the CPU 50
Movement direction of the hand movement (rightward, leftward) based on the movement vector between the upper center of gravity indicating the fingertip of the current frame and the upper center of gravity indicating the fingertip of the past frame used when calculating the maximum movement distance , Upward or downward), and proceeds to step SP5 of the routine RT1 (FIG. 5).

In step SP5, the CPU 50 detects the maximum moving distance and the moving direction of the maximum moving distance, and thus the unit time in the coordinate values of the pixel data of the current frame and the pixel data of the previous frame in the entire detected palm area. It is determined whether or not the detected moving speed of the entire palm region exceeds a predetermined speed based on the moving amount of the hit.

Here, if a negative result is obtained, this means that the detected movement speed of the entire palm region does not exceed a predetermined speed, that is, since the movement is relatively slow, the actual palm region is not the palm region but the face region. It is determined that there is a possibility, and the process returns to step SP2 again to repeat the above processing.

On the other hand, if an affirmative result is obtained in step SP5, this means that the detected movement speed of the entire palm region exceeds a predetermined speed, that is, the palm region is moving relatively fast, so that the palm region may be a palm region. It is determined that the property is even higher, and the routine goes to the next step SP6.

As described above, when there are two or more candidates considered to be a palm region in the input image of the same frame, the CPU 50 can more accurately determine the palm region and the face region.

In step SP6, the CPU 50 determines the movement direction of the movement of the hand recognized based on the cyber gesture program by using a jog dial API (Applicat).
and a recognition result indicating how the notebook personal computer 1 has recognized the moving direction (gesture) of the user's hand is displayed on the gesture recognition screen 100 by visual feedback. Move to the next step SP7.

For example, as shown in FIG. 21, the CPU 50 sets the targets 107F and 107 in the gesture recognition display area 106 of the gesture recognition screen 100.
G, 107C, 107H and 107I to the target 1
By alternately displaying the target 107F in an upward direction (in the direction of arrow E) in red from the lower side where 07I is located (the direction of arrow E) (the target 107H is currently displayed in red in the figure), the movement of the user's hand is shown. The recognition result for the moving direction is visually recognized by the user.

Here, the API is a program interface open to the application software by the OS, and the application software basically performs all processing via the API. Incidentally, the API of the general OS at present takes the form of a function, and the function of the API is called from application software by designating appropriate arguments (parameters).

Incidentally, the CPU 50 operates the jog dial 24.
And the recognition result of the movement direction of the hand movement based on the cyber gesture program are taken in the same input format, and supplied to the jog dial server program via the dual jog dial API. The processing can be simplified.

At step SP7, after recognizing the moving direction of the movement of the user's hand, the CPU 50 supplies a predetermined command corresponding to the moving direction of the hand movement from the jog dial server program to the application software to perform predetermined processing. Then, the process returns to step SP2 to repeat the above-described processing.

Since the CPU 50 is executing a predetermined process according to the recognition result at this time, the gesture recognition process is not performed on the input image for several frames immediately after the recognition of the hand movement. ing.

Thus, the CPU 50 of the notebook personal computer 1
Performs a masking process on an input image so that the user's hand can be recognized based on only image data obtained from an area corresponding to the gesture recognition display area 106 necessary for recognizing the moving direction of the user's hand movement. Can be accurately recognized, a predetermined command corresponding to the recognition result is supplied to the application software via the jog dial server program, and a predetermined image corresponding to the command is displayed on an active window screen based on the application software. The feeding operation can be performed.

Actually, the CPU 50 of the notebook computer 1
When the gesture is recognized as a hand movement from below to above, the still image of the active window screen displayed on the background of the gesture recognition screen 100 is advanced by one.

As described above, the user operates the jog dial 24.
Without directly operating the CCD camera 8 of the imaging unit 11.
Gesture recognition screen 1 just by holding your hand over
The still image of the active window screen displayed on the background of the image No. 00 can be image-forwarded.

(1-4) Operation and Effect in First Embodiment In the above configuration, the CPU 50 of the notebook computer 1
Indicates the user located in front of the display unit 3
An image is captured by the CD camera 8, the resulting input image is captured, and mask processing is performed using the mask image 164 to leave only the gesture recognition display area 106 necessary for gesture recognition.

This eliminates the need for the CPU 50 to obtain image data of, for example, an elbow portion or other portions other than the image data necessary for recognizing the hand movement from the input image. Can be prevented beforehand.

That is, when the user's hand is moved along the vertical direction of the gesture recognition display area 106 as shown in FIG. 18, the CPU 50 displays image data relating to the movement of the elbow portion in the vertical direction of the gesture recognition display area 106. Is not read from the area, the user's gesture can be more accurately recognized, and the operation desired by the user can be executed accurately and reliably.

When the mask processing is performed in the mask mode, the CPU 50 displays the gesture recognition display area 106.
As a result, image data necessary for recognizing hand movements is obtained only from the area corresponding to the gesture recognition display area 106 by a special method compared to the no-mask mode. Therefore, the data processing amount of the gesture recognition processing can be reduced and the accurate recognition result can be detected in a short time without changing the data processing method in the mask mode and the no-mask mode.

According to the above configuration, the notebook personal computer 1 masks the input image picked up by the CCD camera 8 with the mask image 164 so that only the image data necessary for recognizing the hand movement can be obtained. Get efficiently,
Gestures can be more accurately recognized.

As a result, the CPU 50 recognizes a command corresponding to the movement of the user's hand, and can execute a predetermined page-turning operation corresponding to the command accurately and surely. Predetermined processing corresponding to the gesture operation can be executed reliably only by performing the operation.

(2) Second Embodiment (2-1) Overall Configuration of Network System In FIG. 22, reference numeral 200 denotes a network system to which a mobile phone MS3 to which the present invention is applied is connected as a whole. Base station CS, which is a fixed radio station, in a cell obtained by dividing the service area of
1 to CS4 are provided.

The base stations CS1 to CS4 include mobile information terminals MS1 and MS2, which are mobile radio stations, and digital mobile phones MS3 and MS4 with cameras, for example, WC.
Wireless connection is made by a code division multiple access method called DMA (Wideband-Code Division Multiple Access), and a large capacity at a maximum data transfer rate of 2 [Mbps] using a frequency band of 2 [GHz]. Data is transmitted at high speed.

Thus, the portable information terminals MS1 and MS2
And camera-equipped digital mobile phones MS3 and MS4
The W-CDMA system enables high-speed data communication of large-volume data, so it can perform not only voice calls but also various data communications such as sending and receiving e-mails, browsing simple homepages, and sending and receiving images. It is made to be able to do.

The base stations CS1 to CS4 are connected to a public line network INW via a wired line. The public line network INW includes the Internet ITN, many subscriber wired terminals (not shown), computer networks, and corporate networks. The internal network is connected.

An access server AS of an Internet service provider is also connected to the public network INW, and a content server TS owned by the Internet service provider is connected to the access server AS.

[0139] The content server TS converts content such as a simple homepage into, for example, a compact HTML (Hyper Text Markup Language) in response to a request from a subscriber wired terminal or mobile information terminal MS1, MS2 and a digital mobile phone with camera MS3, MS4. ) Format file.

Incidentally, a number of WWW servers WS1 to WSn are connected to the Internet ITN,
Subscriber wired terminal or portable information terminal M according to P protocol
S1, MS2 and digital mobile phone MS with camera
3. The MS 4 can access the WWW servers WS1 to WSn.

Incidentally, the portable information terminals MS1 and MS2 and the digital cellular phones with cameras MS3 and MS4 communicate with the base stations CS1 to CS4 (not shown) using a simple transport protocol of 2 [Mbps].
WWW server W from S4 via Internet ITN
S1 to WSn are communicated by the TCP / IP protocol.

It is to be noted that the management control unit MCU is provided with the public network I
Subscriber wired terminal or portable information terminal MS1, M via NW
S2 and digital mobile phone with camera MS3, MS4
And the subscriber's wired terminal, the portable information terminals MS1, MS2, and the camera-equipped digital cellular phone MS.
3. Authentication processing, charging processing, and the like for the MS 4 are performed.

(2-2) External Configuration of Digital Mobile Phone with Camera Next, the digital mobile phone with camera M to which the present invention is applied
The external configuration of S3 will be described. As shown in FIG. 23, the digital mobile phone with camera MS3 is divided into a display unit 212 and a main body 213 with a central hinge 211 as a boundary, and is foldable via the hinge 211.

[0144] A transmitting / receiving antenna 214 is attached to the display unit 212 at the upper left portion so as to be able to be pulled out and housed.
It is designed to transmit and receive radio waves between and.

The display section 212 is provided with a camera section 215 which is rotatable in an angle range of about 180 degrees at the center of the upper end, and a CCD camera 216 of the camera section 215 is provided.
Thus, a desired imaging target can be imaged.

Here, when the camera section 215 is rotated by approximately 180 degrees and positioned by the user, the display section 212 is provided with a speaker 217 provided at the center of the rear side of the camera section 215 as shown in FIG. Side, thereby switching to a normal voice call state.

Further, the display unit 212 is provided with a liquid crystal display 218 in front of the display unit 212 so as to receive radio wave reception status, remaining battery power, a destination name and telephone number registered as a telephone directory, a transmission history, and the like. The content of the e-mail, the simple homepage, and the image captured by the CCD camera 216 of the camera unit 215 can be displayed.

On the other hand, the main body 213 has “0” on its surface.
There are provided operation keys 219 such as a number key, a call key, a redial key, an end call and power key, a clear key, an e-mail key, and the like of "9", and various instructions are inputted using the operation keys 219. Have been made to gain.

The main body 213 is provided with a memo button 220 and a microphone 221 below the operation keys 219, so that the memo button 220 can record the voice of the other party during a call, and the microphone 221 can be used to record a call. The sound of the user is collected.

Further, the main body 213 is provided with a rotatable jog dial 222 above the operation key 219.
3 is provided so as to protrude slightly from the surface,
Various operations such as a scrolling operation of a telephone directory list or an e-mail displayed on the liquid crystal display 218, a page turning operation of a simple homepage, an image sending operation, and the like are performed in response to the turning operation on the jog dial 222. ing.

For example, the main body 213 operates in response to a rotation operation of the jog dial 222 by the user.
A desired telephone number is selected from a plurality of telephone numbers in the telephone directory list displayed at 18 and the jog dial 22 is selected.
When the user presses the button 2 toward the inside of the main body 213, the selected telephone number is determined and the calling process is automatically performed on the telephone number.

The main body 213 has a battery pack (not shown) inserted on the back side thereof. When the call is ended and the power key is turned on, power is supplied from the battery pack to each circuit section, and the main body 213 can operate. Start up in a proper state.

The main body 213 has the main body 213.
A memory stick slot 224 for inserting a detachable memory stick (trademark of Sony Corporation) 223 is provided on the upper left side of the device. When the memo button 220 is pressed, a call can be made to the memory stick 223 during a call. Record the voice of the other party, send e-mail,
A simple homepage is provided so that images captured by the CCD camera 216 can be recorded.

Here, the memory stick 223 is a kind of flash memory card developed by Sony Corporation, the present applicant. This memory stick 2
Reference numeral 23 denotes an EEPROM (Electrically Era) which is a nonvolatile memory which can be electrically rewritten and erased in a small and thin plastic case having a size of 21.5 × 50 × 2.8 [mm].
sable and Programmable Read Only Memory), which stores flash memory elements.
Various data such as images, sounds, and music can be written and read via pin terminals.

The memory stick 223 can be used in response to a change in the specification of the built-in flash memory due to an increase in capacity or the like.
Adopts a unique serial protocol that can ensure compatibility with the device used, with a maximum writing speed of 1.5 [MB / S]
It achieves high-speed performance with a maximum read speed of 2.45 [MB / S] and secures high reliability by providing an erasure prevention switch.

Therefore, the camera-equipped digital mobile phone MS
3 is configured to be able to insert such a memory stick 223, so that data can be shared with another electronic device via the memory stick 223. .

(2-3) Circuit Configuration of Digital Mobile Phone with Camera As shown in FIG.
In step S3, the power supply circuit unit 251, the operation input control unit 252, the image encoder 253, the camera interface unit 254, and the LCD control unit 250 control the main control unit 250 that controls the display unit 212 and the main body 213 in an integrated manner. (Li
quid Crystal Display) control unit 255, image decoder 2
56, a demultiplexing unit 257, a recording / reproducing unit 262, a modulation / demodulation circuit unit 258, and an audio codec 259.
0 and connected to each other via the image encoder 2
53, an image decoder 256, a demultiplexing unit 257, a modulation / demodulation circuit unit 258, and an audio codec 259.
1 are connected to each other.

When the call end and the power key are turned on by the user's operation, the power supply circuit unit 251 supplies power to each unit from the battery pack to enable the digital cellular phone with camera MS3 to operate. To start.

The camera-equipped digital mobile phone MS3 is
Under the control of the main control unit 250 including a CPU, a ROM, a RAM, and the like, the
1 is converted into digital audio data by the audio codec 259, and this is converted into a digital demodulation circuit unit 2.
The spread spectrum processing is performed at 58, and the digital signal is subjected to digital / analog conversion processing and frequency conversion processing at the transmission / reception circuit section 262, and then transmitted via the antenna 214.

A digital mobile phone with camera MS3
After amplifying the received signal received by the antenna 214 in the voice communication mode, performing a frequency conversion process and an analog-to-digital conversion process, performing a spectrum despreading process in the modulation / demodulation circuit unit 258, and converting the received signal into an analog voice signal by the voice codec 259. Is output via the speaker 217.

Further, digital mobile phone MS with camera
3 transmits the text data of the e-mail input by operating the operation keys 219 and the jog dial 222 to the main control unit 250 via the operation input control unit 252 when transmitting the e-mail in the data communication mode.

The main control unit 250 performs spread spectrum processing on the text data in the modulation / demodulation circuit unit 258, and performs digital / analog conversion processing and frequency conversion processing in the transmission / reception circuit unit 262, and then transmits the base station CS3 via the antenna 214.
(FIG. 17).

On the other hand, when receiving an electronic mail in the data communication mode, the camera-equipped digital mobile phone MS3 performs a spectrum despreading process on the received signal received from the base station CS3 via the antenna 214 in the modulation / demodulation circuit unit 258. After the original text data is restored, the data is displayed as an electronic mail on the liquid crystal display 218 via the LCD control unit 255.

Thereafter, the camera-equipped digital mobile phone MS
3 can also record an e-mail received in response to a user operation on the memory stick 223 via the recording / reproducing unit 262.

On the other hand, camera-equipped digital mobile phone MS3
When transmitting image data in the data communication mode,
The image data captured by the CCD camera 216 is transferred to the image encoder 253 via the camera interface unit 254.
To supply.

Incidentally, a digital mobile phone MS with a camera
3 does not transmit image data, it is also possible to directly display image data captured by the CCD camera 216 on the liquid crystal display 218 via the camera interface unit 254 and the LCD control unit 255.

The image encoder 253 is a CCD camera 2
The image data supplied from the MPU 16 is, for example, MPEG (Movi)
g Picture Experts Group) 2, converts the encoded image data into encoded image data by compression encoding according to a predetermined encoding method such as MPEG4, and sends it to the demultiplexing unit 257.

At this time, the camera-equipped digital mobile phone MS3 simultaneously converts the sound collected by the microphone 221 during imaging by the CCD camera 216 as digital sound data via the sound codec 259 into the demultiplexing unit 257.
To send to.

[0169] The demultiplexing unit 257
3 and the coded image data supplied from 3 and the audio codec 2
The multiplexed data is multiplexed with the audio data supplied from 59 in a predetermined manner, and the resulting multiplexed data is
Then, the transmission / reception circuit unit 262 performs a digital-to-analog conversion process and a frequency conversion process, and then transmits the data via the antenna 214.

On the other hand, when receiving digital image data such as a simple homepage in the data communication mode, the camera-equipped digital mobile phone MS3 modulates the received signal received from the base station CS3 via the antenna 214 by the modem circuit 258. The spectrum despreading process is performed, and the resulting multiplexed data is sent to the demultiplexing unit 257.

The demultiplexing unit 257 separates the multiplexed data into coded image data and audio data, supplies the coded image data to the image decoder 256 via the synchronization bus 261, and Is supplied to the audio codec 259.

The image decoder 256 generates reproduced image data by decoding the encoded image data using a decoding method corresponding to a predetermined encoding method such as MPEG2 or MPEG4, and outputs the reproduced image data via the LCD control unit 255. The image is displayed on the liquid crystal display 218 as an image linked to, for example, a simple homepage.

At this time, the audio codec 259 simultaneously
After converting the audio data into an analog audio signal, the audio data is output via the speaker 217 as, for example, audio linked to a simple homepage.

Also in this case, similarly to the case of the electronic mail, the digital mobile phone with camera MS3 can record the received image data of the simple homepage on the memory stick 223 via the recording / reproducing unit 262 by the operation of the user. is there.

In addition to the above configuration, in the digital mobile phone with camera MS3, the same cyber gesture program and jog dial server program as those in the first embodiment are stored in the ROM of the main control unit 250, and the predetermined application software The active window screen based on the gesture recognition screen 100 is displayed on the liquid crystal display 218 based on the cyber gesture program.
(FIG. 6) can be displayed in a superimposed manner, and an image of the user captured by the CCD camera 216 can be displayed on the gesture recognition screen 100.

Next, the digital cellular phone MS3 with a camera
Is similar to the first embodiment shown in FIGS.
Under the control of the main control unit 250, an input image captured by the CCD camera 216 is captured, and mask processing is performed on the input image so that only an area corresponding to the gesture recognition display area 106 necessary for gesture recognition is left.

The main control unit 250 recognizes the movement direction of the user's hand movement based on image data obtained only from the area corresponding to the gesture recognition display area 106 where the movement direction of the user's hand movement should be recognized. Then, a predetermined command corresponding to the recognition result is supplied to the application software via the jog dial server program,
A predetermined image feed operation according to a command can be executed on an active window screen based on the application software.

Thus, the user can move the jog dial 22
The image of the still image of the active window screen displayed on the background of the gesture recognition screen 100 can be sent simply by moving the hand over the CCD camera 216 without directly operating the camera 2.

(2-4) Operation and Effect in Second Embodiment In the above configuration, digital mobile phone MS with camera
3 captures an image of the user with the CCD camera 216, captures the resulting input image, and performs mask processing using the mask image 164 so as to leave only the gesture recognition display area 106 necessary for gesture recognition.

As a result, the main control unit 250 does not acquire unnecessary image data for recognizing the moving direction of the hand movement from the input image. As a result, it is possible to prevent erroneous recognition from occurring. In addition to the above, the user's gesture can be more accurately recognized, and the user's desired operation can be realized in a non-contact manner.

According to the above configuration, the camera-equipped digital cellular phone MS3 performs mask processing on the input image picked up by the CCD camera 216 so that only the image data necessary for recognizing the movement of the user's hand is obtained. Acquired, the movement of the user's hand can be more accurately recognized based on the image data, and the image feed operation desired by the user can be executed accurately and reliably.

(3) Other Embodiments In the first and second embodiments described above, mask processing is performed using a mask image 164 of a mask shape corresponding to the gesture recognition display area 106 of the gesture recognition screen 100. However, the present invention is not limited to this, and the present invention is not limited to this, and corresponds to the gesture recognition display area having various other shapes, as shown in FIGS. 26A to 26C. A mask image 171, 172 or 173 having a mask shape composed of 172A, 173A and non-shielded areas 171B, 172B, 173B may be used.

In the above-described first and second embodiments, a case has been described in which the cross-shaped mask image 164 is used to recognize the moving direction of the user's hand movement in the horizontal and vertical directions. However, the present invention is not limited to this. In order to recognize the movement of the user's hand in the rotation direction, a mask image 175 including a shielded area 175A and a circular non-shielded area 175B is used as shown in FIG. May be.

In the first and second embodiments described above, the CPU 50 and the main control unit 250 as control means for operating the image feed operation by the jog dials 24 and 222 in accordance with the cyber gesture program.
However, the present invention is not limited to this, and the present invention is not limited to this. The operation of enlarging / reducing the still image displayed on the active window screen, the operation of raising / lowering the volume control, and the operation of moving the channel of the television are performed. Jog dials 24 and 222 and operation keys 4
And 219 may be emulated.

In the first and second embodiments, the targets 107A to 107I and the pointer 1
Although the case where 10 is formed in a square shape has been described, the present invention is not limited to this, and may be formed in various other shapes such as a circle.

Further, in the above-described first and second embodiments, mask processing is performed on the input image using the mask image 164, so that image data necessary only for recognizing the moving direction of the hand movement can be obtained. Although a case has been described in which the input image is obtained from the area corresponding to the gesture recognition display area 106, the present invention is not limited to this. Instead of using the mask image 164, the gesture recognition display area 106 is selected from the input image. By designating and reading only the data of the corresponding area, image data necessary only for recognizing the moving direction of the hand movement may be obtained.

Further, in the first and second embodiments described above, a case has been described where the user's operation is recognized by the CPU 50 and the main control unit 250 as recognition means as recognition targets. However, the present invention is not limited to this. For example, the operation of various other recognition targets such as robots and animals other than the user may be recognized.

In the first and second embodiments described above, the CPU 50 and the main control unit 250
A case has been described in which an input image is masked based on a cyber gesture program stored in advance in a hard disk or ROM to recognize a user's operation. However, the present invention is not limited to this. The above-described operation of the user may be recognized by installing the stored program storage medium in the notebook personal computer 1 and the digital mobile phone with camera MS3.

The state in which the cyber gesture program for executing the above-described series of processing is installed in the notebook personal computer 1 and the digital mobile phone with camera MS3 is executable in the notebook personal computer 1 and the digital mobile phone with camera MS3. For example, a floppy disk, CD-ROM (Compact Disc)
-Read Only Memory), DVD (Digital Versatile Dis
The present invention may be realized not only by package media such as c) but also by a semiconductor memory or a magnetic disk in which a cyber gesture program is temporarily or permanently stored. As a means for storing the cyber gesture program in these program storage media, a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting may be used, and various communication interfaces such as a router and a modem may be used. It may be stored.

Further, in the above-described first and second embodiments, a case has been described in which an input image picked up by the CCD camera 8 in the image pickup unit 11 integrally formed with the display unit 3 is taken as image pickup means. However, the present invention is not limited to this, and an input image captured by various other image capturing means separate from the notebook computer 1 may be captured via the IEEE 1394 terminal 28 or the like.

Further, in the first and second embodiments described above, a case has been described in which the information processing apparatus of the present invention is applied to the notebook personal computer 1 and the digital mobile phone with camera MS3. MS1
And other various information processing devices such as the MS2.

[0192]

As described above, according to the present invention, image data of a predetermined area in an image obtained by imaging an object to be recognized by an imaging means is specified, and recognition is performed based only on the image data of the predetermined area. By recognizing the motion of the target, it is possible to recognize the motion of the recognition target using only image data of a predetermined area necessary for recognizing the motion of the recognition target. The recognition accuracy can be further improved.

[Brief description of the drawings]

FIG. 1 is a schematic perspective view showing an overall configuration of a notebook personal computer according to a first embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating a configuration of a left side surface of a main body.

FIG. 3 is a schematic diagram illustrating a configuration of a rear side surface and a bottom surface of a main body.

FIG. 4 is a block diagram showing a circuit configuration of the notebook personal computer.

FIG. 5 is a flowchart illustrating a gesture recognition processing procedure.

FIG. 6 is a schematic diagram illustrating a gesture recognition screen displayed over the active window screen.

FIG. 7 is a schematic diagram illustrating a configuration of a gesture recognition screen.

FIG. 8 is a schematic diagram illustrating a configuration of a target.

FIG. 9 is a schematic diagram illustrating a detailed setting screen in which a “gesture selection” display unit is displayed at the forefront;

FIG. 10 is a schematic diagram illustrating a gesture recognition screen in a cursor mode.

FIG. 11 is a schematic diagram illustrating a gesture recognition screen in the Internet mode.

FIG. 12 is a schematic diagram illustrating a detailed setting screen in which an “effect setting” display unit is displayed at the forefront;

FIG. 13 is a schematic diagram illustrating a detailed setting screen in which a “recognition level” display unit is displayed at the forefront;

FIG. 14 is a schematic diagram illustrating a detailed setting screen in which a “window display mode” display unit is displayed at the forefront;

FIG. 15 is a schematic diagram illustrating a detailed setting screen in which a “camera mirror surface setting” display unit is displayed at the forefront;

FIG. 16 is a schematic diagram illustrating a mask shape of a mask image corresponding to a jog dial mode.

FIG. 17 is a flowchart illustrating an acquisition processing procedure regarding hand position information.

FIG. 18 is a schematic diagram illustrating a pointer and a recognition frame corresponding to a position above the center of gravity.

FIG. 19 is a flowchart illustrating a determination processing procedure regarding a gesture operation.

FIG. 20 is a schematic diagram used for explaining calculation of a maximum movement distance.

FIG. 21 is a schematic diagram for explaining a visual feedback display.

FIG. 22 is a schematic diagram illustrating an overall configuration of a network system according to a second embodiment.

FIG. 23 is a schematic perspective view showing an external configuration of a digital mobile phone with a camera.

FIG. 24 is a schematic perspective view showing the display unit when the camera unit is rotated.

FIG. 25 is a block diagram showing a circuit configuration of a digital mobile phone with a camera.

FIG. 26 is a schematic diagram illustrating a mask shape (1) of a mask image according to another embodiment.

FIG. 27 is a schematic diagram illustrating a mask shape (2) of a mask image according to another embodiment.

[Explanation of symbols]

1. Notebook type personal computer, 2, 2
13 ... body, 3, 212 ... display unit, 4, 219 ...
Operation keys, 8, 216 ... CCD camera, 10, 218
...... liquid crystal display, 11 ... imaging part, 24, 222
…… Jog dial, 50 …… CPU, 53 …… RA
M, 100: gesture recognition screen, 106: gesture recognition display area, 164, 171, 172, 1
73, 175: mask image, 200: network system, MS1, MS2: portable information terminal, MS3,
MS4 ... Digital mobile phone with camera, CS1-C
S4: Base station, 250: Main control unit.

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Keigo Ihara, Inventor 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Takahiko Sueyoshi 6-35, 7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony F term in reference (reference) 5L096 BA08 CA02 CA24 DA05 EA37 FA14 GA10 HA04

Claims (12)

[Claims] An identification unit configured to identify image data of a predetermined region in an image obtained by imaging the recognition target by an imaging unit; and recognizing an operation of the recognition target based on the image data of the predetermined region. An information processing apparatus comprising: motion recognition means. 2. The information processing apparatus according to claim 1, further comprising control means for executing a predetermined process corresponding to the motion of the recognition target recognized by the motion recognition means. 3. The image processing apparatus according to claim 1, wherein the specifying unit superimposes a mask image including a shielding pattern corresponding to the predetermined area on the image so as to leave only the predetermined area of the image, and synthesizes the image data of the predetermined area. The information processing apparatus according to claim 1, wherein the information is specified. 4. The information processing apparatus according to claim 1, wherein the motion recognition means recognizes the motion of the recognition target by detecting a motion vector based on the image data. 5. A specifying step of specifying image data of a predetermined area in an image obtained by imaging a recognition target by an imaging unit, and recognizing an operation of the recognition target based on the image data of the predetermined area. A motion recognition processing method, comprising a motion recognition step. 6. The motion recognition method according to claim 5, further comprising a control step of executing a predetermined process corresponding to the motion of the recognition target recognized in the motion recognition step. 7. The image data of the predetermined area is synthesized by superimposing a mask image formed of a shielding pattern corresponding to the predetermined area on the image so as to leave only the predetermined area of the image. The method according to claim 5, wherein the operation is identified. 8. The motion recognition processing method according to claim 5, wherein in the motion recognition step, the motion of the recognition target is recognized by detecting a motion vector based on the image data. 9. A specifying step of specifying image data of a predetermined area in an image obtained by capturing an image of a recognition target by an imaging unit; and recognizing an operation of the recognition target based on the image data of the predetermined area. A program storage medium for causing an information processing apparatus to execute a program, comprising a motion recognition step. 10. The program storage medium according to claim 9, further comprising a control step of executing a predetermined process corresponding to the motion of the recognition target recognized in the motion recognition step. 11. The image data of the predetermined area is synthesized by superimposing a mask image formed of a shielding pattern corresponding to the predetermined area on the image so as to leave only the predetermined area of the image. The program storage medium according to claim 9, wherein the program storage medium is specified. 12. The program storage medium according to claim 9, wherein in the motion recognition step, the motion of the recognition target is recognized by detecting a motion vector based on the image data.