JPH05232908A - Instruction input device - Google Patents

Abstract

PURPOSE:To provide an instruction input device capable of inputting information to a computer or the like based upon the appearance of an operator. CONSTITUTION:The instruction input device is constituted of a video camera 3 for picking up the images of an operator 1, image memories 57, 75c for storing the images of the operator inputted from the camera 3, a discriminating means for inputting an image relating to operator's operation executed at optional time which is stored in the memories 57, 75c and an image relating to operator's operation executed at time different from the optional time and discriminating the operator's operation by comparing these images, a judging means for judging operator's gesture from operator's operation discriminated by the discriminating means, an input means for specifying the start of the gesture by an input based upon the prescribed operation of the operator 1, and a processing means for giving the concrete expression of the gesture discriminated by the discriminating means when the start of the gesture is specified by the input means.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an instruction input device for transmitting an operator's instruction or intention to an arbitrary device such as a computer so that the operator can easily use the device.

[0002]

2. Description of the Related Art Conventionally, a keyboard or a mouse is generally used to input an instruction to a computer by using various commands. In order to convey more information or to give an instruction more easily, it has been considered to use the line of sight of the operator, that is, information about where the operator is looking, and gestures and gestures. There is a dedicated device that demands the line of sight, for example, an eye camera, but in the case of this eye camera, the operator must wear the eye camera and make adjustments to achieve alignment beforehand, which is a heavy burden. Will be In addition, with regard to the hand signature, instructions can be input to the computer using a device such as a data glove. In any case, when using such a device, the operator is burdened with the adjustment for mounting and aligning.

Therefore, it has been considered that an image pickup device such as a video camera is used as an input device to recognize an operator's facial expression, line of sight, gesturing and hand gestures from the image, and use it for instructing a computer. For example, head readers by Mase, Watanabe, and Suenaga: Real-time detection of head movements by images (The Institute of Electronics, Information and Communication Engineers, Vol.J74-D-II, No.3, pp.398-406)
There is.

However, since recognition by image processing is not always reliable, even if it is decided that a predetermined instruction is expressed by a certain specific facial expression, the facial expression when the operator does not intend is erroneously recognized as the instruction. Input may occur.
Even if there is no mistake in the recognition of facial expressions themselves, even if the operator suddenly moves the face without being conscious, such as when moving the target on the display screen with the line of sight or the direction of the face, it will be interpreted as an instruction, This may obstruct smooth instructions from the operator.

As described above, the recognition by image processing is not always reliable. Especially, when the background of the operator is complicated or the lighting conditions are not good, the conventional method of inputting a still image and then performing recognition processing. Then, it often fails to cut out the object.

For example, in the case of extracting a face from the background, the method of extracting the part of the image where the brightness is high assuming that the face part is brighter than the background does not work well when there is a bright part or a dark part in the background. Further, if the background brightness changes, the processing often fails.

[0007]

As described above, in the conventional device, it is not possible to reliably use the appearance of the operator for inputting instructions without imposing a burden on the operator.

The present invention has been made in view of the above-mentioned problems of the related art, and the information can be surely input to the computer or the like by the appearance of the operator without the operator wearing a special device. It is an object of the present invention to provide a device that enables the following.

[0009]

In order to achieve the above object, the first invention of the present application is the first invention capable of inputting an operator's intention indication.
Input means, second input means for inputting a predetermined operation of the operator, and operation by the first input means by inputting the predetermined operation of the operator to the second input means. And a processing unit that embodies the intention of the person.

The second aspect of the present invention is an image pickup means for picking up an image of an operator, a storage means for storing an image of the operator input from the image pickup means, and an arbitrary time stored in the storage means. An image relating to the operation of the operator and an image relating to the operation of the operator at a time different from the arbitrary time, and identifying means for identifying the operation of the operator by comparing these images; The determination means for determining the operator's intention display from the operation of the operator identified by the means, the input means for instructing the start of the intention display by the operator's predetermined operation, and the input means for displaying the intention display. The gist of the present invention is to have a processing unit that embodies the intention display judged by the judgment unit when the start is instructed.

The third invention of the present application is defined by claims 1 and 2.
In the described instruction input device, the gist is that the input by the input means is an instruction operation for starting the intention display of the operator identified by the identification means.

Further, a fourth invention of the present application is to provide the first and second inventions.
In the instruction input device described in (3) and (3), the processing means embodies an operator's intention display at the same time as an input time by a predetermined operation of the operator, or at a time point a predetermined time back from the input time. To do.

[0013]

With the above-described structure, the instruction input device according to the first aspect of the invention of the present application is such that when the second input means inputs an operator's predetermined motion, the operation input by the first input means is performed. The person's intention is embodied.

In the instruction input device according to the second aspect of the present invention, the image relating to the operation of the operator at a certain arbitrary time, which is input from the image pickup means and stored in the storage means in the identifying means, is different from the image. The operation of the operator is identified by comparison with the image related to the operation of the operator at the time.
Further, the discriminating means discriminates the intention of the operator from the action of the operator identified by the identifying means. Furthermore, when the input means is instructed to start the intention display,
The intention display determined by the determination means is embodied.

The instruction input device of the third invention of the present application is such that the input by the input means is performed by the operation of the operator identified by the identification means.

The instruction input device according to the fourth aspect of the present invention is configured to embody the operator's intention display at the same time as the input time by a predetermined operation of the operator or at a time point a predetermined time back from the input time. It is a thing.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an instruction input device according to the present invention.

As shown in FIG. 1, the video camera 3 as an image pickup means captures the movement of the operator 1, in this embodiment, the change in the facial expression of the operator 1.

The image processing section 5 comprises a CPU 51, an A / D converter 53, a buffer 55 and an image memory 57. The CPU 51 controls the data flow of the image processing section 5 as a whole, and the A / D converter 53. Is an A / D of an image of the operation of the operator 1 captured by the video camera 3.
The conversion is performed and the input analog signal is output as a digital signal. The digital image signal output from the A / D converter 53 is stored in the image memory 57 via the buffer 55.

The central processing unit 7 comprises a CPU 71, an input side interface (I / F) 73a, an output side interface (I / F) 73b, a ROM 75a, a RAM 75b and an image memory 75c. The input side I / F 73a controls the data flow of the entire instruction input device.
1, keyboard 13, mouse 15, microphone 17
And a foot switch 19 and the like to connect various signals input from them. Also, the output side I /
F73b is the display device 9 and the voice synthesis circuit 21.
Is connected to the speaker 23 via the, and signals are connected to these from the central processing unit 7.

The ROM 75a stores a program relating to the control of the central processing unit 7 and the instruction input device as a whole, and also includes an identification means, a discrimination means and a processing means. The RAM 75b is, for example, an input side I /
Various signals input from the infrared sensor 11, the keyboard 13, the mouse 15 and the like input via the F73a,
Alternatively, a program or the like which is input and set by the keyboard 13 is input, and the image memory 75c as a storage means is an image stored in the image memory 57 of the image processing unit 5 or image-processed by the central processing operation unit 7. The stored images are stored.

The display device 9 displays an image of the operation of the operator 1 captured by the video camera 3, an image processed by the image processing unit 5 or the central processing unit 7, and the like.

The infrared sensor 11 detects that the operator 1 has been positioned in front of the system, and can thereby automatically start the operation of the system.

The keyboard 13 is for inputting various instructions, such as command input or attention calling operation, to the system. The mouse 15 and the microphone 17 are used.
Also, the foot switch 19 can be used to input an alerting operation or the like. In the present embodiment, the keyboard 13 is provided with a dedicated key 13a described later,
It is easy to use.

The voice synthesizing circuit 21 synthesizes a predetermined voice signal based on the signal generated and output by the central processing unit 7 and outputs the voice from the speaker 23.

The spotlight 25 provides contrast between the face of the operator 1 and the background to facilitate image processing, and illuminates the operator 1 from a position slightly oblique to the front of the operator 1. Is arranged as.

The keyboard 13, the dedicated key 13a, the mouse 15, the microphone 17 and the foot switch 19 constitute an input means and a second input means for instructing the start of the intention display by an input by a predetermined operation of the operator. ..

Next, the operation of this embodiment will be described. In addition,
Although various methods are possible for the operator to use what physical expression to give an instruction such as an alerting operation, in the description of this embodiment, as an example, one of the facial features, the mouth The case of using shapes and movements will be described.

First, when the operator 1 sits in front of the display device 9, the infrared sensor 11 is a human being, that is, the operator 1
Is detected, the central processing unit 7 is activated. Then, an image is prepared from the video camera 3 directed to the operator 1 so that the image can be input at a necessary time point in the processing described below. The image is analog-to-digital converted by the A / D converter 53 and temporarily stored in the image memory 57 via the buffer 55.

This image is stored in the ROM 75 of the central processing unit 7.
It is processed by the software stored in a, and face / mouth part detection, mouth part tracking, mouth movement identification, and the like are performed. At this time, various methods can be realized depending on how the three processes of face / mouth part detection, mouth part tracking, and mouth movement identification are performed over time. Hereinafter, each process will be sequentially described.

First, the face / mouth part detection processing will be described with reference to the flowchart of FIG. Step S1
Then, as described above, the imaging is started by the detection of the operator 1 by the infrared sensor 11. The image at the current time t _{n + 1 and} the image captured at a predetermined time t _n before that are captured in step S3, and subsequently, in step S5, the absolute value of the difference is obtained for each pixel of the captured image. Then, after the binarization process is performed in step S7, a pixel having a large value is extracted from the difference result and is set to "1".

Next, in step S9, the projections in the horizontal and vertical directions (in this case, the number of pixels is 1 for each horizontal and vertical line) are obtained. Since the face of the operator 1 slightly moves during the two time points (time t _n , time t _{n + 1} ), the projection portion S as shown in FIG. 3 is extracted by the binarization processing in step S7. The projection S is as shown in the figure showing the horizontal projection and the figure showing the vertical projection.
Therefore, when a large projected values, for example, by taking out the area F, a rectangular region (hereinafter referred to as the face area E _F) surrounding the face out of the way (step S11). As for the lower part of the face, it may be difficult to cut out the face from the projection due to movements of the neck and shoulders. In this case, since the aspect ratio of the face is roughly determined, the width value may be used to determine the lower limit of the face area.

If there is no portion with a large projection value in the above processing, it is determined that the face cannot be detected in step S13, the process returns to step S3, the image at the next time point is input, and the processing is repeated.

Subsequently, the mouth area (area E _M shown in FIG. 4) is detected. In step S15, since the positional relationship between the position of the face and the position of the mouth on the face is determined within a certain range as shown in FIG. 4, the mouth area E _M is estimated based on the predetermined positional relationship. Then, the area around the mouth is taken out (step S17). Furthermore, the position of the mouth is more accurately limited within this mouth region E _M. There are several methods available for this.

For example, the first method is a method of obtaining the projection of the image of the absolute difference value and extracting the portion having a large projection value for the peripheral portion of the mouth similarly to the cutting of the face. Alternatively, using the original image at this point, the area around the mouth is
It is also possible to digitize and take out the mouth part (the part with high density). This is done by labeling connected areas on the binarized result and taking out a large area as a mouth, or obtaining the horizontal and vertical projections of the image on the binarized result as with the difference image. Can be realized by cutting out a region having a large projection value. Alternatively, the horizontal and vertical projections of the original image may be obtained without binarization, and the peripheral area of the mouth may be similarly determined from the projections. Other methods include edge detection and snakes (M.Kass, A.Witkin, D.Terzopou
los: Snakes: Active contour models, Proceedings of 1s
t International Conference on Computer Vision, pp.2
69-276, 1987) may be used to determine the contour of the mouth.

If the area touching the mouth cannot be detected by the above processing, the process returns to step S3 in step S19,
Process the image at the next time point again.

FIG. 5 is a flowchart of the mouth part tracking process. First, capture each image at time t _m prior to the time the time t _{m + 1} of the next point at step S21. Then, with respect to the time t _m at the previous time, it is determined where the mouth has moved in this image (step S23). For this, the tracking method used in another proposal of the inventor (proposition number 13A90X195-1) may be applied to the position around the mouth cut out in the image at the previous time point. Alternatively, using the image around the mouth at the previous time as a template, the sum of absolute differences and the correlation coefficient are obtained for each point around the mouth position at the previous time in the image at this time, and Positions with a high degree of similarity may be determined, or other movements of the mouth may be tracked, for example using the snakes described above. Step S25
If the tracking is successful in, the same process is repeated (step S
25, 21, 23). If the tracking fails, the process normally proceeds to step S27 to return to the face / mouth part detection process.

Such a failure is likely to occur when the operator greatly changes the shape of the mouth. In particular, when a similarity index such as a correlation coefficient is used for tracking or when the shape of the mouth changes, the number of failures increases. The first method and the Snakes method are less susceptible to such changes in mouth shape.

FIG. 6 shows a flowchart of the mouth movement identification processing. In this process, images at two points before and after, for example, the above-mentioned time t _m and time t _{m + 1} are used. The context and the like at the time of image input will be described later. Here, the processing method will be described assuming that the images at the two time points are given. It should be noted that this mouth movement identification processing portion can be similarly realized by a method using images at three time points. This will also be described later.

Since the intention of the operator is finally interpreted by the mouth movement identification process shown in FIG. 6, a simple and reliable process is used here.

First, as the communication with the mouth, usually, the movement by opening and closing the mouth and the movement of the mouth in the horizontal direction and the vertical direction can be used. However, the up, down, left, and right movements of the mouth change the position of the mouth because the mouth does not actually move but the entire face moves.

First, with respect to the area around the mouth obtained by the tracking processing in step S31, the difference between the images at two time points, for example, the time t _m and the time t _{m + 1} is calculated. Here, it is assumed that the bright part in the digital image has a large numerical value, and the dark part has a small numerical value (in reverse, the bright part may have a small numerical value, and the dark part may have a large numerical value. The signs are reversed in the description). Hereinafter, the difference obtained in step S31 will be described as an image obtained by subtracting the image at the previous time point from the image at the time point after the operation.

FIG. 7 shows opening and closing of the mouth in this case.
It is a diagram schematically showing a pattern of difference values during movements of the upper, lower, left and right sides of the mouth. In FIG. 7, the solid line shows the shape before exercise and the broken line shows the shape after exercise. The figure shows the outline of positive and negative difference results. Therefore, it is sufficient to identify these patterns as a process. For this reason, the positive and negative absolute values of the absolute values are equal to or larger than a certain threshold value are extracted by binarization (step S33). Then, for each of them, the projections in the horizontal and vertical directions are obtained as used in the face detection (step S35). The difference result may be divided into positive and negative without binarization, and the projection may be performed as it is or by selecting one having an absolute value of a certain value or more. The projection results are schematically shown in FIGS. 8 to 10 for various movements of the mouth. As can be seen in this figure, the movement of the mouth can be identified by the combination of the presence of mountains in the horizontal and vertical projection results, the width of the mountains, and the positional relationship. Therefore, the presence / width / position of the mountain is checked for each projection result (step S37). Then, the movement of the mouth is identified by the determination table as shown in FIG. 11 (step S39).

Next, a method for implementing the present invention by combining the above processes and adding an operator's attention operation will be described. 12 to 18 show examples of methods of temporally combining processes. In these figures, a vertical line segment with an intermediate length indicates the time when the face / mouth part detection processing is performed. Although there is one line, processing is performed using the images at the time indicated by this line and the next time. A short line segment indicates that the tracking process is being performed. The image of the time indicated by the line segment and the image before the time are used to track the current mouth position. The triangles and squares indicate the input points of the images at the two points used in the mouth movement identification process.

The long line segment indicates the time when the alerting operation occurs. By this operation, the operator indicates that he wants to convey his intention. This is done by pressing a particular key or combination of keys on the keyboard 13. Dedicated key 13 apart from normal key (combination)
a may be provided. Alternatively, the mouse 15 may be pressed by a certain fixed method, or the foot switch 9 may be added and operated. By the operation of alerting, the intention of the instruction is transmitted, the image at that time is input, and the processing between the image at that time and the image at another time realizes processing that is resistant to changes in the surrounding environment such as the background.

The alerting operation is performed by pressing the keyboard 13 or the like as described above, but it is not necessary to immediately release the keyboard 13 after pressing the keyboard 13 or the like, and the input control unit 11 may be kept pressed for a while. The input may be controlled with (normally, the keyboard 13 is input only at the time of pressing). This is because the operator performs a certain intention operation after performing the alerting operation, but the operation of releasing the keyboard 13 at the same time as the operation is
This is because it may be difficult for the operator to use.

FIG. 12 shows a basic implementation method. If the face / mouth detection is successful, the tracking process starts. If tracking fails, face / mouth detection is activated again. Alternatively, even if the detection does not fail, face / mouth detection may be performed from time to time to correct the tracking result. When the attention operation is performed, the images at that time (triangle mark) and the next appropriate time (square mark) are input to perform the mouth movement identification process. In this case, the operator performs a communication operation after performing the alerting operation.

FIG. 13 is the same as FIG. 12, but just before the alerting operation (or before a certain period of time) in the process of identifying the movement.
Two images are used, an image used for tracking at the point of time and an image at the time of the alerting operation. In this case, the operator performs an alerting operation upon completion of the intention indicating operation. It is also possible to perform the alert operation after the operation is completely completed,
It is also possible that the operator intends to make an intention display at the same time that he / she wants to enter the operation of the alerting operation, and when the former actual operation is started, the latter operation is started almost immediately thereafter. Actually, there is a slight time delay from the alerting operation to the image input, and this can be adjusted appropriately to set a timing that is easy to use. This point can be applied not only to this embodiment but also to other embodiments of the present invention. That is, the image input by the alert may be input with an appropriate time delay from the alert operation.

FIG. 14 shows an example in which the tracking part is omitted.
In this case, the image at the time of the alerting operation and the face / mouth portion at a certain time after that (circle in the figure) is obtained, and the image at the certain time after that and the image at any one of the previous time are used. Then, the movement of the mouth is identified.

As a further simplified method, images at the same time may be used for face / mouth detection and mouth movement identification. In this case, both processes are performed using the image at the time of the alerting operation and the image at a certain point after that. This method is sufficient when the intention is to be communicated by the movement of the whole face rather than the movement of the mouth. An example of how to convey your intentions by moving your face will be described later.

FIG. 15 also shows an example in which the tracking part is omitted.
In this example, only the input of images is always performed at appropriate intervals as indicated by the X mark in the figure. Then, an appropriate number of sheets just before are stored in the memory 57.
If there is an attention operation, the face and mouth are detected using the image (circle in the figure) that has been stored for an appropriate time before that and the image at the time of the attention operation, and the The movement of the mouth is identified using the image at the next time point. In this method, the time interval necessary for performing appropriate processing can be freely set as compared with the method of FIG. This method and FIG.
As a combination of the above method, the input time of the square mark is brought before the alerting operation as shown by the arrow in the figure, and
There is also a method of performing the operation as in the case of. Also, as described in the latter half of the description of FIG. 14, there is a method of omitting the face / mouth detection process. In this case, the operation method is as in the case of FIG.

FIG. 16 shows a method in which the tracking is not performed, but the face / mouth detection is performed at appropriate intervals. Compared with the case of FIG. 14, since it is not necessary to perform the face / mouth part detection process after the alerting operation, it is possible to immediately enter the process of identifying the movement of the mouth and provide a quick response.

FIG. 17 is a combination of the cases of FIG. 15 and FIG. 16. Although tracking is not performed, an image is input even during face / mouth detection, and a square mark and a triangle mark image are used. The motion is identified and the operation shown in FIG. 13 is performed.

In FIG. 18, as shown in FIG. 19, the operator touches the mouth (face).
This is a method used when moving the icon on the display 9 by moving up, down, left or right, moving a window of a multi-window, or performing other pointing. First, when a reminder operation occurs, the position of the face / mouth is detected from the images at that time point and the next time point, then the mouth tracking process is executed, and the operation target on the display screen moves according to the movement. When the operator can move to a position where he is satisfied, the operator again performs the alerting operation. The result at the time when this operation occurs is adopted as the result of the final movement or the like. As a reminder operation, the keyboard 13 may be operated every time, but the keyboard is pressed in the first operation and kept pressed, and the second reminder operation is input by releasing it. The side interface 73a may be set in advance. Also,
An embodiment in which the face / mouth detection process is omitted is also possible. In that case, it is not necessary to input the circled image.

The above is the basic embodiment of the present invention.
In the present invention, the method described below may be used as the method of implementing the constituent means. For example, when a large movement of the face is used to convey the operator's intention, the method shown in the flowchart of FIG. 20 may be used. First, in step S41, the two images at the time of the alerting operation and after that (when the method of FIG. 17 is used, the previous time is also possible, in this case, the usage is as in the case of FIG. 15) Take the difference. Hereinafter, the face area is determined in the same manner as steps S3 to S11 shown in FIG. 2 (steps S41 to S4).
7). Then, in step S49, the vertical to horizontal ratio of the face area is obtained, and it is determined that the face is moved horizontally if it is longer in the horizontal direction and vertically if it is longer than the predetermined value. As described above, when it is difficult to find the vertical region of the face, it may be determined to be horizontal if the face moves horizontally, and otherwise to be vertical.

In the method of moving the face, if the face is swayed vertically a few times to nod or it is swayed a few times horizontally to express the negation, some images are used instead of two images. It is also possible to input the image after (or before) the attention operation and add the binarized results obtained by subtracting the images at successive time points or another appropriate combination to determine the face area. By doing so, the determination of the intention when using such an operation becomes more reliable.

Next, another embodiment of the mouth movement identifying process will be described. In this method, a smaller area is provided in the mouth area as shown in FIG. Specifically, a central area e _{C of the} mouth, an upper area e _U , a right area e _R , a left area e _L , and a lower area e _D are provided. In the flowchart shown in FIG. 22, first, for each small area e, the difference between the images at the two points is obtained (step S51). Then, for each small area e, the sum of the difference results for all the pixels in the small area e, or the number of pixels in the positive part and the number of pixels in the negative part whose absolute value is larger than an appropriate threshold value is obtained, and the difference between the two is calculated. Calculate (step S53). By using the determination table of FIG. 23 for the result, the mouth movement is identified (step S5).
5). In FIG. 23, + indicates that the sum of the areas is positive and the absolute value is a certain value or more, or the number of positive pixels is more than the negative pixel number by a certain value or more. -Is the opposite. The shaded lines have smaller absolute values than those with + or −.

In the above embodiments, basically two images were used, but there is also a method of using three images as follows. In this case as well, a method of inputting images at various points in time can be implemented as in the case of two images, but here, a representative example will be used to describe face detection as an example.

Here, as shown in the timing chart of FIG. 24, it is assumed that the image is constantly input (mark X at the time of image input in the figure), and the alerting operation occurs at the time of the triangle. In the face motion detection processing in the present embodiment, an image (circled in the figure) at an appropriate time before the alerting operation,
An image at the time of the alerting operation (triangle mark in the figure) and an image at an appropriate time after that are used (square mark in the figure).

Further, it is assumed that the operator 1 performs an intention indicating operation after the attention calling operation. Further, in this example, the intention display operation is an operation such as moving the face left and right (turning left and right) or moving up and down (looking up, nodding).

In the process, first, a difference image between a triangle mark and a circle mark is obtained (step S61), and from there, step S of FIG.
Similar to steps 43 to S47, the face area is obtained (step S63). Since the face has not moved much since it is before the attention operation, the contour of the whole face is obtained by the difference in this process. Next, the difference between the images at the time of the square mark and the time of the triangle mark is obtained (step S65), and this is also the step S of FIG.
Similar to steps 43 to S47, the face area is obtained (step S67). Then, regarding the size of the face area, it is determined whether the result of step S67 is larger in the vertical direction or the horizontal direction than the result of step S63 (step S69). It is determined that the face has been moved in the direction of increasing the size. This method uses the comparison of the size before the movement and the size after the movement, so that the accuracy is improved.
Further, in this case, since the comparison is made at a close time point, it is possible to judge the movement by the change of the absolute value of the position of the face on the image.
It is determined that the face area has moved in the changed directions of the right edge, the left edge, the upper edge, and the lower edge. The lower end may be difficult to obtain, but in such a case, it can be determined by which direction the upper end has moved.

The present invention can be applied to realize a smoother so-called human interface. Hereinafter, such a device configuration will be described with reference to FIG.

For example, the central processing unit 7 shown in FIG. 26A outputs a predetermined message to the operator 1 on the display device 9 or outputs the voice produced by the voice synthesizer 21 to the speaker 23. When asking a question and requesting a reply from the operator 1 by outputting it via, for example, the central processing unit 7 can be expected to have a reaction from the operator 1 shortly after the reply request. Therefore, the output of the question is regarded as the time of the alerting operation, and subsequently, the state of intention is recognized and the standby state is entered. Therefore, in this case, the operator 1 does not need to particularly perform the alerting operation.

Further, as shown in FIG. 26B, when the central processing unit 7 does not make a particularly clear reply request, the operator 1 is made to perform a caution operation.

In this way, if the operator 1 is asked to answer, and if the operator 1 wants to say something,
That is, when it is desired to make an input, the user's attention is paid and the information is surely transmitted, which improves the operational feeling of the operator 1.

Further, in the above embodiment, the operation by inputting to the keyboard 13 is used as the alerting operation, but the voice of the operator or clapping, stepping, tapping other things with the hand, etc. is used by using the microphone 17. Other sounds may be detected and used as an alerting operation. In this case, the keyboard 13
Since the processing time for voice recognition is longer than that for the input from, pay attention to the timing of the start point of the image input of the intention display.

Although the video camera 3 is used as the sensor for detecting the intention in this embodiment, any other sensor may be used. For example, microphone 1
You may use voice and other sounds using 7. In this case, a sound is input at the time of the alerting operation, and the input voice is recognized. At this time, the input time by voice can shorten the processing time for recognition as compared with the case of inputting by image.
Ordinary voice recognition means may be used for voice recognition, but as in the case of image recognition processing, the sound at the time of the alerting operation and the sound after (or before) are input, and the two points are input. If you find the difference in size or frequency at (or more points) and make it correspond to the intention,
Reliable communication can be realized with simple processing. For example, when a certain key is pressed to perform an alerting operation, when the volume of the sound becomes louder at a later point in time, the object on the screen moves upward.

Further, in each of the above embodiments, all the images are stored in the memory and then processed by the software of the computer. The image at the latest point is not stored in the memory once, but it is hardware-based at the same time as input.
You may make it take the difference with the stored image at the previous time.

In the embodiment, the communication processing part is realized by using the central processing unit 7 used by the operator.
The hardware dedicated to this communication processing may be configured and realized.

As described above in detail, according to the present embodiment, the operator can input an instruction without wearing a special device or instrument on the body and without burden. .. As a result, the frequent occurrence of mistakes in the conventional communication is grasped at the time when the communication is necessary by the alerting operation according to the present embodiment, and after that time (or before).
By using the change in the image at the point of time, it is possible to surely communicate without being affected by the change of the surrounding environment.

In the above embodiment, the case where the movements of the face and mouth are applied to the intention is described, but the present invention is not limited to this. For example, the eyes, ears, nose or hands, fingers. Appropriate parts such as can be applied alone or in an appropriate combination to the intention statement. Also in these cases, the outer shape of the object is obtained by the difference between the images and the shape is obtained by projection or the like, or a small area is set around the object and the brightness change therein is obtained, thereby It can be realized as in the case.

[0072]

As described above, according to the present invention, the operator's predetermined motion is called as a reminder to embody the operator's intention, so that the operator is not burdened and the surroundings are avoided. Instructions can be input without being affected by changes in the environment.

[Brief description of drawings]

FIG. 1 is a block diagram showing a schematic configuration of an embodiment of the present invention together with usage patterns.

FIG. 2 is a flowchart showing a procedure of face / mouth portion detection processing.

FIG. 3 is an explanatory diagram for explaining how to detect a face area from a projection of a face.

FIG. 4 is an explanatory diagram for explaining how to detect a mouth area from a projection of a mouth.

FIG. 5 is a flowchart showing a procedure of a mouth part tracking process.

FIG. 6 is a flowchart showing a procedure of mouth movement identification processing.

FIG. 7 is a diagram illustrating movement of the mouth for each pattern.

FIG. 8 is a diagram showing the shape of a projection associated with the lateral movement of the mouth among the patterns shown in FIG.

9 is a diagram showing the shape of a projection associated with the movement of the mouth in the vertical direction among the patterns shown in FIG.

FIG. 10 is a diagram showing the shape of a projection associated with the opening operation in each of the patterns shown in FIG.

FIG. 11 is a diagram showing a determination table for mouth movement identification in FIGS. 7 and 9 to 10;

FIG. 12 is a timing chart of image input / processing.

FIG. 13 is a timing chart of image input / processing.

FIG. 14 is a timing chart of image input / processing.

FIG. 15 is a timing chart of image input / processing.

FIG. 16 is a timing chart of image input / processing.

FIG. 17 is a timing chart of image input / processing.

FIG. 18 is a timing chart of image input / processing.

FIG. 19 is a diagram illustrating an operation state in the case of the timing chart shown in FIG.

FIG. 20 is a flowchart showing a procedure of a communication process based on a movement of a face.

FIG. 21 is an explanatory diagram of a small area set around the mouth.

FIG. 22 is a flowchart showing a procedure of mouth movement identification processing using a small area.

FIG. 23 is a diagram showing a determination table for mouth movement identification processing using a small area.

FIG. 24 is a timing chart of image input / processing when images at three time points are used.

FIG. 25 is a flowchart showing a procedure of processing when images at three time points are used.

[Fig. 26] Fig. 26 is a diagram illustrating a case where a different attention operation is used.

[Explanation of symbols]

 1 Operator 3 Video Camera 5 Image Processing Section 7 Central Processing Calculation Section 9 Display Device 11 Infrared Sensor 13 Keyboard 15 Mouse 17 Microphone 19 Foot Switch 21 Voice Synthesis Circuit 23 Speaker 25 Spotlight

Claims (4)

[Claims] 1. A first input means capable of inputting an operator's intention indication, a second input means for inputting a predetermined motion of the operator, and the operator to the second input means. And a processing means for embodying the operator's intention display by the first input means by an input relating to the predetermined operation of the instruction input device. 2. An image pickup unit for picking up an image of an operator, a storage unit for storing an image of the operator input from the image pickup unit, and an operation of the operator at an arbitrary time stored in the storage unit. An image and an image relating to the motion of the operator at a time different from the arbitrary time are fetched, and an identification means for identifying the motion of the operator from the comparison of these images, and an operator identified by the identification means The determination means for determining the intention of the operator based on the operation of the operator, the input means for instructing the start of the intention display by the operator's predetermined operation, and the input means for instructing the start of the intention display An instruction input device comprising: a processing unit that embodies the intention display determined by the determination unit. 3. The instruction input device according to claim 1, wherein the input by the input means is an instruction operation for starting the intention display of the operator identified by the identification means. 4. The processing means embodies an operator's intention display at the same time as an input time by a predetermined operation of the operator or at a time point traced back by a predetermined time from the input time. The instruction input device according to items 2 and 3.