JP2010015360A - Control system and control method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform necessary switch operation according to a situation even if a living environment changes. <P>SOLUTION: A switch 17 of an operation target is incorporated with a wireless receiver and a controller, and an identification tag 18 drawn with an identification graphic is disposed near it. The periphery is photographed by an imaging device 15. An information processor 13 analyzes an imaged image, distinguishes whether the identification graphic is present or not, generates an icon corresponding to the identification graphic when detecting the identification graphic, displays it on a display device 12, and sequentially performs emphasis display. The icon to which a user pays attention is distinguished from the bioinformation of brain waves, a view line or the like of a user, and a control signal corresponding to the distinguished icon is transmitted to the wireless receiver of the switch 17 from an wireless communication device 16. In response to the control signal, the controller controls the switch 17. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a control system and a control method for controlling an arbitrary device / device in a living environment.

  As a method for controlling and operating various devices and apparatuses in the living environment, there are a method of directly operating an operation switch and a remote operation using a remote controller or a communication device via a network. A technology is known that enables the switch to be operated by displaying symbols, characters, pictures (icons) on the display panel and selecting the displayed icons when the limbs are inconvenient and cannot be operated. ing.

  For example, Patent Literature 1 discloses a technology that enables a switch to be operated by a user gazing at a displayed icon.

Japanese Patent Application Laid-Open No. 2004-228688 discloses a technique that enables operation of a switch by detecting an electroencephalogram, myoelectricity, electrooculogram, or the like.
Japanese Patent Laid-Open No. 11-73274 JP 10-207592 A

In the conventional switch control device, the operation switch is fixed. For this reason,
As in the case where the user moves, when the use environment (the user's living environment) changes and the switch to be operated changes, this cannot be handled.

  The same problem exists not only when the user is incapable of limbs, but generally when the switch in the living environment is operated using an icon or the like.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique that enables a necessary switch operation depending on the situation even when the living environment changes.

In order to achieve the above object, the control system of the present invention provides:
Detection means for detecting information indicating the presence of the operation target;
In response to detection of information indicating the presence of the control target by the operation means, reading means for reading out information specifying an operation image including an icon from the storage means;
Display control means for generating an operation image including an icon according to the information read by the reading means, and displaying the generated operation image on a display device;
Of the icons included in the operation image displayed on the display device, attention icon determination means for determining the icon that the user is paying attention to,
Control output means for outputting a control signal corresponding to the icon determined by the attention icon determination means;
It is characterized by providing.

  For example, the detecting means is constituted by specific figure detecting means for inputting an image signal and detecting a specific figure in the image represented by the image signal, and stores the specific figure and the operation image specifying information in association with each other. The reading means reads operation image specifying information corresponding to the specific graphic detected by the specific image detecting means, and the display control means is an operation specified by the operation image specifying information read by the reading means. An image is displayed on the display device.

  For example, the detection means includes means for detecting a radio signal indicating the presence of the operation target.

  For example, the detecting means includes means for detecting a non-contact information storage medium arranged in the vicinity of the operation target, and the reading means is an operation stored in the non-contact information storage medium detected by the detection means. Read information identifying the image.

  For example, the display control unit receives an image signal from the imaging device, combines the image defined by the received image signal and the operation image specified by the operation image specifying information, and displays the synthesized image on the display device.

  For example, the display device includes a transmissive display device, and the display control unit displays an operation image on the display device.

  For example, the attention icon discriminating unit acquires the user's biological information, and based on the acquired biological information, pays attention to any icon among the icons included in the operation image displayed on the display unit. Means for determining whether or not.

  For example, the display control unit sequentially highlights icons included in the operation image, the attention icon determination unit acquires a user's biological information, and detects a change in the acquired biological information. Means for discriminating an icon that is in an emphasized state at the time as an attention icon.

  The attention icon determination means acquires, for example, a user's brain wave signal, determines a user's line of sight based on the acquired brain wave signal, and determines the user's line of sight, And at least one of means for discriminating an icon.

  Further, the control method of the present invention detects the presence of an operation target, responds to the detection, reads information specifying an operation image, generates and displays an operation image including an icon according to the read information, and Is identified, and a control signal corresponding to the identified icon is output to the operation target.

  According to the present invention, the presence of an operation target switch or the like is detected, and an operation image including an icon is generated and displayed in order to operate the detected operation target. Therefore, even if the operating object changes due to the environment change, it is possible to operate an arbitrary operating object.

  Hereinafter, a control system according to an embodiment of the present invention will be described with reference to the drawings. This control system is capable of operating all environmental equipment. The operation of environmental equipment broadly means, for example, video / audio equipment, air conditioning equipment, lighting equipment on / off / adjustment, opening / closing of windows and curtains, personal computer on / off / input, etc. present in the living environment .

(First embodiment)
The switch control system 100 according to the present embodiment can operate a device existing in a living environment by using an icon (selection index) displayed on a display panel (monitor) installed at a position close to the user. .

  As shown in FIG. 1, the switch control system 100 includes an electroencephalogram measurement device 11, a display device 12, an information processing device 13, a device control device 14, an imaging device 15, a wireless communication device 16, and a switch 17. , An identification tag 18, a database (DB) server 20, and a network NW.

  As shown in FIG. 2, the electroencephalogram measurement apparatus 11 includes a plurality of probes 111, a signal processing circuit 112, and an output unit 113.

The plurality of probes 111 are mounted on the head of the user 19 and output detected voltage signals.
The signal processing circuit 112 includes an analog signal processing circuit, an A / D converter, a digital signal processing circuit, etc., shapes the voltage signal detected by the probe 111, removes noise, and outputs a detection signal indicating an electroencephalogram. To do.

  The output unit 113 sequentially supplies the detection signal output from the signal processing circuit 112 to the information processing device 13.

  The display device 12 shown in FIG. 1 is arranged in the field of view of the user 19, functions as a user interface for operating the environmental equipment, and has an operation screen 121 as exemplified in FIGS. 3 (a) and 3 (b). Is displayed.

  The operation screen 121 includes a captured image 122 and a steady operation panel image 123.

  The captured image 122 is an image captured by the imaging device 15 and changes with movement of the imaging device 15, change of orientation, change in magnification, and the like. When an image of the identification tag 18 is included in the captured image, an operation image 124 for operating the adjacent switch 17 is displayed superimposed on the identification tag 18. In the example of the figure, two operation images 124 having two sets of icons for turning on and off are superimposed on an identification tag 18 on which a spade is drawn and an identification tag 18 on which a star is drawn. Is displayed.

  The steady operation panel image 123 is a screen for operating the imaging device 15 and other specific devices / devices, and includes a plurality of operation icons.

  FIG. 3A illustrates a steady operation panel image 123 when the icon “imaging device” 126 is selected, and an arrow icon 125 for changing the orientation of the imaging device 15 and an icon for designating another device. 126. FIG. 3B illustrates a steady operation panel image 123 when the icon “air conditioning” 126 is selected, an icon 127 for turning on / off the air conditioner, an icon 128 for raising / lowering the set temperature, and the like. An icon 126 for designating the device is provided.

  A plurality of these icons are highlighted at the same time or individually in a predetermined order (including random). When the user 19 is conscious of selecting the icon at the timing when the desired icon is highlighted, a change in the electroencephalogram is detected by the electroencephalogram measuring apparatus 11 and the highlighted icon is selected and designated. The Details will be described later.

  As illustrated in FIG. 4, the information processing apparatus 13 illustrated in FIG. 1 includes an input unit 131, a tag information storage unit 132, a communication unit 133, a control unit 134, and an electroencephalogram storage unit 135.

  The input unit 131 takes in the detection signal output from the electroencephalogram measurement apparatus 11 and supplies it to the control unit 134.

  The tag information storage unit 132 stores a library of images (materials) constituting the operation image 124 to be displayed on the display device 12. The operation image 124 is, for example, an image illustrated in FIGS. 5A and 5B, and this library includes a frame image illustrated in FIGS. 6A and 6B, and FIG. 7A. The image of the icon (selection element: button) illustrated in (f) is included. Identification information is assigned to each image (material).

  Further, the tag information storage unit 132, as shown in FIG. 8A, is the operation image definition information that defines the identification graphic displayed on the identification tag 18 and the operation image 124 displayed corresponding to the identification graphic. And switch control information for associating information (address) for specifying the switch 17 to be operated.

  As illustrated in FIGS. 8B and 8C, the operation image definition information specifies the identification information of the frame image shown in FIG. 6, the identification information of the icon shown in FIG. 7, and the position in the frame. Information, its function and explanation, text (character string) described in the frame, and information for specifying its position.

  The tag information storage unit 132 further stores a plurality of pre-designed steady operation panel images 123. Each steady operation panel image 123 and an icon 126 for calling it are linked in advance.

  The communication unit 133 illustrated in FIG. 3 receives an image captured by the imaging device 15 via the network NW, and transmits a control signal (command) to the imaging device 15 and the wireless communication device 16.

  The control unit 134 includes a microprocessor, a memory, and the like, and functionally includes a tag detection unit 1341, a discrimination processing unit 1342, and a display output control unit 1343. For example, the control unit 134 receives (1) the captured image of the imaging device 15 by the display output control unit 1343 and displays the captured image on the display device 12. (2) In the captured image of the imaging device 15 by the tag detection unit 1341 Whether or not the identification graphic displayed on the identification tag 18 is included. If it is determined that the identification graphic is included, the display output control unit 1343 controls the operation image based on the switch control information. 124 is generated, and this is superimposed on the captured image of the imaging device 15 to synthesize the captured image 122. Further, the captured image 122 and one of the prepared steady operation panel images 123 are combined to form an image for one screen. (3) The display output control unit 1343 combines the images for one screen. The image of the icon (button) in the image is emphasized periodically or randomly, and (4) is displayed on the display device 12 by the display output control unit 1343. (5) Detection signals (electroencephalogram signals) transmitted from the electroencephalogram measurement apparatus 11 are sequentially stored in the electroencephalogram storage unit 135. (6) The detection signal stored in the electroencephalogram storage unit 135 is analyzed by the discrimination processing unit 1342. Then, it is determined which icon is highlighted when the electroencephalogram signal is changed, and (7) a control signal instructing control corresponding to the determined icon is transmitted via the communication unit 133 by the determination processing unit 1342. To the device control device 14.

  The electroencephalogram storage unit 135 stores the detection signal (electroencephalogram signal) supplied from the electroencephalogram measurement apparatus 11 for several seconds to several tens of seconds.

  As illustrated in FIG. 9, the device control device 14 illustrated in FIG. 1 includes a communication unit 141, an output unit 142, and a control unit 143.

  The communication unit 141 supplies the image from the imaging device 15 to the information processing device 13 via the network NW, receives the control signal from the information processing device 13 via the network NW, and supplies the control signal to the control unit 143. .

  The input / output unit 142 receives video data from the imaging device 15 and supplies the video data to the control unit 143, and supplies control data (direction control signal, zoom control signal, etc.) from the control unit 143 to the imaging device 15. Further, the control data from the control unit 143 is supplied to the wireless communication device 16.

  The control unit 143 includes a processor, captures video data from the imaging device 15 via the input / output unit 142, and supplies the video data to the information processing device 13 via the communication unit 141. Further, the control unit 143 takes in control data from the information processing device 13 via the communication unit 141 and supplies the control data to the imaging device 15 or the wireless communication device 16 via the input / output unit 142.

  An imaging device 15 shown in FIG. 1 includes a CCD (Charge Coupled Device), a CMOS (Complementary MOS) sensor, and the like, captures an image, supplies an image signal to the device control device 14, and receives an image signal from the device control device 14. The direction is controlled according to the control signal, and the aperture and zoom are controlled.

  The wireless communication device 16 includes a wireless communication device and the like, and transmits control data from the device control device 14 to the switch 17 using wireless signals such as radio waves, light, and ultrasonic waves.

  The DB server 20 includes a database (DB) storing graphic information of the identification tag 18, information constituting the operation screen 124, switch control information for controlling an operation target (switch 17), and the like, and a control device. In response to the request from 13, the information is provided.

  The switch 17 can be manually operated and can be remotely controlled by control data from the wireless communication device 16. As shown in FIG. 10A, the switch 17 includes a receiver 171, a controller 172, and an actuator 173.

  The receiver 171 receives control data from the wireless communication device 16 and provides it to the controller 172.

  The controller 172 supplies a drive signal to the actuator 173 according to the operation data.

  The actuator 173 includes a solenoid, a motor, and the like, and controls the movable piece 174 of the switch to turn the switch 17 on and off.

  When adjusting the brightness of the light bulb or the volume of the device, the controller 172 controls the output voltage of the power circuit (or transformer, amplifier) 175 as shown in FIG. In addition, it may be configured to be supplied to the device to be controlled.

  The identification tag 18 shown in FIG. 1 is a card on which a graphic (identification graphic) that can be identified in advance is displayed in a frame, and is arranged adjacent to the switch 17.

(Description of operation)
Next, the operation of the switch control device 100 configured as described above will be described.
First, at a facility construction stage, a switch (including a volume) 17 having a configuration as shown in FIGS. 10A and 10B is installed in accordance with the type of operation target device.

  Subsequently, an identification tag 18 is installed adjacent to the switch 17. At this time, the identification figures displayed on the identification tag 18 are different from each other (identifiable). In the example of FIG. 1, the spade graphic is the identification graphic.

  Next, an operation image necessary for operating the operation target switch (including the volume) 17 is designed as illustrated in FIGS. 5A and 5B, and FIGS. The operation image definition information is created as illustrated in FIG. That is, information for specifying the identification information of the frame constituting the operation image 124, the identification information and position of the icon arranged in the frame, the explanation (help), the text to be displayed and the position thereof are created.

  Further, for each switch 17 to be operated, the identification graphic of the identification tag 18 arranged adjacent thereto, the operation image definition information, and the address of the receiver 171 of the switch 17 are as shown in FIG. And stored in the tag information storage unit 132 of the information processing device 13.

  The probe 111 of the electroencephalogram measurement apparatus 11 is set on the head of the user 19.

  When the switch control system 100 is activated, the signal processing circuit 112 performs signal processing such as waveform shaping and noise removal on the voltage signal detected by each probe 111, and further performs A / D conversion on the signal. The output unit 113 transmits the detection signal output from the signal processing circuit 112 to the signal processing device 13.

  The control unit 134 of the signal processing device 13 takes in the supplied detection signals via the input unit 131 and sequentially stores them in the electroencephalogram storage unit 135 in the FIFO format. As a result, the electroencephalogram storage unit 135 is always in a state where detection signals for the past several seconds are stored.

  In addition, the device control device 14 transmits the captured image of the imaging device 15 to the information processing device 13.

On the other hand, the control unit 134 of the information processing device 13 starts the process illustrated in the flowchart of FIG. 11 and generates an image to be displayed on the display device 12.
First, the control unit 134 of the information processing device 13 captures a captured image supplied via the device control device 14 (step S11).

  Next, the control unit 134 refers to the display control information, and determines whether or not the identification graphic of the identification tag 18 exists in the captured image that has been captured (step S12). That is, it is determined using a pattern matching technique whether or not an identification figure registered in the display control information exists in the captured image. For example, when the display control information shown in FIG. 8A is referred to, the presence / absence of a frame and spade image, a frame and heart image, or the like is determined.

  When the identification graphic is detected, the control unit 134 identifies one of the detected identification graphic (step S13). Next, the control unit 134 synthesizes the operation image 124 corresponding to the identified identification graphic (step S14). That is, a frame graphic is specified in accordance with the operation image definition information corresponding to the specified identification graphic, and each icon and text are arranged at a designated position in the frame to form the operation image 124. Note that information that cannot be stored in the tag information storage unit 132 or related information may be stored in the DB of the DB server 20, read as appropriate, and used to generate the operation image 124.

  Next, it is determined whether or not the operation image 124 has been formed (step S15). That is, it is determined whether or not the corresponding operation image 124 is formed for all the identification figures included in the captured image (step S15). If there is an unprocessed identification figure (step S15; No), the process returns to step S13, the next identification figure is specified, and the same processing is performed.

  If it is determined that the operation image 124 has been formed for all of the identification figures detected in step S12 (step S15; Yes), the operation image 124 formed on the identification figure of the captured image is superimposed and imaging for display is performed. An image 122 is formed (step S16). At this time, it is desirable to make the operation image 124 translucent so that the lower identification figure can be visually recognized.

  Further, the formed captured image 122 and the selected steady operation panel image 123 in the selected state (default in the initial state) are synthesized to synthesize an image for one frame (step S17).

  On the other hand, when it is determined in step S12 that the identification figure does not exist in the captured image (step S12; No), the captured image and the steady operation panel image 123 are combined, and an image for one frame is combined (step S18). ).

  Next, it is determined whether or not there is an icon that is currently highlighted on the image for one frame synthesized in step S17 or S18 (step S19). That is, it is determined whether or not there is an icon that is included in both the current frame image and the previous frame image and is highlighted in the previous frame.

  If there is an icon that is highlighted, the duration for which the icon is highlighted is updated (for example, one frame period is added), and it is further determined whether or not the duration is equal to or longer than a predetermined time. (Step S20). If it is determined that the duration is less than the predetermined time (step S20; No), the highlighted display of the icon is maintained (step S21). For example, make the icon larger than the other icons, change the size over time, make it brighter than other icons, change the brightness over time, and blink (density (gradation) ), And a different color from the other icons, changing the color with the passage of time, moving with the passage of time, and the like.

  Subsequently, the display image for one frame generated in this way is supplied to the display device 12 to be displayed (step S21). Thereafter, the process returns to step S1.

  On the other hand, if it is determined in step S20 that the duration of highlighting is equal to or longer than the predetermined time, another icon is specified and the duration is reset to 0 (step S22). Thereafter, the processing proceeds to step S21. The order of selecting other icons is arbitrary. For example, random (completely random, until all icons are selected, randomly selected from unselected icons, random for each operation image, order within the operation image, etc.), arrangement order (for example, on the screen The order of arrangement from left to right and from top to bottom), combinations thereof, and the like.

  If it is determined in step S19 that there is no emphasized icon (step S19; No), the emphasized icon is specified and the duration is reset to 0 (step S23). Thereafter, the processing proceeds to step S21. The order of selecting the icons to be emphasized is arbitrary.

As a result of performing such processing, taking the display screen shown in FIG. 3A as an example, for example, as shown in FIG. 13A, the state where the icon I11 in the operation image 124a is emphasized for a certain period of time. Then, as shown in FIG. 13B, the state in which the icon I12 is emphasized continues for a certain period of time,..., When the enhancement of the icon in the operation image 124a is completed, FIG. As shown in FIG. 5, the icons I21 to I24 in the operation image 124b are emphasized by a certain time. When the emphasis of the icon in the captured image 122 is completed, as shown in FIG. 13D, the arrow icons I31 to I34 in the steady operation panel image 123 are emphasized for a certain period of time, and then the icons I41 to I43 are sequentially displayed. To be emphasized.
Such a display is repeated.

  On the other hand, the control unit 134 periodically executes the attention icon detection process shown in FIG. When the attention icon detection processing is started, the control unit 134 first analyzes detection signals for the past several seconds (for example, about 5 seconds) stored in the electroencephalogram storage unit 135, and the icon that the user 19 is gazing at. A specific signal that appears when is highlighted is searched (step S31). When the specific signal is detected (step S32; Yes), the control unit 134 specifies the icon that is highlighted at the timing when the specific signal is generated (step S33). When the icon is specified, the process corresponding to the icon is specified with reference to the switch control information (step S34). Subsequently, the control unit 134 executes the specified process (step S35). That is, the control data instructing the set operation is transmitted to the receiver 171 specified by the corresponding address.

  On the other hand, when a specific signal is not detected in the analysis process in step S31 (step S32; No), the control unit 134 ends the current process.

For example, it is assumed that the user 19 wants to turn on the entrance lamp.
In this case, the user 19 thinks about selecting and selecting an icon at the timing when the icon I11 is highlighted as shown in FIG.

  The control unit 134 analyzes the detection signal stored in the electroencephalogram storage unit 135, and determines that a specific signal generated when the icon is watched is output at the timing when the icon I11 is highlighted. (Steps S31 to S33). Subsequently, the control unit 134 refers to the switch control information, determines that the corresponding process is to turn on the switch 1 (step S34), and transmits the corresponding control data to the wireless communication device via the device control device 14. 16 is instructed (step S35). The wireless communication device 16 wirelessly transmits control data to a corresponding address. The receiving unit 171 of the corresponding switch 17 (the switch 17 adjacent to the identification tag 18 on which the spade identification figure is drawn) receives this control data, and the controller 172 drives the actuator 173 according to the received control data to switch Turn on. As a result, the entrance lighting is turned on.

  For example, when it is desired to turn the imaging device 15 to the left, the user 19 thinks about selecting and selecting an icon at the timing when the icon I31 is highlighted as shown in FIG.

  The control unit 134 analyzes the detection signal and determines that the specific signal is output while the icon I31 is highlighted (steps S31 to S33). Subsequently, referring to the switch control information, it is determined that the corresponding process is the leftward rotation control of the imaging device 15 (step S34), and the corresponding control data is obtained via the device control device 14. (Step S35). The imaging device 15 rotates to the left according to the control data.

Further, for example, it is assumed that the user 19 wants to control the air conditioner.
In this case, the user 19 thinks of selecting and selecting an icon at the timing when the icon I41 is highlighted as shown in 13 (e).

  The control unit 134 analyzes the detection signal, determines that the specific signal is output while the icon I41 is highlighted (step S31), and then refers to the switch control information to link the corresponding process. It is determined that the displayed steady operation image is displayed (step S32), and the operation image to be synthesized in step S22 is switched to one corresponding to the instructed icon (step S33).

  As described above, according to the environmental operation system 100 according to this embodiment, any switch can be operated without directly operating it.

  In addition, if switch control information for controlling the switch 17 in the living environment of the user 19 is stored in the tag information storage unit 132 in advance and the identification tag 18 is arranged adjacent to each switch 17, As long as the user 19 moves with the switch control device 100, the arbitrary switch 17 can be controlled.

(Modification)
In the above embodiment, the operation image 124 is displayed on the corresponding identification tag 18 so as to overlap (semi-transparent state), but the display position of the operation image 124 is arbitrary. For example, as shown in FIG. 14, you may arrange | position at the edge part of a screen.

  In the above embodiment, as shown in FIGS. 3A and 3B, the operation image 124 is displayed facing the front. However, the display mode is arbitrary. For example, the inclination and / or distance of the identification tag 18 with respect to the imaging device 15 is obtained from the inclination of the image of the frame displayed on the identification tag 18, and the operation image 124 is set to have the same inclination as this inclination. , It may be processed into a size corresponding to the distance and displayed.

  For example, when the frame (originally rectangular) of the identification figure in the captured image is as shown in FIG. 15A, the coordinates (x1, y1) to (x4, y4) of the vertices P1 to p4 of the frame are obtained. 15B, based on the coordinates of the four vertices P1 to P4 and the distance L of one known side of the frame, the tag inclination angle θ1 (on the plane perpendicular to the optical axis of the imaging device 15). Rotation angle), θ2 (angle formed with the optical axis of the imaging device 15), and the distance D to the tag, and the operation image 124 is deformed and displayed so as to have a similar shape as illustrated in FIG. do it.

(Second Embodiment)
In the first embodiment, it is necessary to register switch control information in advance in the tag information storage unit 132 for each switch 17 to be operated. However, it is not easy to register the switch control information of all the switches 17 in the living environment of the user 19 (within the movable range) in the tag information storage unit 132 in advance. Therefore, a second embodiment that can solve such a problem will be described below.

  In this embodiment, as schematically shown in FIG. 16, a non-contact information storage medium 221 such as an RFID is disposed in a room, a building, the entrance 212 of the area 211, the inner wall 213, or the like.

  As shown in FIG. 17, each non-contact information storage medium 221 includes a memory 231, a wireless communication unit 232, and a control unit 233. In the memory 231, switch control information for each switch 17 arranged in the room, building, or area 211 where the non-contact information storage medium 221 is installed is stored in advance. Upon receiving an inquiry from the reader / writer via the wireless communication unit 232, the control unit 233 reads the switch control information from the memory 231 and transmits the switch control information to the reader / writer via the wireless communication unit 232.

  Moreover, the information processing apparatus 13 of the switch control system 100 includes a wireless communication unit 136 for communicating with the non-contact information storage medium 221 as shown in FIG. Note that the information processing apparatus 13 may be configured to communicate with the non-contact information storage medium 221 via the wireless communication device 16.

  The control unit 134 of the information processing apparatus 13 communicates with the surrounding non-contact information storage medium 221 via the input unit 131 according to a user instruction or periodically, reads the switch control information, and stores it in the tag information storage unit 132. Store. Further, additional information / related information may be read from the DB server 20.

  The subsequent operation is the same as the operation of each device in the first embodiment described above.

  With such a configuration, the switch control system 100 communicates with the non-contact information storage medium 221 at the movement destination, and automatically reflects the switch control information at the movement destination in the processing shown in FIG. Therefore, an operation image is formed and displayed for each switch 17 in the field of view of the imaging device 15, and the operation can be performed.

(Modification of the second embodiment)
Each identification tag 18 may be composed of a non-contact information storage medium such as an RFID and store switch control information for controlling the corresponding switch 17.
In this case, the identification tag 18 has, for example, the configuration shown in FIG. 17 and stores switch control information for operating the switch 17 adjacent to the memory 231. Is drawn.
In addition, the information processing apparatus 13 includes a communication function for communicating with the identification tag 18.

The information processing device 13 communicates with the surrounding non-contact information storage medium 221 according to a user instruction or periodically, reads the switch control information, and stores it in the tag information storage unit 132. If there is additional information / related information, it is read from the DB server 20 as necessary.
The subsequent operation of the information processing apparatus 13 is the same as the operation in the first embodiment described above.
With such a configuration, the information processing device 13 communicates with the surrounding identification tag 18 to automatically acquire switch control information for controlling the surrounding switch 17 and make it operable. It is possible to cope with the case where the user moves.

  The adjacent identification tags 18 are preferably adjusted in advance so as to use different communication channels (frequency, time slot, etc.).

(Third embodiment)
In the above-described embodiment, the display device 12 is an installation type, but for example, a wearable head mounted display or the like can be used.
In this case, as shown in FIG. 19, the head mounted display (display device) 125 and the imaging device 15 may be integrated, and an image in a direction in which the user is facing may be acquired by the imaging device 15. Further, the acquired image may be displayed on the head mounted display 125, or the head mounted display 125 may be translucent to allow the outside to be seen.

In the case of this configuration, normally, the display device 12 is set in a transparent state so that the outside can be seen through the display device 12, and when an identification graphic is detected, an operation image 124 corresponding to the detected identification graphic is formed. Then, it may be displayed in a semi-transparent state at a position where the identification figure can be seen.
A steady operation panel may be displayed at the bottom of the display screen.

(Third embodiment)
In the above-described embodiment, an example in which the switch and the identification tag are associated with 1: 1 is shown, but it is also possible to associate 1: multiple or multi: 1.
Hereinafter, as an example, an example in which the switch to be operated and the identification tag are associated one to many will be described.

First, in the present embodiment, the switch to be operated is a controller of the vehicle 104 such as an electric wheelchair, and the operation assigned to the selected icon by designating the icon displayed on the display device 12. To the controller.
In this case, as shown in FIG. 20, the control data is transmitted from the switch control system 100 to the controller 101, and the controller 101 controls the drive mechanism (drive mechanism) 102 of the vehicle 104 according to the control data.

  Next, for example, a scene of moving on the road 105 illustrated in FIG. 21 is assumed. Identification tags 18 are arranged at the branch points 106 and 107 of the road 105.

  When the vehicle 105 moves and reaches the vicinity of the branch point 106, an image as illustrated in FIG. 22A-1 is displayed on the display device 12. Here, the control unit 134 of the information processing apparatus 13 detects an identification image (spade figure) in the image, and instructs the controller 101 to temporarily stop according to the corresponding switch control information. In accordance with the instruction, the controller 101 stops the drive mechanism 102. Further, the control unit 134 generates an icon as shown in FIG. 22A-2 according to the switch control information, and displays it on the display device 12 as shown in FIG. 22A-3. The user selects an icon that indicates the desired direction. The switch control device 100 detects the selected icon and instructs the controller 101. The controller 101 controls the drive mechanism 102 to advance the vehicle 104 in the instructed direction.

Subsequently, when the vehicle 105 moves and reaches the vicinity of the branch point 107, an image as illustrated in FIG. 22B-1 is displayed on the display device 12. The control unit 134 detects an identification image (heart shape) in the image, instructs the controller 101 to stop temporarily according to the corresponding switch control information, stops the vehicle 105,
Furthermore, an icon as shown in FIG. 22 (b-1) is generated and displayed on the display device 12 as shown in FIG. 22 (b-3). The user selects an icon that indicates the desired direction. The switch control device 100 detects the selected icon and instructs the controller 101. The controller 101 controls the drive mechanism 102 to advance the vehicle 104 in the instructed direction.

  Thus, in this embodiment, a plurality of identification tags 18 are associated with specific switches, and switch operations are associated with the identification tags 18. The identification tag 18 is recognized when the identification tag 18 and the switch control system 100 are close to each other to some extent, that is, when a switch operation is required, and at that time, an icon that indicates the necessary operation is displayed on the display device. 12, the drive mechanism 105 that is the operation target can be operated.

  According to this configuration, the vehicle 104 can be controlled without directly operating the steering wheel, operation lever, and the like of the vehicle 104. For example, the vehicle 104 can be operated even when the limb is disabled. .

  The identification tag 18 may be formed of a non-contact information storage medium (RFID or the like) that stores switch control information so that the identification tag 18 can be recognized when the switch control system 100 approaches a predetermined distance. .

(Fourth embodiment)
In the third embodiment, an example in which the user 19 controls the vehicle 104 by his / her own will has been described. However, the vehicle 104 and the like can be automatically controlled. Hereinafter, an example in which the vehicle 104 is automatically controlled will be described.

  In this example, it is assumed that the floors of the department stores are connected by slopes, and the vehicle 104 can move between the floors. Further, the expected movement route of the vehicle 104 is painted in a specific color on the floor surface of each floor.

  Further, as shown in FIG. 23, the vehicle 104 includes a route detection unit 311 for detecting a travel route, and a route memory 312 for storing a travel course in advance.

  Further, an identification tag 18 on which an identification graphic different from each other is drawn is arranged next to each “interior guide board” of the department store.

  The identification graphic of each identification tag 18 and the course from the position where the identification tag 18 is arranged to an arbitrary sales floor are stored in the path memory 312 in advance.

  In such a configuration, when the control unit 134 of the switch control system 100 detects the identification graphic on the identification tag 18 corresponding to the hall information board in the captured image, for example, as shown in FIG. The operation image 124 to be displayed is displayed. In this example, the identification figure drawn on the identification tag 18 is “spade”, the operation image 124 is an image corresponding to the in-house guidance, and “floor”, “product classification”, and the like are arranged as icons. .

  At the timing when the user 19 highlights an icon indicating the target floor (“1F” to “4F”, “rooftop”) or an icon indicating a product (“restaurant”, “clothing”). Be aware of that choice.

For example, if the user 19 is conscious of selection in the state of FIG. 24B, the icon “clothing” is selected, and the switch control system 100 notifies the controller 101 that “clothing” has been selected. Notice.
The controller 101 reads, from the path memory 312, a path from the installation position of the identification tag 18 on which the spade is drawn to a position (a predetermined area on 2F) where “clothing” is displayed.

Thereafter, the controller 101 controls the drive mechanism 102 while tracing the route painted on the floor by the route detection unit 311 to drive the vehicle 104 to a position where clothing items on the second floor are displayed. Note that moving the vehicle 104 to a destination along a predetermined route is the same as a method of controlling and driving a so-called automatic guided vehicle.
According to such a configuration, the vehicle 104 can be automatically controlled.

(Fifth embodiment)
In the third and fourth embodiments, the example of controlling the electric wheelchair has been shown, but the control target is arbitrary.
For example, in the case of a department store, the present invention can be applied to the case of controlling an elevator. Hereinafter, an example in which the present invention is applied to a system for controlling an elevator will be described.
In the present embodiment, an identification tag 18 is arranged in an elevator, for example, inside a door, and an operation image 124 as shown in FIG. 24A is associated with the identification tag 18.

  When the controller 134 detects an identification graphic corresponding to the hall information board in the captured image, the controller 134 displays a corresponding operation image 124 as shown in FIG. 24A, for example.

At the timing when the user 19 highlights an icon indicating the target floor (“1F” to “4F”, “rooftop”) or an icon indicating a product (“restaurant”, “clothing”). Be aware of that choice.
For example, if the user 19 is aware of the selection in the state of FIG. 24B, the icon “clothing” is selected, and the switch control system 100 determines that “clothing” has been selected. Detect that the clothing corresponds to the second floor. The control unit 134 controls the device control device 14 to transmit a command instructing “second floor” to the receiver 171 installed in the elevator.
The elevator controller 172 sets the destination floor to the second floor.

  According to this configuration, the elevator can be arbitrarily controlled.

(Sixth embodiment)
In the above embodiment, the identification image in the captured image is detected and the icon is displayed thereby. However, the non-contact information storage medium 221 may be detected to display the icon.
In this case, the non-contact information storage medium 221 has the configuration illustrated in FIG. 17, and the memory 231 stores, for example, switch control information having the configuration illustrated in FIG.
This switch control information does not include information related to the identification graphic. The wireless communication unit 232 has a communication distance of about several meters, for example.
Further, the switch control system 100 has a configuration shown in FIG. 18 or a configuration for communicating with the non-contact information storage medium 221 via the wireless communication unit 16.

  Next, the operation of the switch control system 100 having such a configuration will be described with reference to FIG. In this example, the display device 12 is a transmissive type, and includes, for example, a head mounted display as shown in FIG.

First, the control unit 134 of the information processing apparatus 13 communicates with a nearby non-contact information storage medium 221 via the wireless communication unit 136 (step S41).
Next, the control unit 134 determines whether or not there is a non-contact information storage medium 221 that can be communicated (step S42).

  When it is determined that the control unit 134 exists, the control unit 134 reads the switch control information from each detected non-contact information storage medium 221 (step S43). Note that information that cannot be stored in the non-contact information storage medium 221 or related information is stored in the tag information storage unit 132 and / or the DB of the DB server 20, and arbitrary information is read out to generate the operation image 124. You may use it.

  Next, the control unit 134 synthesizes the operation image 124 based on the read switch control information (step S44). That is, according to the operation image definition information included in each read switch control information, the figure of the frame is specified, and the operation image 124 is formed by arranging each icon and text at a designated position in the frame. .

  Next, when the operation image 124 is formed for all the non-contact information storage media 221 detected in step S41, the generated operation image 124 is in a selected state (default in the initial state) and a steady operation panel image 123. Are combined to synthesize an image for one frame (step S45).

  On the other hand, if it is determined in step S42 that the non-contact information storage medium 221 does not exist (step S42; No), the steady operation panel image 123 to be displayed is selected (step S46).

  Next, it is determined whether or not there is an icon that is currently highlighted on the image for one frame synthesized in step S45 or S46 (step S47).

  If there is an icon that is highlighted, the duration for which the icon is highlighted is updated (for example, one frame period is added), and it is further determined whether or not the duration is equal to or longer than a predetermined time. (Step S48). When it is determined that the duration time is less than the predetermined time (step S48; No), the highlighted display of the icon is maintained (step S49).

  Subsequently, the display image for one frame generated in this way is supplied to the display device 12 to be displayed (step S50). Thereafter, the process returns to step S41.

  On the other hand, if it is determined in step S48 that the duration of highlighting is equal to or longer than the predetermined time, another icon is specified and the duration is reset to 0 (step S51). Thereafter, the processing proceeds to step S49.

  If it is determined in step S47 that there is no emphasized icon (step S47; No), the emphasized icon is specified and the duration is reset to 0 (step S52). Thereafter, the processing proceeds to step S21.

  As a result of such processing, an operation image 124 for enabling the switch operation specified by the switch control information stored in the non-contact information storage medium 221 within a certain distance is illustrated in FIG. ,Is displayed.

Then, the icon on the operation image 124 and the icon on the steady operation panel image 123 are sequentially highlighted.
Then, when the user 19 desires to select the icon at the timing when the desired icon is highlighted, a change appears in the electroencephalogram, the change is detected, and the icon highlighted at the timing is displayed. The operation identified and associated with the icon is executed.

  According to such a configuration, the switch control information corresponding to the switch 17 to be operated is stored and arranged in the non-contact information storage medium 221, thereby enabling remote operation of each switch.

  In the above description, an example of detecting an electroencephalogram change and selecting an icon has been described. However, a method for specifying an icon is arbitrary.

  For example, the icon being watched may be discriminated by processing the electroencephalogram with a discriminant function. Further, the electroencephalogram is not limited to that measured by the electrode on the scalp but may be measured by the subdural electrode or the like. Furthermore, this system can be operated using other brain function measurement methods such as a magnetoencephalograph. Further, the signal is not limited to an electroencephalogram but may be a signal from any muscle.

Further, it may be a delicate movement of each part of the body, for example, a finger or a head.
Also, for example, as shown in FIG. 27, an imaging device that captures the face of the user 19 is arranged, the direction of the line of sight of the user 19 is determined from the image captured by the imaging device, and the icon at the position of the line of sight May be selected. In this case, it is not necessary to highlight the icons sequentially.

  As described above, the control system / method of the present invention is not limited to the control of switches in a narrow sense such as a changeover switch, on / off switch, volume switch, etc. It is generally applicable to what is given. In addition, this is particularly effective when the user 19 is a physically handicapped person or the like, but the user's attributes are arbitrary, and it is also effective for use by a healthy person.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can be arbitrarily applied and modified.

It is a block diagram which shows the structure of the switch control system which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the electroencephalogram measurement apparatus shown in FIG. (A), (b) is a figure which shows the example of the screen displayed on the display apparatus of FIG. It is a block diagram which shows the structural example of the information processing apparatus shown in FIG. (A), (b) is a figure which shows the example of an operation image. (A), (b) is a figure which shows the example of the figure of the frame as a raw material which comprises an operation image. (A)-(f) is a figure which shows the example of an icon. (A) is a figure which shows the structural example of switch control information, (b) And (c) is a figure which shows the structural example of switch image definition information which comprises switch control information. It is a block diagram which shows the structural example of the apparatus control apparatus shown in FIG. (A), (b) is a block diagram which shows the structural example of the switch shown in FIG. 2 is a flowchart for explaining image display processing of the switch control system shown in FIG. 1. It is a flowchart for demonstrating the attention icon detection process of the switch control system shown in FIG. (A)-(e) is a figure for demonstrating the example of the emphasis display of an icon. It is a figure which shows the other example of a display image. (A)-(c) is a figure for demonstrating the example which deform | transforms and displays a switch operation image and an icon. It is a top view of the room for demonstrating the example which arrange | positions a non-contact information storage medium. It is a block diagram which shows the structural example of a non-contact information storage medium. FIG. 12 is a block diagram illustrating another configuration example of the information processing apparatus. It is a figure which shows the other example of a display apparatus. It is a block diagram for demonstrating the structure which controls vehicles (electric wheelchair etc.) by a switch control system. It is a figure for demonstrating the structure which controls a vehicle by a switch control system, and is a figure of a road. (A-1)-(b-3) is a figure for demonstrating the structure which controls a vehicle by a switch control system, and is a figure which shows the example of a display image. It is a block diagram for demonstrating the structure in the case of controlling a vehicle by a switch control system. (A), (b) is a figure for demonstrating the structure which controls a vehicle by a switch control system, and is a figure which shows the example of a display image. It is a figure which shows the other example of switch control information stored in a non-contact information storage medium. It is a flowchart for demonstrating the other example of the image display process of a switch control system. It is a block diagram for demonstrating the other example of a structure of a switch control system.

Explanation of symbols

11 EEG measurement device 12 Display device 13 Information processing device 14 Device control device 15 Imaging device 16 Wireless communication device 17 Switch 18 Identification tag 19 User 20 Database (DB) server 100 Switch control system 111 Probe 112 Signal processing circuit 113 Output unit 121 Operation screen 122 Captured image 124 Operation image 123 Steady operation panel image 125 Head mounted display 131 Input unit 132 Tag information storage unit 133 Communication unit 134 Control unit 135 Brain wave storage unit 141 Communication unit 142 Input / output unit 143 Control unit 171 Receiver 172 Controller 173 Actuator

Claims (10)

Detection means for detecting information indicating the presence of the operation target;
In response to detection of information indicating the presence of the control target by the detection means, reading means for reading out information specifying an operation image including an icon from the storage means;
Display control means for generating an operation image including an icon according to the information read by the reading means, and displaying the generated operation image on a display device;
Of the icons included in the operation image displayed on the display device, attention icon determination means for determining the icon that the user is paying attention to,
Control output means for outputting a control signal corresponding to the icon determined by the attention icon determination means;
A control system comprising: The detection means is composed of specific graphic detection means for inputting an image signal and detecting a specific graphic in the image represented by the image signal,
The storage means stores a specific figure and operation image specifying information in association with each other,
The reading means reads operation image specifying information corresponding to the specific graphic detected by the specific graphic detecting means,
The display control means displays an operation image specified by the operation image specifying information read by the reading means on the display device.
The control system according to claim 1.   The control system according to claim 1, wherein the detection unit includes a unit that detects a radio signal that indicates the presence of an operation target. The detection means includes means for detecting a non-contact information storage medium arranged in the vicinity of the operation target,
The control system according to claim 1, wherein the reading unit reads information specifying an operation image stored in a non-contact information storage medium detected by the detection unit.   The display control means receives an image signal from an imaging device, combines an image defined by the received image signal and an operation image specified by operation image specifying information, and displays the synthesized image on the display device. The control system according to claim 1. The display device is composed of a transmissive display device,
The control system according to claim 1, wherein the display control unit displays an operation image on the display device. The attention icon discrimination means
Including means for acquiring biometric information of the user and determining which icon of the icons included in the operation image displayed on the display means is focused based on the acquired biometric information.
The control system according to claim 1. The display control means sequentially highlights icons included in the operation image,
The attention icon determination means includes a means for acquiring the user's biological information, detecting a change in the acquired biological information, and a means for determining an icon that is in an emphasized state at the time of the change as an attention icon.
The control system according to claim 1. The attention icon discrimination means
At least one of a means for acquiring a user's brain wave signal and determining an icon that the user is paying attention to based on the acquired brain wave signal and a means for determining a user's line of sight and determining an icon on the line of sight With one side,
The control system according to claim 1. Detect the presence of the operation target,
In response to detection, read the information that identifies the operation image,
According to the read information, an operation image including an icon is generated and displayed,
Determine which icon the user is paying attention to,
Output a control signal corresponding to the determined icon to the operation target;
A control method characterized by that.