JP2021051760A - Control device, control method, and program - Google Patents

Abstract

To allow for adaptively changing a user interface mode for utilizing a projected image depending on where the image is projected.SOLUTION: A control device includes a projection controller configured to control projection of an image onto a projection surface by a projector, and a UI controller configured to switch the user interface (UI) mode associated with the projected image between two or more UI modes on the basis of the positional relationship of the projector with the projection surface.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to control devices, control methods and programs.

Conventionally, a projector that projects an image onto a projection surface is known. Patent Document 1 not only projects an image onto a projection surface and shows it to the user, but also acquires information added on the projection surface by the user through a camera, and links and records the acquired information on the projected image. We are proposing a method.

Japanese Unexamined Patent Publication No. 2003-044839

The inventors of the technique according to the present disclosure recognize that the method proposed in Cited Document 1 can also be applied to a user interface via a projected image. However, what kind of user interface is desirable depends on where the image is projected.

According to the present disclosure, a projection control unit that controls projection of an image onto a projection surface by a projector and a user interface (UI) related to the projected image based on the positional relationship of the projector with respect to the projection surface. A UI control unit that switches a mode between two or more UI modes is provided, and the projector detects a touch position by a user by using an exploration light emitted toward an exploration region in the vicinity of the projector. A control device having a first user input detection system to be used, and the UI control unit selects a first UI mode using the first user input detection system based on a measurement result of a sensor included in the projector. Is provided.

Further, according to the present disclosure, in a projection control device that controls the projection of an image onto a projection surface by a projector, a user interface (UI) related to the projected image based on the positional relationship of the projector with respect to the projection surface. ) Is switched between two or more UI modes, the projector detecting a user's touch position using exploration light emitted towards an exploration region in the vicinity of the projector. A control method is provided which has a first user input detection system and selects a first UI mode using the first user input detection system based on a measurement result of a sensor included in the projector.

Further, according to the present disclosure, the image projected by the processor of the projection control device based on the positional relationship between the projection control unit that controls the projection of the image on the projection surface by the projector and the projector with respect to the projection surface. The user interface (UI) mode associated with the above functions as a UI control unit that switches between two or more UI modes, and the projector emits exploration light toward an exploration region in the vicinity of the projector. Has a first user input detection system that detects a touch position by the user, and the UI control unit uses the first user input detection system based on the measurement result of the sensor of the projector. A program is provided for the function of selecting one UI mode.

According to the technique according to the present disclosure, it is possible to provide a user with a desirable user interface according to the positional relationship of the projector with respect to the projection surface.
It should be noted that the above effects are not necessarily limited, and either in combination with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.

It is the schematic which shows the appearance of the projection control device which concerns on one Embodiment. It is explanatory drawing for demonstrating the 1st installation posture of a projection control device. It is explanatory drawing for demonstrating the 2nd installation posture of a projection control device. It is explanatory drawing for demonstrating the outline of the touch input detection method which can be adopted in the 1st user input detection system. It is a block diagram which shows an example of the detailed structure of the 1st user input detection system. It is explanatory drawing for demonstrating the operation of the polarization beam splitter shown in FIG. It is explanatory drawing for demonstrating the detection of a touch position using an exploration image. It is a block diagram which shows an example of the structure of the projection control device which concerns on one Embodiment. It is explanatory drawing for demonstrating the 1st example of switching of a UI mode based on the positional relationship of a projector with respect to a projection plane. It is explanatory drawing for demonstrating the 2nd example of switching of UI mode based on the positional relationship of the projector with respect to a projection plane. It is explanatory drawing for demonstrating the 3rd example of the switching of the UI mode based on the positional relationship of the projector with respect to the projection plane. It is explanatory drawing which shows an example of the user input which can be detected by the 1st user input detection system. It is explanatory drawing which shows the 1st example of the user input which can be detected by the 2nd user input detection system. It is explanatory drawing which shows the 2nd example of the user input which can be detected by the 2nd user input detection system. It is explanatory drawing for demonstrating an example of the user interface for rotating a projected image. It is explanatory drawing for demonstrating an example of the projection image which changes based on the input condition which does not depend on the user input. It is a flowchart which shows an example of the flow of the user interface control processing which concerns on one Embodiment. It is a flowchart which shows 1st example of the detailed flow of a UI mode selection process. It is a flowchart which shows the 2nd example of the detailed flow of the UI mode selection process. It is a flowchart which shows the 3rd example of the detailed flow of the UI mode selection process. It is a flowchart which shows the 4th example of the detailed flow of the UI mode selection process. It is a flowchart which shows the 5th example of the detailed flow of the UI mode selection process.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

In addition, explanations will be given in the following order.
1. 1. Outline of the device 1-1. Appearance of the device 1-2. Various installation postures 1-3. Mechanism of touch input detection system 2. Device configuration 2-1. Overall configuration 2-2. Switching UI mode 2-3. Example of user input 2-4. Input conditions that do not depend on user input 3. Process flow 3-1. UI control processing 3-2. UI mode selection process 4. Summary

<1. Device overview>
[1-1. Appearance of the device]
FIG. 1 shows the appearance of the projection control device 100 according to the embodiment. The projection control device 100 is a so-called projector that projects an image onto a projection surface existing outside the device. In the example of FIG. 1, the projection control device 100 has a rectangular parallelepiped shape, and can be placed on some plane with any surface as the bottom surface. The shape of the projection control device 100 shown in FIG. 1 is only an example. The projection control device 100 may have any shape as long as it can be stably installed in the real space.

One surface of the housing of the projection control device 100 is an injection surface 10a. The injection surface 10a is provided with an opening 11 for projected light and an opening 12 for exploration light. The projection control device 100 further includes a camera 13, a microphone 14, and a speaker 15. The projection control device 100 forms a projected image on the projection surface by projecting the projected light onto the projection surface through the opening 11 for the projection light. Further, as will be described in detail later, the projection control device 100 detects the touch input by the user by emitting the exploration light toward the exploration region near the projector through the opening 12 for the exploration light.

In the present embodiment, the projection control device 100 includes two user input detection systems 17 and 18. The first user input detection system 17 detects the touch input by the user with relatively high detection accuracy by using the above-mentioned exploration light. The second user input detection system 18 may have a coarser touch detection accuracy than the first user input detection system, or may not be able to detect touch. The second user input detection system 18 may detect the gesture input from the user based on the image captured by the camera 13, or detect the voice input from the user based on the voice collected by the microphone 14. You may.

[1-2. Various installation postures]
The projection control device 100 can be installed in the real space in various installation postures. In this section, two typical installation postures of the projection control device 100 will be described.

FIG. 2A is an explanatory diagram for explaining the first installation posture of the projection control device 100. Referring to FIG. 2A, the projection control device 100 is mounted on the table surface 3a in the first installation posture. In the first installation posture, the injection surface 10a of the projection control device 100 is substantially orthogonal to the table surface 3a, which is the installation surface. That is, when the table surface 3a is a horizontal plane, the injection surface 10a is a vertical surface. In the first installation posture, the projection control device 100 can project the projected image 2a on the table surface 3a. In this case, the table surface 3a becomes the projection surface below the projection control device 100. In the example of FIG. 2A, three users surround the table, and each user can see the projected image 2a on the table surface 3a.

FIG. 2B is an explanatory diagram for explaining the second installation posture of the projection control device 100. Referring to FIG. 2B, the projection control device 100 is installed on the floor surface in the second installation posture. In the second installation posture, the injection surface 10a of the projection control device 100 is the upper surface of the projection control device 100. That is, when the floor surface is a horizontal plane, the injection surface 10a is also a horizontal plane. In the second installation posture, the projection control device 100 can project the projected image 2b on the wall surface 3b. In the example of FIG. 2B, the user is located on the opposite side of the wall surface 3b with respect to the projection control device 100, and the user can see the projected image 2b on the wall surface 3b.

Comparing the first installation posture of FIG. 2A and the second installation posture of FIG. 2B, it is easier for the user to touch the table surface 3a, which is the projection surface, in the first installation posture. In the second installation posture, it is not so easy to touch the wall surface 3b which is the projection surface. As an example of another installation posture, when the projection control device 100 is installed upside down from the first installation posture, the projection control device 100 can project a projected image onto the ceiling surface. Again, it is not so easy for the user to touch the ceiling surface, which is the projection surface.

Thus, depending on the position of the projector with respect to the projection plane, touch input to the projection plane may be suitable for the user interface (UI), and such touch input may not be suitable for the UI. There is also a case. Therefore, the technique according to the present disclosure incorporates a mechanism for adaptively changing the method of the user interface utilizing the projected image depending on where the image is projected, as will be described in detail in the present specification.

[1-3. Mechanism of touch input detection system]
Here, an example of a touch detection method for detecting a touch input to the projection surface will be described. The method described here is only an example, and other touch detection methods may be used.

(1) Outline of Touch Input Detection Method FIG. 3 is an explanatory diagram for explaining an outline of a touch input detection method that can be adopted in the first user input detection system 17. In FIG. 3, the projection control device 100 is placed on the table surface 3a in the first installation posture as in the example of FIG. 2A. The projection control device 100 projects the projected image 21 onto the table surface 3a by emitting the projected light through the opening 11 for the projected light. Further, the projection control device 100 forms the exploration surface 22 by emitting the exploration light toward the exploration region near the projector through the opening 12 for the exploration light. The exploration light may be invisible light such as infrared light.

The opening 12 for the exploration light is provided, for example, slightly above the bottom surface of the projection control device 100 in the first installation posture (for example, the height from the bottom surface may be about several mm to several tens of mm). The exploration surface 22 formed by the trajectory of the exploration light may be substantially parallel to, for example, the bottom surface of the projection control device 100, in which case the exploration surface 22 is a short distance from the table surface 3a which is the projection surface. It becomes a plane that is almost parallel to each other. Such an exploration surface 22 may be understood as a barrier film composed of trajectories of exploration light that covers the projected image 21. Normally, the projection plane is flat (however, the technique according to the present disclosure can be applied even in cases where the projection plane is not always flat). If there is no shield on or near the projection surface, the exploration light passes over the projected image 21 without being shielded. However, when the user touches the projected image with a finger or other operator, the exploration light is diffusely reflected by the operator and heads in all directions. The first user input detection system 17 of the projection control device 100 receives the exploration light reflected in this way through the opening 11 for the projection light, and is arranged at a position optically conjugate with the light valve of the projector. Detect with a sensor. As a result, the projection control device 100 can detect where on the projected image is touched (pointed) with an accuracy close to that of a general type touch sensor such as a resistive film type or a capacitance type. ..

(2) Configuration of Detection System FIG. 4 is a block diagram showing an example of a detailed configuration of the first user input detection system 17. In the example of FIG. 4, the first user input detection system 17 is a part of the projector module 110. The projector module 110 includes an image projection system 16, a first user input detection system 17, a polarization beam splitter 46, a projection lens 47, and a projector controller 55. The image projection system 16 includes a first light source 41, a light bulb 42, and a first signal processing unit 43.
The first user input detection system 17 includes a second light source 31, a detection lens unit 51, an image sensor 52, and a second signal processing unit 53. The second light source 31 is arranged in the projector submodule 110a connected to the opening 12 for exploration light, while the remaining components of the projector module 110 shown in FIG. 4 are connected to the opening 11 for projected light. It is arranged in the projector sub-module 110b.

The first light source 41 is a light source that generates visible light, and irradiates the generated visible light toward the light bulb 42 via a polarizing beam splitter 46. The visible light generated by the first light source 41 may be, for example, laser light containing three color components of red, green, and blue. The light bulb 42 may be a reflective liquid crystal device such as LCOS (Liquid Crystal On Silicon). The light valve 42 modulates, for example, the first polarization component of visible light incident from the first light source 41 (for example, the s polarization component Ls1 described later) based on the image signal input from the first signal processing unit 43. , The second polarization component (for example, the p-polarization component Lp2 described later) is rotated. Then, visible light (modulated light) having a second polarization component is emitted toward the projection lens 47 via the polarization beam splitter 46. The light bulb 42 can also express black by reflecting the first polarized component of the incident visible light in the polarized state as it is.

The projection lens 47 forms a projected image 21 on the projection surface by projecting the modulated light 21a incident from the light valve 42 through the polarizing beam splitter 46 toward the projection surface through the projection light opening 11. Further, the exploration light 22b (reflected by some operator) is incident on the projection lens 47 from the direction opposite to the traveling direction of the modulated light 21a through the projection light opening 11. The exploration light 22b is further incident on the polarizing beam splitter 46 via the projection lens 47.

The polarization beam splitter 46 separates the incident probe light into a first polarization component (for example, an s polarization component) and a second polarization component (for example, a p-polarization component), and directs each of them in different directions. For example, the first polarization component of the exploration light is selectively reflected by the polarization beam splitter 46 and incident on the detection lens unit 51. On the other hand, the second polarization component of the exploration light is selectively transmitted by the polarization beam splitter 46 and incident on the light bulb 42. In addition, instead of the polarization beam splitter 46, another type of polarization separation element such as a wire grid may be used.

The operation of the polarization beam splitter shown in FIG. 4 will be further described with reference to FIG. The polarization beam splitter 46 has a first optical surface 46a, a second optical surface 46b, a third optical surface 46c and a fourth optical surface 46d, and a polarization separation surface 46e. The first optical surface 46a and the third optical surface 46c are arranged back to back in the left-right direction in the drawing, and the second optical surface 46b and the fourth optical surface 46d are arranged back to back in the vertical direction in the drawing. The first optical surface 46a faces the first light source 41 (not shown in FIG. 5). The second optical surface 46b faces the light bulb 42. The third optical surface 46c faces the detection lens unit 51. The fourth optical surface 46d faces the projection lens 47 (not shown in FIG. 5).

The visible light L1 emitted from the first light source 41 is incident on the first optical surface 46a, and the first polarization component (s polarization component Ls1) of the visible light L1 is reflected by the polarization separation surface 46e to be reflected on the second optical surface. Emit from 46b. The second polarized light component (p-polarized light component Lp1) of the visible light L1 passes through the polarization separation surface 46e and is emitted from the third optical surface 46c. The modulated light Lp2 from the light bulb 42 is incident on the second optical surface 46b, and the second polarized light component (p polarized light component Lp2) of the modulated light Lp2 is emitted from the fourth optical surface 46d.

On the other hand, the exploration light L3 is incident on the fourth optical surface 46d, and the first polarization component (s polarization component Ls3) of the exploration light L3 is reflected by the polarization separation surface 46e and emitted from the third optical surface 46c. The second polarization component (p polarization component Lp3) of the exploration light L3 passes through the polarization separation surface 46e and is emitted from the second optical surface 46b. The first polarization component Ls3 of the exploration light L3 passes through the relay lenses 51a and 51b of the detection lens unit 51, and a visible light cut filter and a bandpass filter (not shown), and reaches the image sensor 52. The bandpass filter selectively passes light in a specific wavelength region (including the wavelength of the exploration light).

Referring again to FIG. 4, the second light source 31 is a light source that generates exploration light 22a for exploring the controls used for touch input, and transfers the generated exploration light to the exploration region near the projector. Irradiate toward. As described above, the exploration light 22a may be invisible light such as infrared light (for example, near infrared light). The exploration light 22a from the second light source 31 is emitted radially through, for example, the opening 12 for the exploration light so as to form an exploration surface 22 substantially parallel to the bottom surface of the projection control device 100 in the example of FIG. .. Then, the first polarization component of the exploration light 22b reflected by the operator touching the projected image 21 is further reflected by the polarization beam splitter 46 and reaches the detection lens unit 51 as described above.

The detection lens unit 51 includes one or more relay lenses that form an image of the exploration light 22b at a position optically conjugate with the light bulb 42. For the purpose of miniaturizing the image sensor 52, the detection lens unit 51 may reduce the image of the exploration light 22b and equivalently reduce the image of the exploration light 22b. The image sensor 52 acquires an image of the exploration light 22b formed at a position optically conjugate with the light bulb 42 to generate an exploration image signal. The image sensor 52 may be any type of image sensor such as CMOS (Complementary Metal-Oxide Semiconductor) or CCD (Charge Coupled Device).

The second signal processing unit 53 executes predetermined signal processing such as development and noise removal on the search image signal output from the image sensor 52. Then, the second signal processing unit 53 outputs the processed search image signal to the projector controller 55.

The first signal processing unit 43 causes the driver circuit of the light bulb 42 to supply a voltage to each pixel of the light bulb 42 according to the projected image signal input via the projector controller 55. As a result, the above-mentioned visible light (projected light) is modulated, and the projected image 21 is reproduced on the projection surface.

The projector controller 55 outputs a projected image signal supplied from the outside of the projector module 110 to the first signal processing unit 43, and projects the projected image 21 onto the projection surface through the image projection system 16. The projector controller 55 may perform general projection control functions such as autofocus and keystone correction. Further, in the present embodiment, the projector controller 55 detects the touch input on the projected image based on the exploration image represented by the exploration image signal input from the first user input detection system 17.

FIG. 6 is an explanatory diagram for explaining the detection of the touch position using the exploration image. Also in the example of FIG. 6, the exploration surface 22 is formed so as to cover the projected image 21. The user touches _{the position Pprj} on the projected image 21 with, for example, a finger. The exploration light reflected by the finger touching the position P _prj is captured by the image sensor 52 and forms an _{operator image R fin in the exploration image 23.} For example, the projector controller 55 executes a known feature point detection method for _{such an operator image R fin} , and detects a _{touch position P tou in the exploration image 23.} The feature point position (that is, the touch position) may be, for example, the position of the tip of the finger or the position of the center of gravity of the _{operator image R fin.} As described above, since the exploration image is captured by the image sensor 52 arranged at a position substantially optically conjugate with the light bulb 42, _{the coordinates of the touch position P tou} in the exploration image 23 are the projection image 21. _{It matches} the coordinates of the touch position Pprj in.

By using the first user input detection system 17 as described above, the touch input on the projection surface can be detected with high detection accuracy. In particular, in the first installation posture shown in FIG. 2A, the operator normally reflects the exploration light only in the immediate vicinity of the projection surface, so that the touch input detection accuracy is close to that of a general type touch sensor. Can be achieved. On the other hand, in the second installation posture shown in FIG. 2B, the exploration surface is not necessarily in the vicinity of the projection surface, and even if the user touches the projection surface, the touch position is equivalent to the case of the first installation posture. It is not easy to detect with the accuracy of. Touch detection may be hindered as a result of obstacles other than the controls (eg, parts of the user's body) intervening between the projection plane and the projector. In the embodiment described in detail in the next section, the positional relationship of the projector with respect to the projection surface is dynamically determined in consideration of such circumstances, and the user interface mode is set to two or more UI modes based on the determination result. Can be switched between.

<2. Device configuration>
[2-1. Overall configuration]
FIG. 7 is a block diagram showing an example of the configuration of the projection control device 100 according to the embodiment. Referring to FIG. 7, the projection control device 100 includes a projector module 110, a recognition module 120, a sensor module 130, a software module 140, a communication interface 150, and integration in addition to the camera 13, microphone 14 and speaker 15 shown in FIG. A control unit 160 is provided. The projector module 110 includes the image projection system 16 and the first user input detection system 17 described above. The camera 13, the microphone 14, and the speaker 15 together with the recognition module 120 form a second user input detection system 18.

(1) Projector Module The projector module 110 is a module that projects an image on a projection surface and controls the projection. An example of the detailed configuration of the projector module 110 is as described with reference to FIGS. 4 and 5. For example, under the control of the integrated control unit 160, the projector module 110 projects an image generated by the software module 140, which will be described later, toward the projection surface through the image projection system 16. The projected image may be a still image or each of the frames constituting the moving image.

As described above, the first user input detection system 17 of the projector module 110 can detect the touch position by the user by using the exploration light emitted toward the exploration region in the vicinity of the projection control device 100. When the projection control device 100 is installed in the first installation posture described above, the exploration area forms an exploration surface substantially parallel to the installation surface. On the other hand, when the projection control device 100 is installed in the second installation posture described above, the exploration region forms an exploration surface substantially orthogonal to the installation surface. When the projection plane is substantially parallel to the exploration plane and the distance between the projection plane and the exploration plane is small, the first user input detection system 17 can achieve a sufficient level of touch detection accuracy. .. When the projection planes are substantially parallel to the exploration planes, in addition to the fact that the planes are strictly parallel to each other, the touch detection error due to the angle formed by the planes (for example, for touch input adaptive software described later). ) Including being within an acceptable range. When the first user input detection system 17 detects the touch input, the first user input detection system 17 notifies the integrated control unit 160 of the occurrence of the touch event. The touch event information may include an event type such as touch, tap, double tap, flick, pinch in or pinch out, and one or more touch position coordinates associated with the event. In a certain UI mode, the first user input detection system 17 can be used as a main input interface. In other UI modes, the first user input detection system 17 may be used as an auxiliary input interface or may be disabled.

(2) Recognition module The recognition module 120 is a module that executes recognition processing for detecting user input in the second user input detection system 18. The recognition module 120 may detect the user input based on the image input from the camera 13, for example. The camera 13 is typically arranged in the projection control device 100 so that it has an angle of view toward the direction in which the user is assumed to be present in the second installation posture of the projection control device 100. In the example of FIG. 2B, the camera 13 is arranged on a surface opposite to the projection surface with reference to the projection control device 100. In another example, the camera 13 may be disposed on a side surface perpendicular to the projection plane. Further, the camera 13 may be an all-sky camera or a half-sky camera. The recognition module 120 may detect the gesture input as the user input by recognizing the gesture of the user reflected in the captured image input from the camera 13, for example. Further, the recognition module 120 may detect the line-of-sight input as the user input by recognizing the movement of the user's eyes reflected in the captured image input from the camera 13. When user input is detected based on such image input, even if it is possible to virtually touch (or point) a point on the projected image by user input, the touch detection accuracy (pointing accuracy) is It is relatively coarse as compared with the first user input detection system 17. In some situations, touch detection (pointing at the desired point) is not possible.

Additional or alternative, the recognition module 120 may detect the user input based on, for example, the voice input from the microphone 14. The recognition module 120 can detect a voice command as a user input by recognizing a predetermined voice pattern included in the voice collected by the microphone 14, for example. When user input is detected based on such voice input, pointing of a desired point on the projection plane is impossible.

When the recognition module 120 detects a user input that can correspond to a gesture input, a line-of-sight input, or a voice command, the recognition module 120 notifies the integrated control unit 160 of the occurrence of a user input event. The type of user input that can be detected by the second user input detection system 18 is not limited to the above-mentioned example. For example, the second user input detection system 18 may detect user input via a mechanical interface such as a button, switch or lever arranged on the device, or recognize vibration of the device or exhalation of the user. User input may be detected through.

(3) Sensor module The sensor module 130 includes one or more sensors used for determining the positional relationship of the projection control device 100 with respect to the projection surface. As an example, the sensor module 130 may include a posture sensor 131 that measures the posture of the projection control device 100. The attitude sensor 131 may be, for example, an acceleration sensor or a gyro sensor. The posture sensor 131 measures the tilt angle of the projection control device 100 at that time with respect to the reference posture of the projection control device 100, and outputs the measurement result to the integrated control unit 160.

As another example, the sensor module 130 uses a depth sensor 133 that measures the distance from the projection control device 100 to the projection surface (hereinafter referred to as the depth of the projection surface) instead of the attitude sensor 131 or in addition to the attitude sensor 131. Can include. The depth sensor 133 may be any type of depth sensor, for example, a stereo camera type or a TOF (Time of Flight) type. The depth sensor 133 typically measures the depth of the projection surface with reference to the position of the projection control device 100, and outputs the measurement result to the integrated control unit 160.

(4) Software module The software module 140 has a processor such as a CPU (Central Processing Unit) and a memory that can include a ROM (Read Only Memory) and a RAM (Random Access Memory), and generates an image to be projected. Run the software. As an example, the software module 140 selectively executes the touch input adaptive software 141 and the touch input non-adaptive software 143 under the control of the integrated control unit 160. The touch input adaptive software 141 may include touch input adaptive basic software (eg, a mobile operating system such as Android®, iOS or Windows Mobile®). Further, the touch input adaptive software 141 may include touch input adaptive application software. Touch input adaptive application software includes various software such as web browsers, mailers, text editors or instant messengers designed to run on mobile devices such as smartphones or tablet PCs (Personal Computers). Good.

The touch input non-adaptive software 143 may be any software that does not require fine pointing to operate. The touch input non-adaptive software 143 may also include any combination of basic software and application software. Some examples of touch input non-adaptive software 143 will be further described later.

(5) Communication interface The communication interface 150 mediates communication between the projection control device 100 and other devices. The communication interface 150 is a cellular communication protocol such as WiMAX, LTE (Long Term Evolution) or LTE-A (LTE-Advanced), or a wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). Other wireless communication protocols may be implemented. Further, the communication interface 150 may implement a wired communication protocol such as USB (Universal Serial Bus) or HDMI (High-Definition Multimedia Interface). The projector module 110 may project an image received from another device via the communication interface 150 onto the projection surface instead of the image generated by the software module 140.

(6) Integrated Control Unit The integrated control unit 160 has a processor such as a CPU and a memory that can include a ROM and a RAM, and controls the overall operation of the projection control device 100. For example, the integrated control unit 160 causes the software module 140 to execute any software when the projection control device 100 is activated. In addition, the integrated control unit 160 causes the projector module 110 to project an image generated by the executed software (for example, an image of an operating system or an image of an application) onto a projection surface. Further, when the integrated control unit 160 is notified of some user input event from the first user input detection system 17 or the second user input detection system 18, the integrated control unit 160 passes the user input event to the executing software. When the notified user input event indicates a specific user input, the integrated control unit 160 may process the user input event by itself instead of passing the user input event to the software (for example, stop operation). The power of the projection control device 100 is turned off in response to the instructed voice command).

In the present embodiment, the integrated control unit 160 determines the positional relationship of the projection control device 100 with respect to the projection surface, and based on the determination result, sets two user interface (UI) modes related to the projected image. Switch between the above UI modes. The first UI mode is a mode in which the first user input detection system 17 is used as the main input interface. The second UI mode is a mode in which the first user input detection system 17 is not used as the main input interface. In the following description, the first UI mode is referred to as "touch mode" and the second UI mode is referred to as "non-touch mode".

The integrated control unit 160 may cause the software module 140 to execute different software depending on whether the touch mode or the non-touch mode is selected. For example, the integrated control unit 160 may cause the software module 140 to execute the touch input adaptive software 141 when the touch mode is selected. Further, the integrated control unit 160 may cause the software module 140 to execute the touch input non-adaptive software 143 when the non-touch mode is selected.

[2-2. UI mode switching]
The integrated control unit 160 typically sets the touch mode when it is determined from the positional relationship of the projection control device 100 with respect to the projection surface that the first user input detection system 17 can achieve a certain level of touch detection accuracy. select. For example, as illustrated in FIG. 2A, when the projection control device 100 is placed on the table surface in the first installation posture and the image is projected onto the table surface, the exploration surface is in the vicinity of the projected image. Since it is formed so as to cover the projected image, the first user input detection system 17 can achieve a level of touch detection accuracy sufficient for the touch input adaptive software 141. Therefore, the integrated control unit 160 may select the touch mode in this case. Such a selection may be made based on a simple determination using the attitude sensor 131 or the depth sensor 133 that the projection plane is located substantially horizontally below the projection control device 100. On the other hand, as illustrated in FIG. 2B, when the projection control device 100 is installed in the second installation posture, there is a certain distance between the exploration surface and the projection surface of the projection control device 100, and the first It can be estimated that the user input detection system 17 does not achieve a sufficient level of touch detection accuracy. Therefore, the integrated control unit 160 may select the non-touch mode in this case.

As a first example of determining the positional relationship, the integrated control unit 160 controls the projection when the measurement result of the attitude of the projection control device 100 input from the attitude sensor 131 indicates that the search surface is substantially horizontal. It may be presumed that the device 100 is installed on the projection surface in the above-mentioned first installation posture. FIG. 8A is an explanatory diagram for explaining a first example of switching the UI mode based on the positional relationship of the projector with respect to the projection surface. Here, among the surfaces of the housing of the projection control device 100 orthogonal to the injection surface 10a described with reference to FIG. 1, the surface 10b closest to the opening 12 for exploration light (lower side in the drawing) is in the posture. It shall be the reference plane for measurement. The reference plane 10b is substantially parallel to the exploration plane. On the left side of FIG. 8A, the measurement result input from the attitude sensor 131 indicates that the reference surface 10b is horizontal and that the normal line of the reference surface 10b points in the direction of gravity. This measurement result may mean that the projection control device 100 is installed in the first installation posture on a horizontal plane such as a table surface. Therefore, the integrated control unit 160 can select the touch mode based on the measurement result. On the other hand, on the right side of FIG. 8A, the measurement result input from the attitude sensor 131 indicates that the reference plane 10b is perpendicular to the horizontal plane. This measurement result may mean that the projection control device 100 is not installed in the first installation posture. Therefore, the integrated control unit 160 may select the non-touch mode based on the measurement result. Table 1 shows the framework for selecting the UI mode based on the attitude of the projector.

As a second example of determining the positional relationship, the integrated control unit 160 determines that the measurement result indicating the depth of the projection surface input from the depth sensor 133 indicates that the distance between the search surface and the projection surface is small. , The projection control device 100 may be simply determined to be installed on the projection surface in the above-mentioned first installation posture. FIG. 8B is an explanatory diagram for explaining a second example of switching the UI mode based on the positional relationship of the projector with respect to the projection surface. Here, the depth sensor 133 measures the depth of the reference plane 10b in the normal direction. On the left side of FIG. 8B, the measurement result input from the depth sensor 133 indicates a depth smaller than a certain threshold, which means that an installation surface such as a table surface exists directly below the reference surface 10b. Therefore, the integrated control unit 160 can select the touch mode based on the measurement result. On the other hand, on the right side of FIG. 8B, the measurement result input from the depth sensor 133 shows a depth larger than a certain threshold value because the reference surface 10b is not in contact with any object. Therefore, the integrated control unit 160 may select the non-touch mode based on the measurement result. Table 2 shows the framework for selecting UI modes based on the depth of these projection planes.

According to the second example described above, for example, in the situation shown in FIG. 8B, when another object accidentally touches the reference surface 10b, the measurement result of the depth sensor 133 is smaller than the threshold value. It indicates the depth, and the integrated control unit 160 may mistakenly select the touch mode. However, when an unintended UI mode is selected, the user can easily cause the projection control device 100 to reselect the UI mode by moving the obstructed object. Therefore, the possibility of such misselection does not impair the usefulness of the above-mentioned method.

As a third example, the integrated control unit 160 determines the positional relationship of the projection control device 100 with respect to the projection surface based on both the first measurement result from the attitude sensor 131 and the second measurement result from the depth sensor 133. You may judge. FIG. 9 is an explanatory diagram for explaining a third example of switching the UI mode based on the positional relationship of the projector with respect to the projection surface. On the left side of FIG. 9, the first measurement result input from the attitude sensor 131 indicates that the reference surface 10b is horizontal and that the normal line of the reference surface 10b points in the direction of gravity. Therefore, the integrated control unit 160 can select the touch mode based on the measurement result. In the upper right of FIG. 9, the measurement result input from the attitude sensor 131 indicates that the reference plane 10b is perpendicular to the horizontal plane. Therefore, the integrated control unit 160 further acquires the second measurement result from the depth sensor 133 indicating the depth of the projection surface. Since the integrated control unit 160 indicates a depth smaller than a certain threshold value in the acquired second measurement result, the exploration surface of the projection control device 100 is sufficiently close to the projection surface, and the first user input detection is performed. It can be determined that the system 17 can achieve a sufficient level of touch detection accuracy. Therefore, the integrated control unit 160 may select the touch mode in this case. On the other hand, as shown in the lower right of FIG. 9, when the second measurement result acquired from the depth sensor 133 indicates a depth larger than the threshold value, the integrated control unit 160 is sufficient for the first user input detection system 17. It can be determined that the level touch detection accuracy is not achieved. Therefore, the integrated control unit 160 may select the non-touch mode in this case. Table 3 shows a framework for selecting UI modes based on both the orientation of the projector and the depth of the projection plane.

The UI mode selection framework shown in Table 3 is only an example. In another example, the integrated control unit 160 does not touch regardless of the orientation of the reference plane when the second measurement result from the depth sensor 133 indicating the depth of the projection plane indicates a depth larger than a certain threshold value. You may select the mode. Then, when the second measurement result indicates a depth smaller than the threshold value, the integrated control unit 160 acquires the first measurement result from the attitude sensor 131, and the normal of the reference plane faces the direction of gravity. The touch mode may be selected when the first measurement result indicates that the above is true, and the non-touch mode may be selected when the first measurement result does not. According to such an example, in a situation where there is a step between the surface on which the projection control device 100 is installed and the projection surface and the search surface is not formed in the vicinity of the projection surface, sufficient touch detection accuracy is obtained. It is possible to avoid the inconvenient situation that the touch mode is selected even though it is not achieved. In addition, even if the depth of the projection surface is small, there is an advantage that the non-touch mode is selected instead of the touch mode in the situation where the shadow of the finger becomes large and the operability of the touch input deteriorates when the image is projected on the wall surface. obtain. Table 4 shows a framework for selecting such UI modes, which is different from Table 3.

Up to this point, an example in which the mode can be switched between two UI modes has been described, but the integrated control unit 160 has a mode between three or more UI modes based on the positional relationship of the projection control device 100 with respect to the projection surface. May be switched. For example, there are three UI modes: "table mode" when the table surface is a projection surface, "wall mode" when the wall surface is a projection surface, and "ceiling mode" when the ceiling surface is a projection surface. You may. The table mode and the wall mode may correspond to the touch mode and the non-touch mode described above, respectively. Ceiling mode may also support non-touch mode. However, in the "ceiling mode", the rotation of the projected image based on the user input is enabled, whereas in the "wall mode", the rotation of the projected image can be invalidated. The integrated control unit 160, for example, when the measurement result input from the attitude sensor 131 indicates that the reference surface 10b is horizontal, but the normal line of the reference surface 10b points in the direction opposite to the direction of gravity. Such a ceiling mode may be selected.

[2-3. User input example]
As described above, the integrated control unit 160 uses the first user input detection system 17 as the main input interface in the touch mode. That is, the integrated control unit 160 activates the first user input detection system 17 in the touch mode. Then, when the touch input is detected by the first user input detection system 17, the integrated control unit 160 passes the touch event notified from the first user input detection system 17 to the executing software.

FIG. 10 is an explanatory diagram showing an example of user input that can be detected by the first user input detection system 17. Referring to FIG. 10, the projection control device 100 is placed on a table surrounded by three users. The projection control device 100 projects the projected image Im10 onto the table surface 3a. The projected image Im10 is an image of a touch input adaptive Web browser that operates on a mobile operating system. For example, the user Ua is to transition the page displayed by the Web browser into the desired linked Web pages, tap the position _{P 10} on the projection image IM10. Then, the first user input detection system 17 captures the image of the user Ua who tapped _{the position P 10} by the exploration light, and notifies the integrated control unit 160 of a touch event indicating _{that the position P 10 has been tapped.} The integrated control unit 160 passes the notified touch event to the running Web browser (via the operating system), and as a result, the Web page of the user Ua's desired link destination is displayed by the Web browser. In the touch mode, the first user input detection system 17 achieves sufficient touch detection accuracy, so that the user feels as if he / she is operating on the touch sensor of a smartphone or tablet PC through the projected image. You can freely interact with the running software.

In the non-touch mode, the integrated control unit 160 selects a second UI mode in which the second user input detection system 18 is used as the main input interface. Then, when some user input is detected by the second user input detection system 18, the integrated control unit 160 passes the user input event notified from the second user input detection system 18 to the executing software.

FIG. 11 is an explanatory diagram showing a first example of user input that can be detected by the second user input detection system 18. Referring to FIG. 11, the projection control device 100 is installed between the wall surface 3b and the user Ub. The projection control device 100 projects the projected image Im11 onto the wall surface 3b. The projected image Im11 is an image of a content viewer that is not adaptable to touch input. For example, user Ub plays gesture G1 that moves his hand to the right or left. Gesture G1 may be interpreted as a so-called swipe. Then, the recognition module 120 of the second user input detection system 18 recognizes the gesture G1 reflected in the captured image input from the camera 13, and notifies the integrated control unit 160 of the gesture input event. The integrated control unit 160 passes the notified gesture input event to the content viewer, and as a result, the display of the content in the projected image is scrolled in the direction corresponding to the gesture input.

In addition to (or instead of) swiping as described above, the recognition module 120 may recognize any kind of gesture, such as raising, waving, putting and releasing hands. In such a gesture interface, the second user input detection system 18 may be able to detect touch (or pointing), for example, based on changes in the position and area of the hand in the captured image. However, the detection accuracy is coarser than the level of accuracy that can be achieved by the first user input detection system 17. Even when the user's line of sight recognized from the captured image is used as the user input, touch (or pointing) is impossible, or only coarse touch detection accuracy is achieved, if not impossible.

FIG. 12 is an explanatory diagram showing a second example of user input that can be detected by the second user input detection system 18. Also in FIG. 12, the projection control device 100 is installed between the wall surface 3b and the user Ub. For example, the user Ub utters a voice command C1 for calling the calendar. Then, the recognition module 120 of the second user input detection system 18 recognizes the voice command C1 included in the voice input from the microphone 14, and notifies the integrated control unit 160 of the voice input event. The integrated control unit 160 causes the software module 140 to start the calendar application in response to the notified voice input event. As a result, the projected image Im12 displaying the calendar generated by the calendar application is projected onto the wall surface 3b. The calendar application is touch input non-adaptive application software, which can be manipulated by the user, for example, through gesture input or voice input. The recognition module 120 may recognize any kind of voice command. User interfaces that use voice commands are usually untouchable.

The second user input detection system 18 may include any kind of user input means, not limited to the examples of FIGS. 11 and 12. For example, a motion sensor that detects the presence of a human body within a range of several tens of centimeters may be included in the second user input detection system 18 as a user input means for detecting user input with two on / off values.

In FIGS. 11 and 12, images of the content viewer and the calendar application are shown as images of the touch input non-adaptive software. However, touch input non-adaptive software is not limited to such examples. For example, touch input non-adaptive software having various purposes such as clock, calendar or home energy management may be implemented, and any image of such software may be projected in non-touch mode.

When the touch mode is selected, the integrated control unit 160 uses the second user input detection system 18 as described with reference to FIGS. 11 and 12 as an input interface in addition to the first user input detection system 17. You can do it. On the other hand, when the non-touch mode is selected, the integrated control unit 160 may disable the first user input detection system 17 in order to prevent an erroneous touch operation intended by the user (for example, of the exploration light). Stop irradiation, etc.). Further, when the non-touch mode is selected, the integrated control unit 160 may use the first user input detection system 17 as an input interface in addition to the second user input detection system 18. In the non-touch mode, fine pointing through touch input can be difficult due to the large distance between the projection plane and the exploration plane. However, it is still possible to use the first user input detection system 17 as a simple user input means (for example, detecting two values of on / off) such as whether or not to block the exploration light by hand.

When the integrated control unit 160 determines that the projection surface is substantially horizontal, the integrated control unit 160 enables a user interface for rotating the projected image (hereinafter referred to as a rotation UI), and determines that the projection surface is substantially vertical. If so, the rotation UI may be disabled. The rotating UI may include one or more of GUI ((Graphical User Interface) and audio UI) displayed with the projected image.

FIG. 13 is an explanatory diagram for explaining an example of the rotation UI. Referring to the left side of FIG. 13, the projection control device 100 is placed on a table surrounded by two users. The projection control device 100 projects the projected image Im13a onto the table surface 3a. The projected image Im13a is an image of a touch input adaptive Web browser that operates on a mobile operating system. Buttons J1a, J1b, J1c and J1d are displayed at the four corners of the projected image Im13a. These buttons J1a, J1b, J1c and J1d are GUI items for rotating the projected image. For example, when the user Uc taps the button J1b, the first user input detection system 17 notifies the integrated control unit 160 of a touch event indicating that the position of the button J1b has been tapped. The integrated control unit 160 rotates the display of the Web browser by 90 ° clockwise in response to the notified touch event. As a result, as shown on the right side of FIG. 13, the projected image Im13b that displays the Web page in the direction suitable for viewing by the user Uc is projected.

When the rotating UI is a voice UI, the sound may be acoustic or a user's utterance. For example, the sound generated when the user hits the table may be used as a voice input for rotating the projected image. Further, the second user input detection system 18 may determine in which direction the projected image should be rotated from the arrival direction of the voice recognized by the array type microphone.

[2-4. Input conditions that do not depend on user input]
In the non-touch mode, only touch with coarse touch detection accuracy is possible, or touch input is not possible. Therefore, the integrated control unit 160 executes touch input non-adaptive software that changes the projected image based on input conditions that do not depend on user input on the software module 140 in order to enrich the user experience in the non-touch mode. You may let me. The input condition that does not depend on the user input here may be a condition related to at least one of, for example, the date, the time, the weather, the installation location of the projector, and the recognized user.

FIG. 14 is an explanatory diagram for explaining an example of a projected image that changes based on an input condition that does not depend on user input. Referring to FIG. 14, the projection control device 100 is installed in the second installation posture in front of the wall surface 3b. On the left side of FIG. 14, the clock indicates the time of day, and the weather is assumed to be sunny. The integrated control unit 160 can know the weather from, for example, the weather data acquired from the external server via the communication interface 150. Based on these input conditions, the integrated control unit 160 projects a projected image displaying a background expressing a sunny outdoor state in the daytime onto the wall surface 3b. On the right side of FIG. 14, the clock indicates the time of night and the weather is cloudy. Based on these input conditions indicated by the time and weather data, the integrated control unit 160 projects a projected image displaying a background expressing a cloudy outdoor state at night onto the wall surface 3b.

The recognition module 120 may identify the user viewing the projected image using known personal recognition techniques such as face recognition, fingerprint authentication or iris recognition. When a calendar or clock application is run in non-touch mode, they may superimpose pre-registered appointment information for the identified user on the calendar or clock image. Further, the recognition module 120 may recognize the installation location of the projector based on the position measurement result input from the GPS (Global Positioning System) sensor or the positioning sensor such as PlaceEngine (registered trademark). In non-touch mode, for example, if the projection control device 100 is located in the living room of the home, an image of a calendar or clock is projected, while if the projection control device 100 is located in the kitchen, the recommended dish. Recipe information may be projected.

<3. Process flow>
In this section, an example of the flow of processing that can be executed by the projection control device 100 described above will be described with reference to some flowcharts. Although a plurality of processing steps are described in the flowchart, the processing steps do not necessarily have to be executed in the order shown in the flowchart. Some processing steps may be performed in parallel. Further, additional processing steps may be adopted, and some processing steps may be omitted.

[3-1. UI control process]
FIG. 15 is a flowchart showing an example of the flow of the user interface control process according to the embodiment. Here, it is assumed that the projector module 110 of the projection control device 100 has already been started.

Referring to FIG. 15, first, the integrated control unit 160 executes a UI mode selection process in order to select a UI mode based on the positional relationship of the projection control device 100 with respect to the projection plane (step S110). Some more detailed examples of the UI mode selection process performed here will be described further later. Subsequent processing branches depending on whether or not the touch mode is selected in the UI mode selection processing (step S130).

When the touch mode is selected as the UI mode, the touch input adaptive software 141 is executed in the software module 140, and an image of the touch input adaptive software 141 is generated (step S140). The projector module 110 projects the generated image of the touch input adaptive software 141 onto the projection surface (step S140).

Next, the integrated control unit 160 determines whether or not the touch input is detected by the first user input detection system 17 of the projector module 110 (step S145). When the touch input is detected by the first user input detection system 17, the integrated control unit 160 passes the touch event notified from the first user input detection system 17 to the running touch input adaptive software 141 (). Step S150). As a result, the touch input adaptive software 141 can execute a process corresponding to the touch event (for example, read a Web page at the tip of the tapped link) and generate a new projected image. If no touch input is detected, the process can proceed to step S165.

When the non-touch mode is selected as the UI mode in step S110, the touch input non-adaptive software 143 is executed in the software module 140, and an image of the touch input non-adaptive software 143 is generated (step S155). The projector module 110 projects the generated image of the touch input non-adaptive software 143 onto the projection surface (step S160).

Next, the integrated control unit 160 determines whether or not any user input has been detected by the second user input detection system 18 of the projector module 110 (step S165). When some user input is detected by the second user input detection system 18, the integrated control unit 160 passes the user input event notified from the second user input detection system 18 to the executing software (step S170). .. As a result, the running software can execute processing according to the user input event and generate a new projected image.

Next, the integrated control unit 160 determines whether or not to change the UI mode (step S180). For example, if the measurement result input from the sensor module 130 indicates that the attitude of the projection control device 100 or the depth of the projection surface has changed, or if a user input instructing to change the UI mode is detected, integration is performed. The control unit 160 may determine that the UI mode is changed. When the integrated control unit 160 determines that the UI mode is changed, the user interface control process of FIG. 15 returns to step S110. When the integrated control unit 160 determines that the UI mode is not changed, the user interface control process of FIG. 15 returns to step S130, and the above-mentioned software image generation and projection and user input monitoring are repeated.

[3-2. UI mode selection process]
(1) First Example FIG. 16 is a flowchart showing a first example of a detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 16, first, the integrated control unit 160 acquires the measurement result regarding the positional relationship of the projection control device 100 with respect to the projection surface from the sensor module 130 (step S113). Next, the integrated control unit 160 determines whether or not the projection surface is positioned substantially horizontally below the projection control device 100 based on the acquired measurement result (step S115).

Then, when the integrated control unit 160 determines that the projection surface is located substantially horizontally below the projection control device 100, the integrated control unit 160 selects the touch mode as the UI mode (step S120). On the other hand, if this is not the case, the integrated control unit 160 selects the non-touch mode as the UI mode (step S128).

(2) Second Example FIG. 17A is a flowchart showing a second example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 17A, first, the integrated control unit 160 acquires the measurement result of the posture of the projection control device 100 from the posture sensor 131 (step S114). Next, the integrated control unit 160 determines whether or not the projection control device 100 is installed on the projection surface based on the acquired measurement result (step S116).

For example, when the measurement result from the attitude sensor 131 indicates that the reference surface 10b is horizontal, the integrated control unit 160 determines that the projection control device 100 is installed on the projection surface in the first installation posture. obtain. When the integrated control unit 160 determines that the projection control device 100 is installed on the projection surface in this way, the integrated control unit 160 selects the touch mode as the UI mode (step S120).

On the other hand, when the measurement result from the attitude sensor 131 indicates that the reference surface 10b is not horizontal, for example, the integrated control unit 160 further acquires the measurement result indicating the depth of the projection surface from the depth sensor 133 (step S122). Next, the integrated control unit 160 determines whether or not the distance between the search surface and the projection surface of the first user input detection system 17 (that is, the depth of the projection surface) is below the threshold value based on the acquired measurement result. Is determined (step S124). Then, when the integrated control unit 160 determines that the depth of the projection surface is below the threshold value, the integrated control unit 160 selects the touch mode as the UI mode (step S120). On the other hand, when the integrated control unit 160 determines that the depth of the projection surface does not fall below the threshold value, the integrated control unit 160 selects the non-touch mode (step S128).

(3) Third Example FIG. 17B is a flowchart showing a third example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 17B, first, the integrated control unit 160 acquires a measurement result indicating the depth of the projection surface from the depth sensor 133 (step S122). Next, the integrated control unit 160 determines whether or not the depth of the projection plane is below the threshold value based on the acquired measurement result (step S124). When the integrated control unit 160 determines that the depth of the projection surface does not fall below the threshold value, the integrated control unit 160 selects the non-touch mode (step S128).

On the other hand, when the integrated control unit 160 determines that the depth of the projection surface is below the threshold value, the integrated control unit 160 further acquires the measurement result of the attitude of the projection control device 100 from the attitude sensor 131 (step S125). Next, the integrated control unit 160 determines whether or not the reference surface 10b of the projection control device 100 is horizontal based on the measurement result from the attitude sensor 131 (step S126).

Then, when the measurement result from the attitude sensor 131 indicates that the reference surface 10b is horizontal, the integrated control unit 160 selects the touch mode as the UI mode (step S127). On the other hand, when the measurement result from the attitude sensor 131 indicates that the reference surface 10b is not horizontal, the integrated control unit 160 selects the non-touch mode as the UI mode (step S128).

(4) Fourth Example FIG. 18 is a flowchart showing a fourth example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 18, first, the integrated control unit 160 acquires a measurement result indicating the depth of the projection surface from the depth sensor 133 of the sensor module 130 (step S122). Next, the integrated control unit 160 determines whether or not the projection plane exists in the direction orthogonal to the search plane based on the acquired measurement result (step S123). When the projection plane does not exist in the direction orthogonal to the search plane, the integrated control unit 160 selects the non-touch mode as the UI mode (step S128).

On the other hand, when the projection plane exists in the direction orthogonal to the search plane, the integrated control unit 160 determines whether or not the depth of the projection plane is below the threshold value based on the measurement result acquired from the depth sensor 133 (step). S124). Then, when the integrated control unit 160 determines that the depth of the projection surface is below the threshold value, the integrated control unit 160 selects the touch mode as the UI mode (step S127). On the other hand, when the integrated control unit 160 determines that the depth of the projection surface does not fall below the threshold value, the integrated control unit 160 selects the non-touch mode (step S128).

(5) Fifth Example All of the first, second, third, and fourth examples described so far are based on the premise that the integrated control unit 160 autonomously selects the UI mode using the sensor data. It is supposed to be. However, when the user explicitly specifies the UI mode, the UI mode specified by the user may take precedence over the autonomously selected UI mode.

FIG. 19 is a flowchart showing a fifth example of the detailed flow of the UI mode selection process shown in FIG. Referring to FIG. 19, first, the integrated control unit 160 determines whether or not to autonomously select the UI mode (step S111). For example, the integrated control unit 160 may determine that the UI mode is autonomously selected when the user input specifying the UI mode is not detected.

When the integrated control unit 160 determines that the UI mode is autonomously selected, the integrated control unit 160 acquires the measurement result of the posture of the projection control device 100 from the posture sensor 131 of the sensor module 130 (step S114). Next, the integrated control unit 160 determines whether or not the projection control device 100 is installed on the projection surface based on the acquired measurement result (step S116). When the integrated control unit 160 determines that the projection control device 100 is installed on the projection surface, the integrated control unit 160 selects the touch mode as the UI mode (step S120). On the other hand, when the integrated control unit 160 determines that the projection control device 100 is not installed on the projection surface, the integrated control unit 160 selects the non-touch mode as the UI mode (step S128).

Further, when the integrated control unit 160 determines in step S111 that the UI mode is not autonomously selected, the integrated control unit 160 is based on the user input that can be detected via the first user input detection system 17 or the second user input detection system 18. , UI mode is selected (step S129). The UI mode selection based on such user input may be combined with any of the first to fourth examples of the UI mode selection process described above.

<4. Summary>
Up to this point, embodiments of the technique according to the present disclosure have been described in detail with reference to FIGS. 1 to 19. According to the above-described embodiment, when an image is projected on a projection surface by a projector, the UI mode related to the projected image is switched between two or more UI modes based on the positional relationship of the projector with respect to the projection surface. Be done. Therefore, it is possible to provide the user with a desirable user interface according to the positional relationship of the projector with respect to the projection surface. For example, when there is a high possibility that an obstacle other than the operator interferes with touch detection when the user tries to touch the projection surface, smooth user interaction is ensured by mainly using user input other than touch input. can do.

Further, according to a certain embodiment, when the projection surface is located substantially horizontally below the projector, the first user input detection system that detects the touch input to the projection surface can be used as the main input interface. For example, when a portable projector is placed on a horizontal surface such as a table surface and an image is projected on the table surface, such a positional relationship can be easily estimated using a posture sensor built in the projector. In this case, a mechanism for determining whether or not the projector and the projection surface are in a positional relationship suitable for touch input can be constructed at low cost by using a small attitude sensor.

Further, according to a certain embodiment, the projector includes a first user input detection system that detects a touch position by a user by using exploration light emitted toward an exploration region in the vicinity of the projector. In such a method using the exploration light, when the projector is in a specific positional relationship with respect to the projection surface, the exploration surface can be formed so as to cover the projection image in the vicinity of the projection image, and high touch detection accuracy can be achieved. Therefore, this method is suitable for the above-mentioned method of switching the UI mode based on the positional relationship of the projector with respect to the projection surface. For example, it is convenient to use the exploration light by mainly using the first user input detection system only when it is determined from the above positional relationship that the first user input detection system can achieve a sufficient level of touch detection accuracy. It is possible to avoid deterioration of the user experience due to a decrease in touch detection accuracy while making the best use of the characteristics.

Further, according to an embodiment, when the exploration surface formed in the method using the exploration light described above is substantially orthogonal to the installation surface of the projector (in this case, if the installation surface is horizontal, the exploration surface is vertical). And the projection plane is also vertical), further determination can be made based on the depth of the projection plane relative to the projector. Then, when the depth of the projection surface is larger than the threshold value, it is determined that the first user input detection system does not achieve a sufficient level of touch detection accuracy, and the second user input detection system replaces the first user input detection system. It can be mainly used. Therefore, for example, in an application in which a large screen-like content is projected onto the wall surface of a room and the user views the content, it is accurately determined that it is difficult for the user to touch the projection surface, and the application is used. A suitable user interface (eg, gesture UI, voice UI, line-of-sight UI, etc.) can be provided to the user.

Further, according to an embodiment, the projector may selectively execute touch input adaptive software and touch input non-adaptive software depending on the selected UI mode. Therefore, when it is determined that the first user input detection system can achieve a sufficient level of touch detection accuracy, execution of software designed to operate on a mobile device such as a smartphone or tablet PC is executed. While tolerating, it is possible to automatically run alternative software if this is not the case.

The various processes described in the present specification may be realized by using software, hardware, or a combination of software and hardware. The programs constituting the software are stored in advance in, for example, a storage medium (non-transitory media) provided inside or outside the device. Then, each program is read into RAM at the time of execution and executed by a processor such as a CPU.

Further, some of the logical functions of the device described in the present specification may be mounted on a device existing in the cloud computing environment instead of being mounted on the device. In that case, the information exchanged between the logical functions may be transmitted or received via the communication interface illustrated in FIG.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can come up with various modifications or modifications within the scope of the technical ideas described in the claims. Of course, it is understood that the above also belongs to the technical scope of the present disclosure.

Moreover, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A projection control unit that controls the projection of an image onto the projection surface by the projector,
A UI control unit that switches the mode of the user interface (UI) related to the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface.
A control device comprising.
(2)
The UI control unit uses a first user input detection system that detects a touch input to the projection surface as a main input interface when the projection surface is located substantially horizontally below the projector. The control device according to (1) above, which selects a mode.
(3)
The projector includes the first user input detection system for detecting a touch position by a user by using exploration light emitted toward an exploration region in the vicinity of the projector, according to the above (1) or (2). The control device described.
(4)
The UI control unit uses the first user input detection system as a main input interface when it is determined from the positional relationship that the first user input detection system can achieve a certain level of touch detection accuracy. The control device according to (3) above, which selects the UI mode of the above.
(5)
In the first installation posture of the projector, the exploration area forms an exploration surface substantially parallel to the installation surface of the projector.
The UI control unit determines that the first user input detection system can achieve the level of touch detection accuracy when the projector is installed on the projection surface in the first installation posture.
The control device according to (4) above.
(6)
The UI control unit determines the positional relationship of the projector with respect to the projection surface based on the first measurement result from the posture sensor of the projector, according to any one of (3) to (5). The control device described.
(7)
When the first measurement result from the attitude sensor indicates that the exploration surface formed by the exploration region is substantially horizontal, the UI control unit uses the first user input detection system as a main input interface. The control device according to (6) above, which selects the first UI mode to be used.
(8)
In the second installation posture of the projector, the exploration area forms an exploration surface substantially orthogonal to the installation surface of the projector.
In the UI control unit, when the projector is installed in the second installation posture and the depth of the projection surface is larger than the threshold value, the first user input detection system has the level of touch detection accuracy. Judge not to achieve,
The control device according to (5) above.
(9)
The control device according to (8), wherein the UI control unit acquires the depth of the projection surface based on the second measurement result from the depth sensor of the projector.
(10)
The projector further includes a second user input detection system, which has coarser touch detection accuracy than the first user input detection system or cannot detect touch.
The UI control unit uses the second user input detection system as the main input interface in the second UI mode in which the first user input detection system is not mainly used.
The control device according to any one of (2) to (9) above.
(11)
The control device according to (10) above, wherein the second user input detection system detects one or more of a user's gesture, voice, and line of sight as user input.
(12)
The projector further comprises a camera having an angle of view towards the direction in which the user is expected to be present when the second UI mode is selected.
The second user input detection system detects user input based on the image input from the camera.
The control device according to (11) above.
(13)
The projector projects the image of the touch input adaptive software or the touch input non-adaptive software that is selectively executed onto the projection surface.
The touch input adaptive software is executed in the first UI mode in which the first user input detection system is mainly used.
The control device according to any one of (2) to (12).
(14)
The control device according to (13) above, wherein the touch input adaptive software includes one or more of touch input adaptive basic software and touch input adaptive application software.
(15)
The control device according to (13) or (14) above, wherein the touch input non-adaptive software changes a projected image based on an input condition that does not depend on user input.
(16)
The control device according to (15) above, wherein the input condition independent of user input is related to at least one of a date, a time, a weather, an installation location of the projector, and a recognized user.
(17)
The item according to any one of (1) to (16) above, wherein the UI control unit activates the UI for rotating the projected image when it is determined that the projection surface is substantially horizontal. Control device.
(18)
The control device according to (17), wherein the UI for rotating the projected image includes one or more of a GUI and a voice UI displayed together with the image.
(19)
In a projection control device that controls the projection of an image onto a projection surface by a projector,
Switching the mode of the user interface (UI) associated with the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface.
Control methods including.
(20)
Projection controller processor,
A projection control unit that controls the projection of an image onto the projection surface by the projector,
A UI control unit that switches the mode of the user interface (UI) related to the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface.
A program to function as.

2a, 2b Projected image 3a, 3b Projection surface 11 Opening for projected light 12 Opening for exploration light 13 Camera 14 Microphone 15 Speaker 16 Image projection system 17 First user input detection system 18 Second user input detection system 21 Projected image 21a Projected light 22 Exploration surface 22a, 22b Exploration light 23 Exploration image 100 Projection control device 110 Projector module 120 Recognition module 130 Sensor module 131 Attitude sensor 133 Depth sensor 140 Software module 141 Touch input adaptive software 143 Touch input non-adaptive software 150 Communication Interface 160 Integrated control unit

Claims (20)

A projection control unit that controls the projection of an image onto the projection surface by the projector,
A UI control unit that switches the mode of the user interface (UI) related to the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface.
With
The projector has a first user input detection system that detects a touch position by a user by using exploration light emitted toward an exploration region in the vicinity of the projector.
The UI control unit is a control device that selects a first UI mode using the first user input detection system based on a measurement result of a sensor included in the projector. The UI control unit selects the first UI mode when it indicates from the first measurement result from the attitude sensor of the projector that the exploration surface formed by the exploration region is substantially horizontal. The control device according to claim 1. The UI control unit uses the first user input detection system when the depth from the projector to the projection surface is smaller than a predetermined value from the second measurement result from the depth sensor of the projector. The control device according to claim 1, wherein the UI mode of 1 is selected. From the second measurement result from the depth sensor of the projector, the UI control unit does not use the first user input detection system when the depth from the projector to the projection surface is larger than a predetermined value. The control device according to claim 1, wherein the UI mode of 2 is selected. The UI control unit uses the first user input detection system based on the first measurement result from the attitude sensor of the projector and the second measurement result from the depth sensor of the projector. The control device according to claim 1, wherein the mode is selected. The UI control unit uses a first user input detection system that detects a touch input to the projection surface as a main input interface when the projection surface is located substantially horizontally below the projector. The control device according to claim 1, wherein the UI mode is selected. The UI control unit is a first UI mode in which the first user input detection system is used as a main input interface when the search area is a search area formed so as to cover the projected image in the vicinity of the projected image. The control device according to claim 1, wherein the control device is selected. In the first installation posture of the projector, the exploration area forms an exploration surface substantially parallel to the installation surface of the projector.
The UI control unit determines that the first user input detection system can achieve touch detection accuracy equal to or higher than a predetermined threshold value when the projector is installed on the projection surface in the first installation posture. ,
The control device according to claim 7. In the second installation posture of the projector, the exploration area forms an exploration surface substantially orthogonal to the installation surface of the projector.
In the UI control unit, when the projector is installed in the second installation posture and the depth of the projection surface is larger than the threshold value, the first user input detection system is equal to or higher than the predetermined threshold value. Judge that the touch detection accuracy is not achieved,
The control device according to claim 8. The projector further includes a second user input detection system, which has coarser touch detection accuracy than the first user input detection system or cannot detect touch.
The UI control unit uses the second user input detection system as the main input interface in the second UI mode in which the first user input detection system is not mainly used.
The control device according to claim 6. The control device according to claim 10, wherein the second user input detection system detects one or more of a user's gesture, voice, and line of sight as user input. The projector further comprises a camera having an angle of view towards the direction in which the user is expected to be present when the second UI mode is selected.
The second user input detection system detects user input based on the image input from the camera.
The control device according to claim 11. The projector projects the image of the touch input adaptive software or the touch input non-adaptive software that is selectively executed onto the projection surface.
The touch input adaptive software is executed in the first UI mode in which the first user input detection system is mainly used.
The control device according to claim 6. The control device according to claim 13, wherein the touch input adaptive software includes one or more of a touch input adaptive basic software and a touch input adaptive application software. The control device according to claim 13, wherein the touch input non-adaptive software changes a projected image based on an input condition that does not depend on user input. The control device according to claim 15, wherein the input conditions that do not depend on user input relate to at least one of a date, a time, a weather, a location of the projector, and a recognized user. The control device according to claim 1, wherein the UI control unit activates a UI for rotating a projected image when it is determined that the projection surface is substantially horizontal. The control device according to claim 17, wherein the UI for rotating the projected image includes one or more of a GUI and a voice UI displayed together with the image. In a projection control device that controls the projection of an image onto a projection surface by a projector,
Switching the mode of the user interface (UI) associated with the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface.
Including
The projector has a first user input detection system that detects a touch position by a user by using exploration light emitted toward an exploration region in the vicinity of the projector.
A control method for selecting a first UI mode using the first user input detection system based on a measurement result of a sensor included in the projector. Projection controller processor,
A projection control unit that controls the projection of an image onto the projection surface by the projector,
A UI control unit that switches the mode of the user interface (UI) related to the projected image between two or more UI modes based on the positional relationship of the projector with respect to the projection surface.
To function as
The projector has a first user input detection system that detects a touch position by a user by using exploration light emitted toward an exploration region in the vicinity of the projector.
The UI control unit is a program for functioning to select a first UI mode using the first user input detection system based on the measurement result of the sensor included in the projector.

Images