JP2021131891A - Terminal device - Google Patents

Abstract

To enable an input position shift resulting from a display image shift between a terminal device and a display device to be easily corrected in accordance with a function of the display device.SOLUTION: The terminal device is a smartphone, for example, and displays a first image on a display and outputs a second image partly including the first image to a display device such as a navigation device. In addition, the terminal device acquires calibration information indicating relative positional information on the first image with respect to the second image from an external server. When a user makes a touch input on the second image displayed on the display device, its input position is transmitted from the display device to the terminal device. The terminal device acquires conversion necessity/unnecessity information indicating whether or not the input position on the second image needs to be converted on the basis of the calibration information, and in a case where conversion is necessary, converts the input position on the second image into an input position on the first image on the basis of the calibration information.SELECTED DRAWING: Figure 6

Description

The present invention relates to the calibration of touch inputs of interconnected terminal devices and display devices.

In recent years, in-vehicle devices such as car navigation devices and terminal devices such as smartphones have been connected and used.

For example, Patent Document 1 describes a system in which a mobile terminal and an in-vehicle device are linked to operate an application executed on the mobile terminal from the in-vehicle device.

In Non-Patent Document 1, by incorporating a dedicated module into an application installed on a smartphone, a display image of the smartphone is transmitted to the in-vehicle device and displayed on the display unit of the in-vehicle device, and a touch on the display unit of the in-vehicle device is provided. We are proposing a technology to operate smartphone applications by input.

When the display image of the smartphone is transmitted to the display device and displayed as in Non-Patent Document 1, the number of pixels and the aspect ratio of the display image do not always match between the smartphone and the display device. Therefore, depending on the smartphone model, black frame areas may be added to the top, bottom, left, and right of the output image in order to secure the aspect ratio of the output image. That is, the image displayed on the touch panel of the display device is an image in which black frame areas are added to the top, bottom, left, and right of the display image of the smartphone.

In this case, since the area of the black frame is in the display image, the display device cannot determine whether or not the area of the black frame is displayed. Therefore, when the black frame part is touched on the touch panel of the display device, it is determined that the area in the displayed image is touched on the smartphone side even though the area is an area without buttons etc. as a smartphone. There is a risk that it will end up. Further, since the area of the image displayed on the touch panel of the display device does not match the actual touch area, the touch coordinates by the user may shift even in the area inside the black frame area.

Therefore, Patent Document 2 describes a method of preparing calibration information for each combination of a smartphone and a display device and converting the user's input coordinates on the display device into the input coordinates on the smartphone based on the calibration information. ..

JP-A-2010-130670 International Publication WO 2013/150637 Gazette

AppRadio Pioneer Electronics USA http://www.pioneerelectronics.com/PUSA/Car/AppRadio

However, some display devices mounted on the vehicle have a function uniquely corresponding to the display of the black frame area as described above. A display device having such a function may not require the coordinate conversion process as in Patent Document 2. Further, when the display device independently executes the processing for the black frame area, and further performing the processing as in Patent Document 2 on the smartphone side, there may be a problem that the coordinates recognized by the smartphone side are shifted. ..

Examples of the problems to be solved by the present invention include the above. An object of the present invention is to make it possible to easily correct the deviation of the input position due to the deviation of the display image between the terminal device and the display device according to the function of the display device.

The invention according to claim 1 is a terminal device, which includes a display unit for displaying a first image and an image output unit for outputting a second image including the first image to the display device. The input position acquisition unit that acquires the input position on the second image indicating the position of the touch input performed by the user with respect to the second image displayed on the display device from the display device, and the second The relative position information acquisition unit that acquires the relative position information indicating the relative positional relationship of the first image with respect to the image from an external server, and the input position on the second image are converted based on the relative position information. The conversion necessity information acquisition unit that acquires conversion necessity information indicating whether or not conversion is necessary from the display device, and the conversion necessity information acquisition unit that indicates that conversion is necessary It is characterized by including a conversion unit that converts an input position on the second image into an input position on the first image based on the relative position information when acquired.

The invention according to claim 4 is a calibration method executed by a terminal device including a display unit, which comprises a display step of displaying a first image on the display unit and a part of the first image. An image output step of outputting the second image including the second image to the display device, and an input position on the second image indicating the position of the touch input performed by the user with respect to the second image displayed on the display device. , The input position acquisition step of acquiring from the display device, the relative position information acquisition step of acquiring the relative position information indicating the relative positional relationship of the first image with respect to the second image from an external server, and the first. Conversion necessity information acquisition step of acquiring conversion necessity information indicating whether or not it is necessary to convert the input position on the image of 2 based on the relative position information from the display device, and conversion are required. When the conversion necessity information acquisition step acquires the conversion necessity information indicating that, the input position on the second image is converted into the input position on the first image based on the relative position information. It is characterized by having a conversion step to be performed.

The invention according to claim 5 is a calibration program executed by a terminal device including a display unit and a computer, wherein a display step of displaying a first image on the display unit and the first image are combined. An image output step of outputting the second image included in the unit to the display device, and an input on the second image indicating the position of the touch input performed by the user with respect to the second image displayed on the display device. An input position acquisition step of acquiring a position from the display device, a relative position information acquisition step of acquiring relative position information indicating a relative positional relationship of the first image with respect to the second image from an external server, and The conversion necessity information acquisition step of acquiring the conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information from the display device, and the conversion are necessary. When the conversion necessity information acquisition step acquires the conversion necessity information indicating that, the input position on the second image is changed to the input position on the first image based on the relative position information. It is characterized in that the computer is made to execute the conversion step of converting to.

The configuration of the system according to the embodiment of the present invention is shown. It is a block diagram which shows the functional structure of the system when the display device which does not have an image size adjustment function is used. An example of an image in which a black frame is added to adjust the aspect ratio is shown. An example of the input position conversion method is shown. It is a block diagram which shows the functional structure of the system when the display device which has an image size adjustment function is used. It is a flowchart of the conversion process of an input position. An example of a calibration image is shown.

In a preferred embodiment of the present invention, the terminal device includes a display unit that displays a first image, an image output unit that outputs a second image including the first image to the display device, and the above. An input position acquisition unit that acquires an input position on the second image indicating the position of the touch input performed by the user with respect to the second image displayed on the display device from the display device, and the second image. It is necessary to convert the relative position information acquisition unit that acquires the relative position information indicating the relative positional relationship of the first image with respect to the image from an external server and the input position on the second image based on the relative position information. The conversion necessity information acquisition unit that acquires the conversion necessity information indicating whether or not there is, and the conversion necessity information acquisition unit that indicates that conversion is necessary are acquired from the display device. In this case, it includes a conversion unit that converts the input position on the second image into the input position on the first image based on the relative position information.

The terminal device is, for example, a smartphone or the like, and displays a first image on a display unit and outputs a second image including the first image to a display device such as a navigation device. Further, the terminal device acquires relative position information indicating the relative positional relationship of the first image with respect to the second image from an external server. When the user makes a touch input to the second image displayed on the display device, the input position is transmitted from the display device to the terminal device. The terminal device acquires conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information, and if conversion is necessary, is based on the relative position information. Then, the input position on the second image is converted into the input position on the first image. As a result, the specifications of the display unit of the terminal device and the display unit of the display device are different, and when coordinate conversion is required on the terminal device side, the input position for the display device can be correctly converted to the input position for the terminal device. Become. On the other hand, when the display device has a coordinate conversion function and the coordinate conversion on the terminal device side is unnecessary, the coordinate conversion is not performed, so that the input position is shifted by executing the unnecessary coordinate conversion. Can be prevented.

In one aspect of the terminal device, the relative position information acquisition unit stores the relative position information acquired from the outside in a storage unit in the terminal device, and the conversion unit stores the relative position information acquired in the past. If it is stored in, the relative position information stored in the storage unit is used. This eliminates the need to acquire relative location information from an external server each time, and can reduce the amount of communication.

Another aspect of the terminal device is a calibration image output unit that supplies a calibration image to the display device, and a calibration input that indicates the position of a user's touch input with respect to the calibration image displayed on the display device. A calibration input position acquisition unit that acquires a position from the display device, a relative position information generation unit that generates relative position information based on the acquired calibration input position, and a generated relative position information to an external server. It includes a communication unit for transmitting.

In this aspect, when the relative position information cannot be obtained from an external server, the relative position information can be created based on the user's input to the calibration image. In addition, by uploading the created relative location information to the server, it becomes possible to share it with a large number of users.

In another preferred embodiment of the present invention, the calibration method performed by the terminal device including the display unit includes a display step of displaying the first image on the display unit and a part of the first image. An image output step of outputting the second image including the second image to the display device, and an input position on the second image indicating the position of the touch input performed by the user with respect to the second image displayed on the display device. , The input position acquisition step of acquiring from the display device, the relative position information acquisition step of acquiring the relative position information indicating the relative positional relationship of the first image with respect to the second image from an external server, and the first. Conversion necessity information acquisition step of acquiring conversion necessity information indicating whether or not it is necessary to convert the input position on the image of 2 based on the relative position information from the display device, and conversion are required. When the conversion necessity information acquisition step acquires the conversion necessity information indicating that, the input position on the second image is converted into the input position on the first image based on the relative position information. It has a conversion step to be performed. As a result, when the specifications of the display unit of the terminal device and the display unit of the display device are different and the coordinate conversion is required on the terminal device side, the input position for the display device can be correctly converted to the input position for the terminal device. It becomes.

In another preferred embodiment of the present invention, the calibration program executed by the terminal device including the display unit and the computer includes a display step of displaying the first image on the display unit and the first image. An image output step of outputting the second image included in the unit to the display device, and an input on the second image indicating the position of the touch input performed by the user with respect to the second image displayed on the display device. An input position acquisition step of acquiring a position from the display device, a relative position information acquisition step of acquiring relative position information indicating a relative positional relationship of the first image with respect to the second image from an external server, and The conversion necessity information acquisition step of acquiring the conversion necessity information indicating whether or not the input position on the second image needs to be converted based on the relative position information from the display device, and the conversion are necessary. When the conversion necessity information acquisition step acquires the conversion necessity information indicating that, the input position on the second image is changed to the input position on the first image based on the relative position information. The computer is made to execute the conversion step of converting to. As a result, when the specifications of the display unit of the terminal device and the display unit of the display device are different and the coordinate conversion is required on the terminal device side, the input position for the display device can be correctly converted to the input position for the terminal device. It becomes.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[Basic configuration]
FIG. 1 shows a configuration of a system including a terminal device according to an embodiment of the present invention. As shown in FIG. 1, this system includes a server 1, a smartphone 10, and a display device 20.

The smartphone 10 and the display device 20 are configured to be able to communicate with each other. The smartphone 10 corresponds to the terminal device of the present invention. The display device 20 is a device mounted on a vehicle, and examples thereof include a navigation device and an in-vehicle AV device.

In this system, the display image of the smartphone 10 is transmitted to the display device 20 and displayed on the display device 20. When the user inputs an operation to the display device 20, the display device 20 transmits a signal corresponding to the operation input to the smartphone 10. As a result, the user can operate the smartphone 10 by inputting an operation to the display device 20.

The smartphone 10 displays an image on a touch panel (display unit) 14 using a liquid crystal display or the like. The smartphone 10 transmits the image D1 displayed on the touch panel 14 to the display device 20 by, for example, HDMI (High Definition Multimedia Interface) or the like.

The display device 20 includes a touch panel (display unit) 21 using a liquid crystal display or the like. The display device 20 receives the image D1 from the smartphone 10 and displays it on the touch panel 21. Further, the display device 20 detects the operation input performed by the user on the touch panel 21, and transmits the input position (position coordinate) D2 to the smartphone 10. The input position D2 is transmitted to the smartphone 10 by, for example, SPP (Serial Port Profile) of Bluetooth (registered trademark).

The server 1 includes a database 2. As will be described later, the database 2 stores relative position information D3 (hereinafter, also referred to as calibration information D3). The smartphone 10 can access the server 1 by a wireless communication function or the like and download the calibration information.

[When the display device does not have the image size adjustment function]
FIG. 2 is a block diagram showing a functional configuration of the system shown in FIG. 1 when a display device 20A having no image size adjustment function is used. In the following description, when referring to a display device having no image size adjustment function, it is referred to as "display device 20A", and when referring to a display device having an image size adjustment function, it is referred to as "display device 20B". When referring to a display device without being limited to, it is simply referred to as "display device 20".

The image size adjustment function is, for example, a function in which a user arbitrarily cuts out a part of an image displayed on a display device, and the cut out image is stretched (enlarged) and displayed on the entire screen. .. In the example of FIG. 2, since the display device 20A does not have the image size adjustment function, the coordinate conversion described below is performed on the smartphone 10 side, and the image size displayed on the smartphone 10 and the image size displayed on the display device 20A are performed. To be consistent with.

The smartphone 10 includes an application 11, an image output unit 12, a control unit 13, a touch panel 14, and communication units 15 and 16.

The application 11 is an application executed on the smartphone 10, and in this embodiment, the application 11 is for operating the smartphone 10 by the display device 20 as described above. The application 11 is actually realized by executing a program prepared in advance by a computer such as a CPU. The application 11 functions as an input position acquisition unit, a calibration information acquisition unit, a calibration image output unit, a calibration position information acquisition unit, a conversion necessity information acquisition unit, and a calibration information generation unit of the present invention. ..

The image output unit 12 receives the display image of the smartphone 10 from the application 11 and transmits it to the display device 20 as the image D1.

The control unit 13 controls the entire smartphone 10, and is actually realized by the OS and basic applications mounted on the smartphone 10. The control unit 13 displays the image D12 generated by various applications operating on the smartphone 10 on the touch panel 14 and supplies the image D12 to the application 11.

The communication unit 15 has a communication function for wirelessly communicating with the server 1 and receiving the calibration information D3 from the server 1. The communication unit 16 has a communication function for receiving the input position D2 corresponding to the user's operation input from the display device 20.

On the other hand, the display device 20A includes a touch panel 21, a communication unit 22, an input unit 23, and a control unit 25. The touch panel 21 is used when the user performs a desired operation input. The input unit 23 detects the operation input performed by the user on the touch panel 21, and sends the input position D2 to the communication unit 22. The communication unit 22 transmits the input position D2 to the smartphone 10.

The control unit 25 controls the entire display device 20A. Further, the control unit 25 includes a memory 26 inside. The memory 26 stores information indicating whether or not the display device itself has an image size adjustment function. Information on whether or not the image size adjustment function is provided can be set as, for example, a flag (hereinafter, also referred to as an “adjustment function flag”). In one example, a display device having an image size adjustment function stores "adjustment function flag = 1" in its memory 26. On the other hand, in the display device having no image size adjustment function, "adjustment function flag = 0" is stored in the memory 26. In the example of FIG. 2, since the display device 20A does not have the image size adjustment function, "adjustment function flag = 0" is stored in the memory 2.

When the display device 20A is connected to the smartphone 10, the control unit 25 notifies the smartphone 10 of the necessity of coordinate conversion on the smartphone 10 side based on the value of the adjustment function flag stored in the memory 26. do. In the example of FIG. 2, the display device 20A does not have an image size adjustment function, and the smartphone 10 is asked to perform coordinate conversion for absorbing the difference between the image size of the smartphone 10 and the image size of the display device 20. Since it is necessary, the control unit 25 notifies the smartphone 10 that the coordinate conversion is necessary on the smartphone 10 side (“coordinate conversion: necessary”).

Next, the processing performed by the application 11 will be described. The application 11 receives the image D12 generated by various applications executed by the smartphone 10 from the control unit 13 and sends the image D12 to the image output unit 12. The image output unit 12 transmits the image D12 as the image D1 to the display device 20A. The display device 20A displays the received image D1 on the touch panel 21. As a result, the display image of the smartphone 10 is displayed on the touch panel 21 of the display device 20A.

The user looks at the display image of the display device 20A and inputs an operation to the touch panel 21 of the display device 20A. Specifically, the user inputs an operation by touching a button in the image displayed on the touch panel 21. This operation input is detected by the input unit 23, and the corresponding input position D2 is transmitted to the smartphone 10 via the communication unit 22.

In the smartphone 10, the communication unit 16 receives the input position D2 and supplies it to the application 11. The application 11 acquires the input position of the user's operation input to the smartphone 10 and supplies the input position to the control unit 13. The control unit 13 executes a process corresponding to the operation input performed by the user based on the display image D12 and the input position received from the application 11. In this way, the smartphone 10 can be operated from the display device 20A.

(Input position conversion method)
When the image of the smartphone 10 is transmitted to the display device 20A and displayed in this way, the specifications of the touch panel 14 of the smartphone 10 and the touch panel 21 of the display device 20, specifically, the number of pixels and the aspect ratio match. Not always. Therefore, when transmitting an image to the display device 20A, the smartphone 10 may add black frames to the top, bottom, left, and right of the image in order to adjust the number of pixels, the aspect ratio, and the like. For example, in the above-mentioned HDMI standard, the aspect ratio of the image to be transmitted is fixed at 16: 9 or 4: 3, but the aspect ratio of the touch panel 14 of the smartphone 10 varies depending on the model. Therefore, when the smartphone 10 outputs an image by HDMI, the image output unit 12 may add a different black frame for each model to adjust the aspect ratio.

FIG. 3 shows an example of an image in which a black frame is added to adjust the aspect ratio. The image IM is displayed on the touch panel 14 of the smartphone 10. This image IM includes a list of songs belonging to an album and a "BACK" button to return to the previous screen. When this image IM is transmitted to the display device 20A and displayed, the smartphone 10 transmits the image IMx whose aspect ratio is adjusted so as to conform to the HDMI standard to the display device 20A as the image D2. The image IMx displayed on the touch panel 21 of the display device 20A has a black frame area BK added to the outer periphery of the image IM displayed on the smartphone 10.

In this case, since the black frame area BK is also displayed in the touch panel 21, the user can perform touch input to the black frame area BK as well. However, since the black frame area BK does not exist in the image IM of the smartphone 10 and is not an operation input area, the touch input for the black frame area BK must be processed as an inappropriate input. Further, since the scale of the image IM displayed on the smartphone 10 and the image IMx displayed on the display device 20A are different, the position coordinates of the touch input to the area inside the black frame area BK are also the image of the smartphone 10. It is necessary to convert to the position coordinates on the IM.

Here, when the display device 20 has an image size adjustment function, the input position D2 transmitted from the display device 20 already matches the image size of the smartphone 10, so the position coordinates on the smartphone 10 side. There is no need to perform the conversion process. On the other hand, when the display device 20A does not have the image size adjustment function as in the example of FIG. 2, the input position D2 transmitted from the display device 20A corresponds to the image size of the display device 20A. , The image size of the smartphone 10 is not consistent. Therefore, in the present embodiment, when the display device 20A does not have the image size adjustment function and the display device 20A receives the notification of "coordinate conversion: necessary" as the coordinate conversion necessity information, the application 11 of the smartphone 10 Functions as a conversion unit 11x, and it is decided to perform this conversion process.

Specifically, the conversion unit 11x receives the input position D2 on the touch panel 21 (that is, the input position on the image IMx) from the display device 20A, and receives the input position on the touch panel 14 of the smartphone 10 (that is, the input position on the image IM). ). At this time, the conversion unit 11x mainly performs the following two processes.

First, when the touch input to the black frame area BK is a simple touch (that is, an input of a type that touches almost one point and then releases the touch position without moving it), the conversion unit 11x simplifies the touch input. Ignore it. On the other hand, when the touch input to the black frame area BK is an input involving movement of the touch position such as so-called dragging, the conversion unit 11x sets the input position of the touch input to the black frame area BK to the black frame area BK closest to that position. Convert to the position of the inner area (hereinafter referred to as "regular area"). By any of these methods, the conversion unit 11x processes the touch input performed on the black frame area BK.

Second, the conversion unit 11x converts the input position of the touch input with respect to the regular area to the corresponding position on the touch panel 14 of the smartphone 10. As a result, even if the number of pixels of the touch panel 14 of the smartphone 10 and the touch panel 21 of the display device 20A are different, the input position of the display device 20A with respect to the touch panel 21 can be correctly converted to the input position of the smartphone 10 with respect to the touch panel 14.

FIG. 4 shows an example of the above conversion method. First, consider the lateral direction. The number of dots in the horizontal direction of the touch panel 21 of the display device 20A is set to "W", and the left and right widths of the black frame area BK displayed on the touch panel 21 are set to "Wa" and "Wb", respectively. Further, the number of dots in the horizontal direction of the touch panel 14 of the smartphone 10 is defined as "Ws".

In this case, the formula for converting the input position "X" on the touch panel 21 of the display device 20A to the input position "xs" on the touch panel 14 of the smartphone 10 is as follows.

xs = 0: When X <Wa xs = Ws * (X-Wa) / {W- (Wa + Wb)}: When Wa <X <Wb xs = Ws: X> (W-Wb) Is the same. The number of dots in the vertical direction of the touch panel 21 of the display device 20A is set to "H", the vertical widths of the black frame area BK displayed on the touch panel 21 are set to "Hb" and "Ha", respectively, and the vertical direction of the touch panel 14 of the smartphone 10 is set. Let the number of dots of be "Hs".

In this case, the formula for converting the input position "Y" on the touch panel 21 of the display device 20A to the input position "ys" on the touch panel 14 of the smartphone 10 is as follows.

ys = 0: When Y <Ha ys = Hs * (Y-Ha) / {H- (Ha + Hb)}: When Ha <Y <Hb ys = Hs: Y> (H-Hb) Black frame When the touch input to the area BK is a simple touch, the conversion unit 11x may ignore the touch input as described above.

According to the above conversion method, even when the image IMx including the black frame area BK is displayed on the touch panel 21 of the display device 20A, the input position for the touch panel 21 can be correctly converted to the input position for the touch panel 14 of the smartphone 10. ..

By the way, in order to execute the above conversion, the number of dots on the touch panel 14 of the smartphone 10 (the number of dots in the vertical direction: Hs, the number of dots in the horizontal direction: Ws) and the number of dots on the touch panel 21 of the display device 20A (vertical). The number of dots in the direction: H, the number of dots in the horizontal direction: W) and the widths (Wa, Wb, Ha, Hb) of the black frame area BK are required. The application 11 can acquire the number of dots of the touch panel 14 of the smartphone 10 from the control unit 13 of the smartphone 10 or the like, and can acquire the number of dots of the touch panel 21 of the display device 20A from the display device 20A. On the other hand, the widths of the black frame area BK on the top, bottom, left, and right are not the information normally possessed by the smartphone 10 and the display device 20.

Therefore, the application 11 acquires the widths of the black frame area BK from the server 1 as the calibration information D3. Specifically, the width of the black frame area BK is determined based on the specifications of the touch panel 14 of the smartphone 10 and the specifications of the touch panel 21 of the display device. Therefore, the server 1 stores the widths of the black frame area BK in advance as calibration information D3 in association with the combination of the smartphone 10 and the display device 20A. For example, the server 1 associates the ID indicating the model name of the smartphone 10 and the ID indicating the model name of the display device 20A with the calibration information D3 indicating the width of the black frame area BK. Remember. When the smartphone 10 is connected to the display device 20A, the smartphone 10 acquires the ID of the display device 20A, transmits the ID of the smartphone 10 and the ID of the display device 20A to the server 1, and performs calibration information corresponding to the combination thereof. Acquire D3 from server 1. Then, the smartphone 10 uses the acquired calibration information D3 to convert the input position of the display device 20A with respect to the touch panel 21 to the input position of the smartphone 10 with respect to the touch panel 14.

[When the display device has an image size adjustment function]
FIG. 5 is a block diagram showing a functional configuration of the system shown in FIG. 1 when a display device 20B having an image size adjustment function is used. The image size adjustment function is, for example, a function in which a user arbitrarily cuts out a part of an image displayed on a display device, and the cut out image is stretched (enlarged) and displayed on the entire screen. As can be understood by comparing with FIG. 2, the display device 20B in FIG. 5 has a conversion unit 27, converts the input coordinates output from the input unit 23, and supplies the input coordinates as the input coordinates D2 to the communication unit 22. It is different from the display device 20A shown in FIG. 2, but is the same as the display device 20A shown in FIG. 2 except for the above. Further, in FIG. 5, the server 1 and the smartphone 10 are the same as those in FIG.

As described above, when transmitting an image to the display device 20B, the smartphone 10 may add black frames on the top, bottom, left, and right of the image in order to adjust the number of pixels, the aspect ratio, and the like. In this case, since the black frame added by the smartphone 10 is a part of the image data, it is displayed as it is on the display unit 21 of the display device 20B. Therefore, the user uses the image size adjustment function of the display device 20B to stretch (enlarge) the image D1 transmitted from the smartphone 10 so that the black frame is not displayed (by cutting out the image area other than the black frame). )indicate. As a result, the black frame added by the smartphone 10 is not displayed on the display unit 21 of the display device 20B.

Further, the display device 20B grasps how much the user has enlarged the display image by the image size adjustment function, and the conversion unit 27 displays the user's input position in the enlarged display image before the enlargement. It is converted into the coordinates of the input position in the image (that is, the input position corresponding to the image size of the smartphone 10). As a result, even when the user enlarges the display image to an arbitrary size, the input position (input coordinates) of the user's touch input can be correctly converted to the input position in the image size of the smartphone 10. The communication unit 22 transmits the input position D2 after being converted by the conversion unit 27 so as to match the image size of the smartphone 10 to the smartphone 10. This eliminates the need for coordinate conversion on the smartphone 10 side.

Since the display device 20B shown in FIG. 5 has an image size adjustment function, the “adjustment function flag = 1” is stored in the memory 26 of the control unit 25. Therefore, the control unit 25 notifies the smartphone 10 of the coordinate conversion necessity information (“coordinate conversion: unnecessary”) indicating that the coordinate conversion is not necessary on the smartphone 10 side. As a result, the coordinate conversion is not performed on the smartphone 10 side, so that the display device 20B side transmits the input position D2 that correctly matches the image size of the smartphone 10, but the smartphone 10 side further performs unnecessary coordinate conversion. It can be prevented from being stowed.

Further, the display device 20B may acquire the calibration information D3 from the smartphone 10 by using, for example, the communication unit 22, and the conversion unit 27 may perform the above-mentioned conversion process using the acquired calibration information D3. .. With such a configuration, for example, even if the image area in which a black frame is partially left is cut out and enlarged by the user image size adjustment function, appropriate coordinate conversion between the display device 20B and the smartphone 10 can be performed. It can be carried out.

[Input position conversion process]
FIG. 6 is a flowchart of the input position conversion process. This process is executed by the application 11 in the smartphone 10. As a premise, it is assumed that the smartphone 10 and the display device 20 are in a state where they can communicate with each other. Further, this process is executed when the application 11 is started on the smartphone 10.

First, when the application 11 is started, the application 11 requests the display device 20 for the coordinate conversion necessity information, and receives the coordinate conversion necessity information from the control unit 25 of the display device 20 (step S9). The coordinate conversion necessity information is determined based on whether or not the display device 20 has an image size adjusting function as described above. When the coordinate conversion necessity information received from the display device 20 is "coordinate conversion: unnecessary" (step S10: No), the coordinate conversion on the smartphone 10 side is unnecessary, so the application 11 ends the process.

On the other hand, when the coordinate conversion necessity information received from the display device 20 is "coordinate conversion: necessary" (step S10: Yes), the application 11 determines whether or not it is the first activation (step S11). ). This determination is made based on, for example, the processing history information stored in the application 11.

In the case of the first startup, usually, the calibration information does not yet exist in the application 11. Therefore, the application 11 specifies the ID of the smartphone 10 and the ID of the display device 20, requests the server 1 for the calibration information corresponding to the combination thereof, acquires the corresponding calibration information, and stores the storage unit in the smartphone 10. And so on (step S12). Then, the process proceeds to step S14.

On the other hand, if it is not the first activation (step S11; No), the application 11 determines whether or not there is calibration information in the storage unit of the smartphone 10 (step S13). The calibration information in this case means the calibration information corresponding to the combination of the smartphone 10 currently used and the display device 20. When there is no calibration information (step S13; No), the application 11 acquires the calibration information from the server 1 (step S12). On the other hand, if there is calibration information (step S13; Yes), the process proceeds to step S14.

In step S14, the application 11 determines whether or not the acquisition of the calibration information is successful. That is, the application 11 determines whether or not the calibration information has been prepared by the process of step S12 or S13. If the application 11 succeeds in acquiring the calibration information (step S14: Yes), the process proceeds to step S15.

On the other hand, when the application 11 does not succeed in acquiring the calibration information (step S14; No), the application 11 performs a process of creating the calibration information by itself using the display device 20. Specifically, the application 11 transmits a calibration image to the display device 20 and displays the calibration image on the touch panel 21 of the display device 20 (step S16). The reason why the acquisition of the calibration information is not successful is that the calibration information corresponding to the combination of the smartphone 10 and the display device 20 does not exist in the server 1 or the communication with the server 1 cannot be performed normally. And so on.

An example of the calibration image is shown in FIG. In the example of FIG. 7, the application 11 transmits an image expressing the normal area in light color or the like as a calibration image to the display device 20 so as to be distinguishable from the black frame area BK. In this case, the calibration image includes the black frame area BK based on the difference in the number of pixels, the aspect ratio, and the like between the touch panel 14 of the smartphone 10 and the touch panel of the display device 20. Further, the application 11 indicates the positions of the four corners inside the black frame area BK by the marks C1 to C4, and causes the user to touch the four corners inside the black frame area BK. When the user touches the positions of the marks C1 to C4, the display device 20 acquires the coordinates of each position and transmits the coordinates to the smartphone 10.

The application 11 acquires the position coordinates corresponding to the four corners inside the black frame area BK, calculates the width of the black frame area BK up, down, left and right, generates the calibration information, and saves it in the storage unit of the smartphone 10. (Step S17). Further, the application 11 uploads the generated calibration information to the server 1 together with the ID of the smartphone and the ID of the display device (step S18). As a result, the server 1 is provided with calibration information corresponding to the combination of the smartphone and the display device, and can be used by another user thereafter. Then, the process proceeds to step S15.

In step S15, the application 11 converts the input position using the prepared calibration information (step S15). That is, the input position of the display device 20 with respect to the touch panel 21 is acquired from the display device 20 and converted into the input position of the smartphone 10 with respect to the touch panel 14. Then, the input position D13 obtained by the conversion is supplied to the control unit 13 of the smartphone 10 to end the conversion process. The control unit 13 of the smartphone 10 processes the operation input by the user based on the input position D13.

In this way, even when the specifications such as the number of pixels and the aspect ratio are different between the touch panel 14 of the smartphone 10 and the touch panel 21 of the display device 20 and the black frame area BK is displayed in the display image of the display device 20, the display device 20 The input position with respect to the touch panel 21 can be correctly converted to the input position with respect to the touch panel 14 of the smartphone 10.

Since the application 11 uses the calibration information used for converting the input position when it is stored in the smartphone 10, there is no need to download it from the server 1. Further, when the calibration information cannot be obtained from the server 1, the application 11 generates the calibration information based on the input of the user by using the calibration image illustrated in FIG. 7, so that the server 1 can be used. Conversion processing is possible even when using a new model smartphone or display device that does not have information. The work of creating the calibration information using the calibration image is not limited to the case where the calibration information is not available as described above. For example, the calibration information creation process may be registered in the menu of the smartphone 10 or the like so that it can be executed when the user desires.

Further, since the application 11 uploads the calibration information thus created to the server 1, other users can also use the calibration information thereafter. When the server 1 receives the calibration information from a plurality of smartphones, the server 1 analyzes the accuracy of the calibration information and performs an averaging process to hold the calibration information with higher accuracy. be able to.

[Modification example]
In the above embodiment, the control unit 25 of the display device 20 transmits the coordinate conversion necessity information of “coordinate conversion: unnecessary” to the smartphone 10 when the display device 20 has the image size adjustment function, and the display device 20 If does not have the image size adjustment function, the coordinate conversion necessity information of "coordinate conversion: necessary" is transmitted to the smartphone 10.

However, even when the display device 20 has the image size adjustment function, the user does not actually use the image size adjustment function (for example, the image size adjustment function is set to off), and the smartphone 10 transmits the image. When the image D1 is displayed on the display unit 21 as it is, the control unit 25 transmits the coordinate conversion necessity information of "coordinate conversion: necessary" to the smartphone 10 and performs the coordinate conversion on the smartphone 10 side. It may be carried out.

Further, for example, when the image size adjustment function on the display device 20 side is capable of not only enlarging the entire image but also enlarging only in the vertical direction and enlarging only in the horizontal direction, the control unit 25 requires coordinate conversion. As information, the smartphone 10 may be notified of the necessity of coordinate conversion in each of the vertical direction and the horizontal direction. For example, when the user adjusts the size only in the vertical direction by using the image size adjustment function of the display device 20 and does not adjust the size in the horizontal direction, the control unit 25 determines based on the setting. Coordinate conversion necessity information such as "vertical coordinate conversion: unnecessary, horizontal coordinate conversion: necessary" may be transmitted to the smartphone 10.

1 Server 10 Smartphone 11 Application 11x, 24 Conversion unit 12 Image output unit 13 Control unit 14, 21 Touch panel 20 Display device

Claims (1)

An image output unit that outputs a second image including the first image to a display device, and an image output unit.
A relative position information acquisition unit that acquires relative position information indicating the relative positional relationship of the first image with respect to the second image, and
A conversion information acquisition unit that acquires conversion information indicating whether or not to convert an input position on the second image based on the relative position information from the display device.
A conversion unit that converts an input position on the second image into an input position on the first image based on the relative position information when the conversion information indicates that conversion is necessary.
A terminal device comprising.

Images