WO2022141271A1

WO2022141271A1 - Control method and control device for platform system, platform system, and storage medium

Info

Publication number: WO2022141271A1
Application number: PCT/CN2020/141711
Authority: WO
Inventors: 杨俊超; 王振动; 王协平
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-12-30
Filing date: 2020-12-30
Publication date: 2022-07-07
Also published as: CN114556904A

Abstract

A control method (100) and control device (600) for a platform system, the platform system, and a storage medium. The platform system comprises a platform mechanism used for carrying a photographing device and adjusting the posture of the photographing device, the photographing device being used for photographing an environment. The control method (100) comprises: acquiring depth data (302) obtained by a depth sensor (301) observing the environment; according to the depth data (302), determining a target object closest to the photographing device in the environment; and controlling the platform mechanism to adjust the posture of the photographing device so as to adjust the target object to a preset position in a photographing screen of the photographing device. Therefore, the depth data (302) is acquired by means of the depth sensor (301) to more accurately determine the target object, thereby adjusting the photographing device by means of the platform mechanism, so as to adjust the target object to a preset position for subsequent focusing and photographing.

Description

Control method, control device, PTZ system and storage medium of PTZ system

technical field

The present application relates to the field of image capturing, and in particular, to a control method of a pan-tilt system, a control device, a pan-tilt system and a storage medium.

Background technique

In recent years, with the development of science and technology, imaging equipment has gradually become popular, and people like taking pictures and sharing videos. In order to avoid the time-consuming and laborious problem of learning photography skills, equipment to assist shooting, such as stabilizers, was invented. People can easily take pictures with stable images and smooth zooming in various scenes and various sports modes.

However, at present, most of the auxiliary shooting devices only support the auxiliary stabilization function in the photography-related functions, and cannot assist people in shooting, which greatly reduces the functions of the auxiliary shooting devices.

SUMMARY OF THE INVENTION

The present application provides a control method of a pan-tilt system, a control device, a pan-tilt system, and a storage medium, which are used for assisting shooting and enhancing auxiliary functions to improve shooting quality.

A first aspect of the present application is to provide a control method of a pan/tilt system, wherein the pan/tilt system includes a pan/tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and the photographing device For photographing an environment, the method includes: acquiring depth data obtained by a depth sensor observing the environment; determining a target object in the environment closest to the photographing device according to the depth data; controlling the cloud The stage mechanism adjusts the posture of the photographing device to adjust the target object to a preset position in the photographing picture of the photographing device.

A second aspect of the present invention is to provide a control device for a pan-tilt system, the pan-tilt system includes a pan-tilt mechanism for carrying a photographing device and adjusting the posture of the photographing device, and the photographing device is used to monitor the environment The control device further includes: a memory and a processor; the memory is used to store a computer program; the processor is used to call and execute the computer program to perform the following steps: acquiring the Depth data obtained by observing the environment; determining the target object in the environment closest to the photographing device according to the depth data; controlling the pan-tilt mechanism to adjust the posture of the photographing device to adjust the target object to a preset position in the photographing screen of the photographing device.

A third aspect of the present invention is to provide a pan/tilt system, including a pan/tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and the control device described in the second aspect.

A fourth aspect of the present invention is to provide a computer-readable storage medium, the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, and the program instructions are used in the first aspect. method described.

An embodiment of the present application provides a control method for a pan-tilt system, wherein the pan-tilt system includes a pan-tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and the photographing device is used for photographing an environment, and the method includes : Obtain the depth data observed by the depth sensor on the environment; determine the target object in the environment with the closest distance to the shooting device according to the depth data; control the pan-tilt mechanism to adjust the posture of the shooting device to adjust the target object to the position in the shooting screen of the shooting device Preset position. In this way, the depth data is acquired by the depth sensor to more accurately determine the target object, so that the camera is adjusted through the pan-tilt mechanism to adjust the target object to a preset position for subsequent focusing and shooting, so as to assist the user in the Shoot, enhance shooting functions, and improve shooting quality.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a schematic flowchart of a control method of a pan-tilt system according to an embodiment of the present invention;

2 is a schematic diagram of a point cloud image provided by an embodiment of the present invention;

3 is a specific schematic diagram of a control stabilizer provided by an embodiment of the present invention;

4 is a schematic diagram of an outline of gesture recognition provided by an embodiment of the present invention;

5 is a schematic structural diagram of a control device of a pan-tilt system according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a control device provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention.

In order to facilitate the understanding of the technical solutions and technical effects of the present application, the prior art is briefly described below:

As can be seen from the foregoing, since most of the current stabilizers only support the auxiliary stabilization function in the photography-related functions, the function of the stabilizer is greatly reduced.

In this embodiment of the present application, the depth data is acquired by the depth sensor to more accurately determine the target object, and the camera is adjusted through the pan-tilt mechanism to adjust the target object to a preset position for subsequent focusing and shooting, thereby assisting the user Shoot, enhance your shooting capabilities, and improve your shooting quality.

Some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments may be combined with each other without conflict between the embodiments.

1 is a schematic flowchart of a control method of a pan-tilt system provided by an embodiment of the present invention; the method 100 provided by an embodiment of the present application may be executed by a pan-tilt system, such as a handheld pan-tilt. The pan-tilt system includes a pan-tilt mechanism for carrying a photographing device and adjusting the posture of the photographing device, and the photographing device is used for photographing the environment, such as a camera, a camera, and the like. The method 100 includes the following steps:

101: Acquire depth data obtained by the depth sensor observing the environment.

102: Determine a target object in the environment that is closest to the photographing device according to the depth data.

103: Control the pan-tilt mechanism to adjust the posture of the photographing device to adjust the target object to a preset position in the photographing screen of the photographing device.

It should be noted that, the embodiments of the present application can also be implemented by a handheld stabilizer, a vehicle-mounted camera camera device stabilizer, an airborne camera camera device stabilizer, and the like, and these implementation devices may have a pan/tilt system.

The above steps are described in detail below:

101: Acquire depth data obtained by the depth sensor observing the environment.

Among them, the depth sensor is used to obtain depth data, which can be an active 3D depth sensor, which can autonomously emit the required visible light, invisible light and other electromagnetic waves or signals such as sound, and receive the signal reflected by the object to obtain the depth data. Depth sensors include binocular vision sensors, TOF (Time of Flight) sensors, millimeter-wave radar or lidar. The depth data may be data corresponding to a depth image or a point cloud image.

Among them, a photographing device, such as a camera, is fixedly installed with a depth sensor. For example, an active 3D depth sensor can be external and fixed on the camera hot shoe.

In addition, the depth sensor can also be built in the P (Pitch, which is rotated around the X axis) axis of the gimbal mechanism. At this time, the calibration needs to be calibrated and the relative position of the camera. The depth sensor can also be built in the gimbal R (Roll, which is rotated around the Z axis) axis. At this time, the relative position of the calibration and the camera and the relative angle of the camera and the camera need to be calculated in real time.

The depth sensor is built in the Y (Yaw, rotating around the Y axis) axis (or handle) of the gimbal. At this time, it is necessary to calibrate the relative position of the camera and the camera. The factory calibration can be done once. If the error between the center of the camera and the center of the gimbal mechanism is ignored , you only need to calibrate and the position of the center of the gimbal, you can do it once and for all.

Wherein, the pan-tilt mechanism includes a bearing base for installing the photographing device; the depth sensor is installed on the bearing base or the photographing device. The optical axis of the camera is parallel to the optical axis of the depth sensor.

The above-mentioned environment is the environment of the shooting screen of the shooting device, that is, the environment in the shooting screen.

For example, according to the above, before taking an image, the user can move the lens of the camera to move the person to be photographed to a position near the center of the camera's field of view, and then the active 3D depth sensor can first actively emit detection into the shooting environment After it actively transmits the detection signal, it records the time when the detection signal is reflected from the transmitted to the object in the shooting environment to calculate the distance between the reflection surface of the object and the active 3D depth sensor. (The movement of the object is not considered here, or the movement of the object is ignored. In theory, according to the Doppler effect, the speed of the moving object can also be calculated according to the change of the wavelength of the detection signal. The active 3D depth sensor actively transmits the detection signal, which is reflected by the surface of the object. To the sensitive device, the processor of the PTZ system can record the time interval between transmitting the detection signal and receiving the reflected signal, and then can calculate the depth information of the current pixel point according to the signal propagation speed, that is, the depth data (that is, the distance between the sensor and the active 3D depth sensor). In combination with the angle change of the transmitted detection signal (generally determined by the signal phase angle), three-dimensional depth information can be obtained, that is, depth data, that is, a point cloud image is obtained, which can also be called a depth image (that is, corresponding to each Depth information or depth data of a pixel). Among them, Figure 2 shows a point cloud image, that is, a depth image.

It should be noted that the embodiment of the present application can solve the problem that the reliability of image capturing of the device is affected under special weather or light conditions through the active 3D depth sensor; The gimbal is externally combined.

The target object may be an object in the environment that is closest to the photographing device and whose image area in the photographing picture is closest to the preset position in the photographing picture. The closest distance to the photographing device can also be determined by the closest distance to the depth sensor (the photographing device can be set in the same vertical direction as the depth sensor), for example, the angle of view of the camera corresponds to the person A who is the closest to the camera, and the person A is the closest to the depth sensor. . And person A is the closest to the preset position in the shooting screen.

Specifically, determining the target object closest to the photographing device in the environment according to the depth data includes: determining the distances between multiple objects in the environment and the depth sensor according to the depth data; Identify the target object.

Wherein, the target object is the object that is closest to the shooting device in the environment and whose image area in the shooting screen is closest to the preset position in the shooting screen; the target object is determined from the multiple objects according to the distances between the multiple objects and the depth sensor, including : Determine the positions of multiple objects in the shooting screen according to the installation position relationship between the depth sensor and the shooting device; determine the target from the multiple objects according to the distances between the multiple objects and the depth sensor and the positions of the multiple objects in the shooting screen object.

For example, according to the foregoing, as shown in FIG. 3 , the processor of the gimbal system obtains the depth image through the depth sensor 301 , that is, after the depth data 302 , the matrix convolution operation combined with various gradient operators can be used to easily obtain points Cloud image, that is, the contour information or contour data of each object in the depth image. According to the center point of the contour information, which may be the depth data corresponding to the center point of the rectangular frame circumscribing the contour information, the physical distance between the object corresponding to the contour information and the active 3D depth sensor is determined. Of course, other points in the contour information, such as the upper left corner, the lower right corner, etc., can also be selected to represent the contour information, so as to determine the distance to the active 3D depth sensor according to the depth data corresponding to the point. After the distance is determined, the object corresponding to the contour information closest to the active 3D depth sensor can be determined. When multiple objects are closest to the active 3D depth sensor (then it can be directly considered that the multiple objects are the closest to the camera, and the error can be ignored), then the installation between the active 3D depth sensor and the camera can be continued according to the The positional relationship can determine the mapping relationship between the coordinate system corresponding to the active 3D depth sensor, that is, the coordinate system where the depth data is located, and the camera coordinate system. Through this mapping relationship, the depth data can be mapped into the camera coordinate system, that is, the position in the shooting picture corresponding to the camera is determined. Therefore, after determining the objects closest to the active 3D depth sensor, the positions of the above-mentioned contour information center points of these objects in the shooting picture can be determined, and whether it is the closest to the preset position in the shooting picture can be determined by selecting The center point closest to the preset position in the shooting picture, so as to deduce the contour information corresponding to the center point and the corresponding target object.

It should be noted that reasonable use of the contour information in the point cloud image is sufficient to implement many functions including the above functions. Considering that the photographed people and the background will not be on the same plane, the focus deviation can be obtained according to the outline information in the point cloud image, without the need to accurately obtain the human shape or other shapes, which can reduce the processing time of the processor, save the code space and In consideration of reducing system power consumption, complex shape recognition is not performed for functions that do not require complex operations. In addition, the acquisition of contour information can also directly obtain the contour information of a specified shape by using a related method of machine learning.

Specifically, determining the target object from the plurality of objects according to the distances between the plurality of objects and the depth sensor includes: determining the object with the smallest distance from the depth sensor among the plurality of objects as the target object.

Since the similar content has been described above, it will not be repeated here, but only to explain: after determining the distances of multiple objects from the depth sensor, the object with the smallest distance from the depth sensor can be directly used as the target object, or the distance can be used as the target object. The smallest depth sensor is considered the smallest distance to the camera, and the object that is the smallest distance to the depth sensor is also the smallest object to the camera.

In order to be able to determine the target object more accurately, the distance between the multiple objects and the photographing device may also be determined based on the installation position relationship between the depth sensor and the photographing device, so as to select the target object.

Specifically, determining the target object from the multiple objects according to the distance between the multiple objects and the depth sensor includes: determining the multiple objects and the shooting device according to the distance between the multiple objects and the depth sensor and the installation position relationship between the depth sensor and the shooting device The distance between the devices; the object with the smallest distance from the photographing device among the multiple objects is determined as the target object.

Since the similar content has been described above, it will not be repeated here, but only to explain: the physical position relationship between the depth sensor and the photographing device can be determined according to the installation position relationship between the two, so that according to this physical position relationship, the The distances between the multiple objects and the depth sensor are converted into distances between the multiple objects and the photographing device, and the object with the smallest distance from the photographing device can be selected as the target object. For example, the active 3D depth sensor can be externally placed and fixed on the camera hot shoe, then according to this positional relationship and the distance between multiple objects and the active 3D depth sensor, conversion can be performed, such as through the triangle formula, to determine multiple The distance between the object and the camera.

Wherein, the preset position is the center of the shooting screen, and may also be other positions on the shooting screen. The preset position is determined by the user, and the user can set it through the photographing device or other display screens provided by the PTZ system, and the user can operate on the display screen or the screen of the photographing device to select the preset position. Thereby, the pan/tilt system determines the preset position in response to this operation.

Specifically, controlling the pan-tilt mechanism to adjust the posture of the photographing device includes: controlling the pan-tilt mechanism to adjust the posture of the photographing device according to the deviation between the position of the target object in the photographing screen and a preset position.

For example, according to the foregoing, the pan-tilt system, as shown in Figure 3, can be corrected according to the deviation obtained between the position of the target object's contour information center point in the shooting picture and the preset position, and the controller of the pan-tilt system is controlled stably device, such as the angle of the gimbal mechanism, so as to adjust the posture of the camera, so that the camera zooms after the posture is adjusted. The controller of the gimbal system can send control commands to the gimbal mechanism to adjust the angle and zoom the camera according to the deviation value corresponding to the deviation. For example, the controller can control the motor rotation of the gimbal mechanism according to the angle deviation, such as controlling the rotation of the P-axis motor, thereby rotating the Pitch axis and adjusting the position of the camera.

Correspondingly, it can also be based on the depth data of the target object (for example, the center point of its corresponding contour information), or the position in the point cloud image and the preset position in the depth data, or the position in the point cloud image (which can be determined by the above two methods. The mapping relationship of each coordinate system is reversed, such as mapping from the data in the camera coordinate system to the data in the depth sensor coordinate system, that is, the depth data in the depth sensor coordinate system) Control the PTZ mechanism to adjust the shooting The posture of the device will not be repeated here.

In order to make adjustments quickly, after obtaining the contour information, the contour information near the middle of the camera's field of view (for example, within a certain preset area in the middle of the field of view) can be retained, and the center position of the corresponding contour information in the shooting picture can be calculated. Location. Then, the deviation between the position and the preset position can be determined, so as to make adjustment, which will not be repeated here.

In addition to the above-mentioned adjustment of the target object to a preset position, such as automatic centering (auto center), the protruding contour at the nearest position of the center point can be automatically aligned. This function can be used to assist people who want to be in the center of the field of view when taking pictures or selfies, and quickly and accurately correct them to the center. The PTZ system can also realize the following functions:

Specifically, the method 100 further includes: determining, according to the depth data, an object occupying the largest image area in the shooting picture of the shooting device; The entirety of the subject is in the shooting frame.

For example, according to the method described above, the pan-tilt system can determine the position of each object in the shooting screen according to the depth data, or the image area of each object in the shooting screen, so as to determine the size of each image area, so as to control the pan-tilt The mechanism and/or the camera are adjusted so that the object with the largest image area is located at the preset position, which will not be repeated here.

Wherein, controlling the pan/tilt mechanism and/or the focal length of the photographing device to be located at a preset position in the photographing screen and making the entire object lie in the photographing frame includes: if the entire object is not in the photographing frame, controlling the focal length of the photographing device So that the whole of the object is located in the shooting screen; the pan-tilt mechanism is controlled to adjust the shooting posture of the shooting device so that the object is located in the preset position in the shooting screen.

For example, according to the foregoing, if the object with the largest image area is not completely in the shooting screen, you can adjust the focus by controlling the focal length of the camera, so that the image area of the object is in the shooting screen. The position of the object is preset by adjusting the camera posture by controlling the pan-tilt mechanism, which will not be repeated here.

In addition, the pan-tilt system uses the pan-tilt mechanism, so that after the photographing device adjusts the object with the largest image area to a preset position, it can compose a picture for the object. The gimbal system can calculate the contour information center of the object and the length and width of the circumscribed rectangle of the contour information. Finally, according to the preset composition ratio, such as the golden ratio, calculate the deviation of each axis of the gimbal mechanism and the deviation of the focal length, and adjust the gimbal mechanism. The angle and the focal length of the shooting device make the composition ratio the closest to the required ratio (the gimbal mechanism does not move, and the proportion of the outline itself is not appropriate. ). Thus, quick composition can be achieved.

Specifically, the method 100 further includes: determining the focus object in the picture of the photographing device; determining the distance of the focusing object according to the depth data; controlling the photographing device to focus according to the distance, so that the photographing device focuses on the focusing object.

Wherein, determining the focus object in the picture of the shooting device includes: identifying the focus object in the shooting picture. The focus object may be the target object described above, such as a person. Further, it can also be a characteristic part of a person. Such as human face, facial features, such as nose, eyes or mouth. The feature part of the person is the feature part of the target object.

For example, according to the above, after the gimbal system moves the target object to the preset position of the shooting screen through the gimbal mechanism, it can identify the focus object of the object, such as the nose, so as to focus on it, that is, the focus object. Then, the distance between the focal point and the camera is determined by the method described above, so as to focus.

It should be noted that when there is no focusing object, the preset position can be used as the focus to focus.

As a result, the gimbal system can enable close-up focus selection. The camera can be controlled to zoom according to the corresponding focal distance (which has been acquired by the depth sensor).

Specifically, the method 100 further includes: recognizing the specific gesture of the user according to the depth data; and controlling the pan-tilt system or the photographing device according to the recognized specific gesture.

The specific gesture refers to a preset gesture, such as a gesture of greeting with both hands, as shown in FIG. 4 .

For example, according to the foregoing, the user can enable the gesture mode through the application program of the PTZ system, and the PTZ system can first recognize the specific gesture, such as a gesture of greeting with both hands. According to the method described above, the PTZ system can obtain the outline information of the gesture, and then compare it with the outline information of the preset gesture to see if it is the same, or if the similarity exceeds the threshold, then it is determined to be a specific gesture, and the specific gesture can be enabled. Corresponding functions, such as waking up the tracking function of the PTZ system, or taking pictures, etc.

It should be noted that the PTZ system can set multiple preset gestures, and different preset gestures can correspond to different functions. After the PTZ system obtains the outline information of the user's gestures, it can be performed separately with multiple preset gestures. Compare and determine the corresponding preset gesture to wake up the function.

Specifically, the method 100 further includes: determining the position of the tracking object in the shooting screen of the shooting device according to the depth data; controlling the pan-tilt mechanism according to the position to adjust the posture of the shooting device so that the tracking object is located in the shooting screen.

The tracking object may be a person, a car, or the like.

For example, according to the foregoing description, the tracking function can be awakened according to the gesture, or the tracking function can be directly enabled to track the tracking object, such as taking a picture or a video. After starting tracking, the gimbal system can use the depth data to determine the position of the tracking object in the shooting screen, such as a preset position, in the way described above, so as to control the gimbal mechanism according to the position to adjust the camera posture for tracking shooting. .

It should be noted that the tracking object can be preset to be a person. Therefore, after obtaining the depth data, the tracking object can be determined according to the corresponding contour information, and the contour information can be compared with the contour information of the preset tracking object to determine. , and then trace. Wherein, the photographing or photographing of the tracking can be implemented in combination with the functions described above.

As for the follow-up tracking, whether to use the calculation results of the active 3D depth sensor or the results of the original non-active sensor, or to fuse the calculation results of the two, can be determined by the settings.

The embodiment of the present application proposes to use an active 3D depth sensor to process it to obtain a 3D point cloud image, that is, to combine the depth data with the control of a stabilizer and a photographing device, such as a camera device, to realize automatic centering, quick composition, close-up focus selection, gestures Recognition and other functions, combined with its active characteristics, increase the reliability of the target tracking function. In addition, this type of sensor can use simpler software methods to achieve better results than image processing.

5 is a schematic structural diagram of a control device of a pan-tilt system according to an embodiment of the present invention; the device 500 can be applied to a pan-tilt system, for example, a hand-held pan-tilt system. For a pan-tilt mechanism for adjusting the posture of the photographing device, the photographing device is used to photograph the environment, and the device 500 can execute the above-mentioned control method of the pan-tilt system. The apparatus 500 includes: an acquisition module 501 , a determination module 502 and a control module 503 . The functions of each module are described in detail below:

The acquiring module 501 is configured to acquire depth data obtained by the depth sensor observing the environment.

The determining module 502 is configured to determine, according to the depth data, a target object that is closest to the photographing device in the environment.

The control module 503 is configured to control the pan-tilt mechanism to adjust the posture of the photographing device to adjust the target object to a preset position in the photographing image of the photographing device.

Specifically, the determining module 502 includes: a first determining unit, configured to determine the distances between multiple objects in the environment and the depth sensor according to the depth data; The target object is determined in the object.

Wherein, the target object is the object that is closest to the shooting device in the environment and whose image area in the shooting screen is closest to the preset position in the shooting screen; the determining module 502 includes: a second determining unit, which is used for determining according to the depth sensor and the shooting device. The installation position relationship between the multiple objects determines the positions of the multiple objects in the shooting picture; the second determining unit is used for determining the target from the multiple objects according to the distances between the multiple objects and the depth sensor and the positions of the multiple objects in the shooting picture. object.

Specifically, the first determining unit is configured to: determine the object with the smallest distance from the depth sensor among the multiple objects as the target object.

Specifically, the first determining unit is configured to: determine the distance between the multiple objects and the shooting device according to the distances between the multiple objects and the depth sensor and the installation position relationship between the depth sensor and the shooting device; The object with the smallest distance between the photographing devices is determined as the target object.

Specifically, the control module 503 is configured to control the pan-tilt mechanism according to the deviation between the position of the target object in the shooting picture and the preset position to adjust the posture of the shooting device.

The target object is an object in the environment that is closest to the photographing device and whose image area in the photographing picture is closest to the preset position in the photographing picture.

The preset position is the center of the shooting screen.

The preset position is determined by the user.

Specifically, the photographing device is fixedly installed with the depth sensor.

Specifically, the pan-tilt mechanism includes a bearing base for installing the photographing device; the depth sensor is installed on the bearing base or the photographing device.

Among them, the depth sensor includes binocular vision sensor, TOF sensor, millimeter wave radar or lidar.

Wherein, the optical axis of the photographing device is parallel to the optical axis of the depth sensor.

In addition, the determining module 502 is further configured to: determine the object that occupies the largest image area in the shooting picture of the shooting device according to the depth data; the control module 503 is further configured to: control the pan-tilt mechanism and/or the focal length of the shooting device to make A preset position in the shooting screen and the entirety of the subject is located in the shooting screen.

Specifically, the control module 503 includes: a first control unit for controlling the focal length of the photographing device if the whole of the object is not in the shooting picture, so that the whole of the object is in the shooting picture; a second control unit for controlling the cloud The stage mechanism is used to adjust the shooting posture of the shooting device so that the object is located at a preset position in the shooting picture.

In addition, the determining module 502 is also used for: determining the focus object in the picture of the photographing device; the determining module 502 is also used for: determining the distance of the focusing object according to the depth data; the control module 503 is also used for: controlling the shooting device according to the distance Focus so that the camera focuses on the subject in focus.

Specifically, the determining module 502 is configured to: identify the focus object in the shooting picture.

In addition, the device 500 further includes: an identification module for identifying a specific gesture of the user according to the depth data; and a control module 503 for controlling the pan-tilt system or the photographing device according to the identified specific gesture.

In addition, the determination module 502 is also used to determine the position of the tracking object in the shooting picture of the shooting device according to the depth data; the control module 503 is also used to: control the pan-tilt mechanism according to the position to adjust the posture of the shooting device so that the tracking object is located in in the shooting screen.

In a possible design, the structure of the control device 500 of the pan-tilt system shown in FIG. 5 may be implemented as an electronic device, and the electronic device may be a control device of the pan-tilt system. As shown in FIG. 6 , the control device 600 of the PTZ system may include: one or more processors 601 and one or more memories 602 . The pan/tilt system includes a pan/tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and the photographing device is used for photographing the environment. The memory 602 is used to store a program that supports the electronic device to execute the control method of the pan-tilt system provided in the embodiments shown in FIG. 1 to FIG. 4 . The processor 601 is configured to execute programs stored in the memory 602 . Specifically, the program includes one or more computer instructions, wherein the one or more computer instructions can implement the following steps when executed by the processor 601:

Run the computer program stored in the memory 602 to achieve: obtain the depth data obtained by the depth sensor on the environment; determine the target object in the environment with the closest distance to the photographing device according to the depth data; control the pan-tilt mechanism to adjust the posture of the photographing device to move the target The subject is adjusted to a preset position in the photographing frame of the photographing device.

Specifically, the processor 601 is specifically configured to: determine the distances between multiple objects in the environment and the depth sensor according to the depth data; and determine the target object from the multiple objects according to the distances between the multiple objects and the depth sensor.

Wherein, the target object is the object that is closest to the shooting device in the environment and whose image area in the shooting screen is closest to the preset position in the shooting screen; the processor 601 is specifically used for: according to the installation position between the depth sensor and the shooting device The relationship determines the positions of the multiple objects in the shooting picture; the target object is determined from the multiple objects according to the distances of the depth sensors of the multiple objects and the positions of the multiple objects in the shooting picture.

Specifically, the processor 601 is specifically configured to: determine the object with the smallest distance from the depth sensor among the multiple objects as the target object.

Specifically, the processor 601 is specifically configured to: determine the distance between the multiple objects and the photographing device according to the distances between the multiple objects and the depth sensor and the installation position relationship between the depth sensor and the photographing device; The object with the smallest distance between the photographing devices is determined as the target object.

Specifically, the processor 601 is specifically configured to: control the pan-tilt mechanism according to the deviation between the position of the target object in the shooting picture and the preset position to adjust the posture of the shooting device.

The preset position is the center of the shooting screen.

The preset position is determined by the user.

In addition, the processor 601 is further configured to: determine the object that occupies the largest image area in the photographing picture of the photographing device according to the depth data; control the pan-tilt mechanism and/or the focal length of the photographing device to be located at a preset position in the photographing picture and The entirety of the subject is in the shooting frame.

Specifically, the processor 601 is specifically configured to: if the whole of the object is not in the shooting screen, control the focal length of the shooting device so that the whole of the object is located in the shooting screen; control the pan-tilt mechanism to adjust the shooting posture of the shooting device to make the object at the preset position in the shooting screen.

In addition, the processor 601 is further configured to: determine the focusing object in the picture of the photographing device; determine the distance of the focusing object according to the depth data; and control the focusing of the photographing device according to the distance, so that the photographing device focuses on the focusing object.

Specifically, the processor 601 is specifically configured to: identify the focus object in the shooting picture.

In addition, the processor 601 is further configured to: recognize the specific gesture of the user according to the depth data; and control the pan-tilt system or the photographing device according to the recognized specific gesture.

In addition, the processor 601 is further configured to: determine the position of the tracking object in the shooting picture of the shooting device according to the depth data; and control the pan-tilt mechanism according to the position to adjust the posture of the shooting device so that the tracking object is located in the shooting picture.

In addition, an embodiment of the present invention provides a computer-readable storage medium, where the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, and the program instructions are used to implement the above-mentioned methods of FIG. 1 to FIG. 4 . .

A pan-tilt system provided by an embodiment of the present invention includes a pan-tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and a control device as shown in FIG. 6 .

The technical solutions and technical features in the above embodiments can be used alone or combined in the case of conflict with the present invention, as long as they do not exceed the cognitive scope of those skilled in the art, they all belong to the equivalent embodiments within the protection scope of the present application .

In the several embodiments provided by the present invention, it should be understood that the disclosed related detection apparatus (eg, IMU) and method may be implemented in other manners. For example, the embodiments of the remote control device described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components. May be combined or may be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of the remote control device or unit, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

The above descriptions are only the embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied to other related technologies Fields are similarly included in the scope of patent protection of the present invention.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims

A control method of a pan/tilt system, wherein the pan/tilt system includes a pan/tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and the photographing device is used to photograph the environment, wherein That is, the method includes:

acquiring depth data obtained by the depth sensor observing the environment;

Determine the target object closest to the photographing device in the environment according to the depth data;

The pan-tilt mechanism is controlled to adjust the posture of the photographing device to adjust the target object to a preset position in the photographing screen of the photographing device.
The method according to claim 1, wherein the determining, according to the depth data, the target object that is closest to the photographing device in the environment comprises:

determining distances of a plurality of objects in the environment from the depth sensor according to the depth data;

The target object is determined from the plurality of objects according to distances of the plurality of objects from the depth sensor.
The method according to claim 2, wherein the target object is an object in the environment that is closest to the photographing device and whose image area in the photographing picture is closest to a preset position in the photographing picture;

The determining the target object from the plurality of objects according to the distances between the plurality of objects and the depth sensor includes:

Determine the positions of the plurality of objects in the shooting picture according to the installation position relationship between the depth sensor and the shooting device;

A target object is determined from the plurality of objects according to the distances of the plurality of objects to the depth sensor and the positions of the plurality of objects in the photographing picture.
The method according to claim 2, wherein the determining the target object from the plurality of objects according to the distances between the plurality of objects and the depth sensor comprises:

The object with the smallest distance from the depth sensor among the plurality of objects is determined as the target object.
The method according to claim 2, wherein the determining the target object from the plurality of objects according to the distances between the plurality of objects and the depth sensor comprises:

determining the distances between the plurality of objects and the photographing device according to the distances of the plurality of objects and the depth sensor and the installation position relationship between the depth sensor and the photographing device;

An object with the smallest distance from the photographing device among the plurality of objects is determined as the target object.
The method according to any one of claims 1-5, wherein the controlling the pan/tilt mechanism to adjust the posture of the photographing device comprises:

The PTZ mechanism is controlled according to the deviation between the position of the target object in the shooting picture and the preset position to adjust the posture of the shooting device.
The method according to any one of claims 1-6, wherein the target object is the closest to the photographing device in the environment and whose image area in the photographing picture is closest to the preset in the photographing picture The object to set the location to.
The method according to any one of claims 1-7, wherein the preset position is the center of the shooting picture.
The method according to any one of claims 1-8, wherein the preset position is determined by a user.
The method according to any one of claims 1-9, wherein the photographing device is fixedly installed with the depth sensor.
The method according to any one of claims 1-10, wherein the pan/tilt mechanism comprises a bearing base for installing the photographing device;

The depth sensor is mounted on the carrying base or the photographing device.
The method according to any one of claims 1-11, wherein the depth sensor comprises a binocular vision sensor, a TOF sensor, a millimeter-wave radar or a lidar.
The method according to claim 1, wherein the optical axis of the photographing device is parallel to the optical axis of the depth sensor.
The method according to any one of claims 1-13, wherein the method further comprises:

Determine the object that occupies the largest image area in the shooting picture of the shooting device according to the depth data;

The focal length of the pan/tilt mechanism and/or the photographing device is controlled so that the preset position in the photographing picture and the entirety of the object are located in the photographing picture.
The method according to claim 14, characterized in that the control of the pan/tilt mechanism and/or the focal length of the photographing device is performed so that the object is located at a preset position in the photographing screen and the whole of the object is completely Located in the shooting screen, including:

If the whole of the object is not in the shooting picture, controlling the focal length of the shooting device so that the whole of the object is located in the shooting picture;

The pan-tilt mechanism is controlled to adjust the shooting posture of the shooting device so that the object is located at a preset position in the shooting picture.
The method according to any one of claims 1-15, wherein the method further comprises:

Determine the focus object in the picture of the photographing device;

determining the distance of the focused object according to the depth data;

According to the distance, the photographing device is controlled to focus, so that the photographing device focuses on the focusing object.
The method according to claim 16, wherein the determining the focus object in the picture of the photographing device comprises:

Identifying a focus object in the shooting picture.
The method according to any one of claims 1-17, wherein the method further comprises:

Identify the user's specific gesture based on the depth data;

The pan-tilt system or the photographing device is controlled according to the recognized specific gesture.
The method according to any one of claims 1-18, wherein the method further comprises:

Determine the position of the tracking object in the shooting picture of the shooting device according to the depth data;

The pan-tilt mechanism is controlled according to the position to adjust the posture of the photographing device so that the tracking object is located in the photographing screen.
A control device of a pan-tilt system, characterized in that the pan-tilt system comprises a pan-tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and the photographing device is used to photograph the environment; The control device further includes: a memory and a processor;

the memory for storing computer programs;

The processor is used to call and execute the computer program to perform the following steps:

acquiring depth data obtained by the depth sensor observing the environment;

Determine the target object closest to the photographing device in the environment according to the depth data;

The pan-tilt mechanism is controlled to adjust the posture of the photographing device to adjust the target object to a preset position in the photographing screen of the photographing device.
The control device according to claim 20, wherein the processor is specifically configured to:

determining distances of a plurality of objects in the environment from the depth sensor according to the depth data;

The target object is determined from the plurality of objects according to distances of the plurality of objects from the depth sensor.
The control device according to claim 21, wherein the target object is an object in the environment that is closest to the photographing device and whose image area in the photographing picture is closest to a preset position in the photographing picture ;

The processor is specifically used for:

Determine the positions of the plurality of objects in the shooting picture according to the installation position relationship between the depth sensor and the shooting device;

A target object is determined from the plurality of objects according to the distances of the plurality of object depth sensors and the positions of the plurality of objects in the photographing picture.
The control device according to claim 21, wherein the processor is specifically configured to:

The object with the smallest distance from the depth sensor among the plurality of objects is determined as the target object.
The control device according to claim 21, wherein the processor is specifically configured to:

determining the distances between the plurality of objects and the photographing device according to the distances of the plurality of objects and the depth sensor and the installation position relationship between the depth sensor and the photographing device;

An object with the smallest distance from the photographing device among the plurality of objects is determined as the target object.
The control device according to any one of claims 20-24, wherein the processor is specifically configured to:

The PTZ mechanism is controlled according to the deviation between the position of the target object in the shooting picture and the preset position to adjust the posture of the shooting device.
The control device according to any one of claims 20-25, wherein the target object is the closest to the photographing device in the environment and whose image area in the photographing picture is closest to the photographing picture Objects in preset positions.
The control device according to any one of claims 20-26, wherein the preset position is the center of the shooting picture.
The control device according to any one of claims 20-27, wherein the preset position is determined by a user.
The control device according to any one of claims 20-28, wherein the photographing device is fixedly installed with the depth sensor.
The control device according to any one of claims 20-29, wherein the pan-tilt mechanism comprises a bearing base for installing the photographing device;

The depth sensor is mounted on the carrying base or the photographing device.
The control device according to any one of claims 20-30, wherein the depth sensor comprises a binocular vision sensor, a TOF sensor, a millimeter-wave radar or a lidar.
The control device according to claim 20, wherein the optical axis of the photographing device is parallel to the optical axis of the depth sensor.
The control device according to any one of claims 20-32, wherein the processor is further configured to:

Determine the object that occupies the largest image area in the shooting picture of the shooting device according to the depth data;

The focal length of the pan/tilt mechanism and/or the photographing device is controlled so that the preset position in the photographing picture and the entirety of the object are located in the photographing picture.
The control device according to claim 33, wherein the processor is specifically configured to:

If the whole of the object is not in the shooting picture, controlling the focal length of the shooting device so that the whole of the object is located in the shooting picture;

The pan-tilt mechanism is controlled to adjust the shooting posture of the shooting device so that the object is located at a preset position in the shooting picture.
The control device according to any one of claims 20-34, wherein the processor is further configured to:

Determine the focus object in the picture of the photographing device;

determining the distance of the focused object according to the depth data;

According to the distance, the photographing device is controlled to focus, so that the photographing device focuses on the focusing object.
The control device according to claim 35, wherein the processor is specifically configured to: identify the focus object in the shooting picture.
The control device according to any one of claims 20-36, wherein the processor is further configured to:

Identify the user's specific gesture based on the depth data;

According to the identified specific gesture, the pan-tilt system or the photographing device is controlled.
The control device according to any one of claims 20-37, wherein the processor is further configured to:

Determine the position of the tracking object in the shooting picture of the shooting device according to the depth data;

The pan-tilt mechanism is controlled according to the position to adjust the posture of the photographing device so that the tracking object is located in the photographing screen.
A pan-tilt system is characterized by comprising a pan-tilt mechanism for carrying a photographing device and for adjusting the posture of the photographing device, and a control device according to any one of claims 20-38.
A computer-readable storage medium, characterized in that the storage medium is a computer-readable storage medium, and program instructions are stored in the computer-readable storage medium, and the program instructions are used to implement any one of claims 1-19 The control method of the PTZ system described in item.