CN118509569B - Naked eye three-dimensional display method and system based on intelligent screen - Google Patents

Abstract

The invention relates to the technical field of display, and discloses a naked eye three-dimensional display method and a system based on an intelligent screen, wherein the method comprises the following steps: positioning pupil center coordinates in an image coordinate system, and obtaining pupil fixation point positions through a geometric transformation algorithm; inputting the nose point fixation point position, the mouth fixation point position and the pupil fixation point position into a first machine learning model trained in advance to obtain the head gesture of a user; inputting the watching distance, the illumination intensity, the head gesture and the mouth fixation point position into a pre-trained second machine learning model to obtain the optimal adjustment parameters of the parallax barrier array; dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen; the invention can ensure that the user can experience the optimal advertising 3D effect under different visual angles.

Description

Naked eye three-dimensional display method and system based on intelligent screen

Technical Field

The invention relates to the technical field of display, in particular to a naked eye three-dimensional display method and system based on an intelligent screen.

Background

With the development of information technology and display technology, the three-dimensional display technology is widely applied in the fields of advertisement, entertainment, education and the like; the traditional 3D display technology mainly depends on wearing special glasses to realize stereoscopic vision effect, but the mode has a plurality of inconveniences such as wearing discomfort, high equipment cost and the like; to solve these problems, naked eye 3D display technology has been developed; the naked eye 3D display technology does not need a user to wear glasses, can directly realize a stereoscopic vision effect through a display screen, and greatly improves user experience and viewing convenience; in public places, especially communities, advertisement information dissemination is an important component of community services; the community advertisement intelligent screen is used as a novel advertisement medium, and various information can be accurately transmitted to community residents in real time through digital and intelligent means; however, most of the existing community advertisement smart screens adopt flat panel display, so that the effect and the attraction of information display are limited, and the advertisement display efficacy is difficult to fully develop.

At present, most of the existing advertisement intelligent screen display systems adopt flat display for displaying, although partial 3D display modes exist, for example, chinese patent with bulletin number of CN115933217A discloses a method and a system for realizing naked eye 3D advertisement on an armrest display terminal, and the invention realizes 3D display of advertisements by integrating a grating film group on a display screen; however, research and practical application of the above method and the prior art have found that the above method and the prior art have at least the following partial drawbacks:

(1) The head gesture of the watching user and the pupil viewpoint of the eyes on the intelligent screen cannot be tracked, so that the abnormal display phenomena such as ghost, distortion or blurring can be seen by the watching user along with the azimuth change of the watching user easily;

(2) The optimal adjustment parameters of the parallax barrier array cannot be determined in real time on the basis of acquiring the head gesture and the pupil viewpoint, the distance and the angle of the parallax barrier array cannot be adjusted in real time, further, the fact that the display content of the intelligent screen is 3D images cannot be ensured to be watched by a user at different angles cannot be ensured, and then the playing display effect of the intelligent screen advertisement and the watching experience of the user on the intelligent screen advertisement are difficult to improve.

Disclosure of Invention

In order to overcome the defects in the prior art, the embodiment of the invention provides a naked eye three-dimensional stereoscopic display method and system based on an intelligent screen.

In order to achieve the above purpose, the present invention provides the following technical solutions:

the utility model provides a three-dimensional stereoscopic display method of bore hole based on wisdom screen, be provided with parallax barrier array and camera module on the wisdom screen, the camera module includes 2 at least infrared cameras, the method includes:

Processing the face image captured by the infrared camera to locate the pupil center coordinate in the image coordinate system, and converting the pupil center coordinate in the image coordinate system by a geometric transformation algorithm to obtain the pupil fixation point position on the intelligent screen; the pupil fixation point positions comprise a left pupil fixation point position and a right pupil fixation point position;

acquiring a nose point gazing point position and a mouth gazing point position, and inputting the nose point gazing point position, the mouth gazing point position and the pupil gazing point position into a first machine learning model trained in advance to acquire the head gesture of a user; the head pose comprises a head pitch angle, a yaw angle and a roll angle;

Acquiring the watching distance from a user to the intelligent screen and the illumination intensity of the external environment, and inputting the watching distance, the illumination intensity, the head posture and the mouth fixation point position into a second machine learning model trained in advance so as to acquire the optimal adjustment parameters of the parallax barrier array; the optimal adjustment parameters comprise an optimal pitch and an optimal angle of the parallax barrier array;

And dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen.

Further, the processing the face image captured by the infrared camera includes:

filtering the noise of the face image, and then carrying out graying treatment to obtain a gray image;

Obtaining a pre-stored pupil template image, setting the step length as 1, placing the pupil template image on a gray level image in a sliding window mode, and performing cross-correlation calculation to obtain the similarity of each overlapping window on the gray level image to obtain Q similarity, wherein Q is an integer larger than zero;

Sorting the Q similarities according to the value from large to small, and marking an overlapping window corresponding to the first similarity as a pupil area;

Carrying out Gaussian blur on the gray level image, highlighting a pupil area, determining a circular boundary of the pupil by using Hough transformation, acquiring a center coordinate of the circular boundary, and marking the center coordinate as the center coordinate of the pupil in an image coordinate system; the pupil center coordinates include a left pupil center coordinate and a right pupil center coordinate.

Further, the processing the face image captured by the infrared camera further includes:

filtering the noise of the face image to obtain a gray image, and carrying out Gaussian blur;

loading a pre-training face detection model in a Dlib library, and inputting the gray level image after Gaussian blur into the pre-training face detection model in a Dlib library to obtain a face boundary region;

loading a 68-point facial feature point detection model of a Dlib library, inputting a face boundary region in an image form into the 68-point facial feature point detection model of the Dlib library to obtain coordinates of U facial feature points, wherein U is an integer greater than zero;

and extracting coordinates of pupil characteristic points in the U facial characteristic points, and marking the coordinates of the pupil characteristic points as pupil center coordinates in an image coordinate system.

Further, the converting pupil center coordinates in the image coordinate system into gaze point positions of the user on the intelligent screen includes:

Calibrating the infrared camera to obtain internal parameters of the camera; the internal parameters include focal length And image optical center coordinates；

Extracting pupil center coordinates in an image coordinate system of a pupilBy an internal reference matrixIs inverse transformed to obtain pupil center coordinates in the camera coordinate system：

；

；

Wherein:；

acquiring camera optical center coordinates According to pupil center coordinates in a camera coordinate systemAnd camera optical center coordinatesCalculating a line-of-sight vectorAnd according to the optical center coordinates of the cameraAnd line of sight vectorSetting a line-of-sight equation: ; wherein: To be a line of sight equation, represent line of sight in Position under parameters; Is a parameter; ；

the screen plane equation of the intelligent screen is set as follows: ; wherein: Is the normal vector of the plane surface and, Is a constant term that is used to determine the degree of freedom,Is the coordinates of a point in space on the screen plane;

substituting the line-of-sight equation into the screen plane equation to solve the parameters ：

；

Parameters are setSubstituting into the line of sight equation to calculate the position of the gaze point：

；

Wherein:。

further, the training logic of the pre-trained first machine learning model is as follows:

acquiring historical head posture training data, and dividing the historical head posture training data into a head posture training set and a head posture testing set; the historical head posture training data comprises head posture characteristic data and corresponding head postures thereof;

The head gesture characteristic data comprise a nose point fixation point position, a mouth fixation point position and a pupil fixation point position;

Constructing a first regression network, taking the head posture characteristic data in the head posture training set as input data in the first regression network, taking the head posture in the head posture training set as output data in the first regression network, and training the first regression network to obtain an initial first regression network;

and performing model verification on the initial first regression network by using the head posture test set, and outputting the initial first regression network which is smaller than or equal to a preset first test error threshold value as a first machine learning model.

Further, the training logic of the pre-trained second machine learning model is as follows:

Acquiring historical adjustment parameter training data, and dividing the historical adjustment parameter training data into an adjustment parameter training set and an adjustment parameter test set; the history adjustment parameter training data comprises adjustment parameter characteristic data and the corresponding optimal adjustment parameters;

the adjustment parameter characteristic data comprise a viewing distance, illumination intensity, head posture and a mouth fixation point position;

Constructing a second regression network, taking the characteristic data of the adjustment parameters in the adjustment parameter training set as input data of the second regression network, taking the optimal adjustment parameters in the adjustment parameter training set as output data of the second regression network, and training the second regression network to obtain an initial second regression network;

and performing model verification on the initial second regression network by using the adjustment parameter test set, and outputting the initial second regression network which is smaller than or equal to a preset second test error threshold value as a second machine learning model.

Further, the optimal adjustment parameters in the historical adjustment parameter training data are obtained by optimizing by utilizing a genetic algorithm;

the logic for obtaining the optimal adjustment parameters by using the genetic algorithm is as follows:

a1: initializing a population: randomly generating an original population, wherein the original population comprises Y individuals, each individual represents the interval and the angle of a group of parallax barrier arrays, and Y is an integer greater than zero;

a2: and (3) adaptability evaluation: acquiring 3D picture definition and stereo parallax of intelligent screen display content under each individual; inputting the definition of the 3D picture and the stereo parallax into a pre-constructed fitness function, and calculating to obtain the fitness of each individual;

a3: selecting: adopting a roulette method to select two individuals with high fitness in an original population as male parents and female parents;

a4: crossing: performing crossover operations on the male parent and the female parent to produce a new individual;

a5: variation: performing mutation operation on the new individuals to obtain E new individuals, combining the E new individuals into a new population, replacing the original population with the new population, and returning to the step a2;

a6: repeating the steps a 2-a 5 until the adaptability of the individuals in the original population or the new population is greater than or equal to a preset adaptability threshold value, or the iteration number is greater than or equal to a preset maximum iteration number threshold value, outputting the distance and the angle of the parallax barrier array represented by the corresponding individuals as the optimal adjustment parameters, and obtaining the optimal distance and the optimal angle of the parallax barrier array.

Further, the calculation formula of the pre-constructed fitness function is as follows: ; wherein: In order to be adaptable to the degree of adaptation, For the definition of the 3D picture,For the reference value of the definition of the picture,For the stereoscopic parallax of the light,Is a reference value for stereoscopic parallax.

An open hole three-dimensional stereoscopic display system based on an intelligent screen, comprising:

The data acquisition module is used for processing the face image captured by the infrared camera to locate pupil center coordinates in an image coordinate system, and converting the pupil center coordinates in the image coordinate system through a geometric transformation algorithm to obtain pupil fixation point positions on the intelligent screen; the pupil fixation point positions comprise a left pupil fixation point position and a right pupil fixation point position;

The gesture determining module is used for acquiring the nose point gazing point position and the mouth gazing point position, and inputting the nose point gazing point position, the mouth gazing point position and the pupil gazing point position into a first machine learning model trained in advance so as to acquire the head gesture of a user; the head pose comprises a head pitch angle, a yaw angle and a roll angle;

The adjustment parameter acquisition module is used for acquiring the watching distance from the user to the intelligent screen and the illumination intensity of the external environment, and inputting the watching distance, the illumination intensity, the head gesture and the mouth fixation point position into a second machine learning model trained in advance so as to acquire the optimal adjustment parameters of the parallax barrier array; the optimal adjustment parameters comprise an optimal pitch and an optimal angle of the parallax barrier array;

The three-dimensional display control module is used for dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen.

Compared with the prior art, the invention has the beneficial effects that:

The application discloses a naked eye three-dimensional display method and a system based on an intelligent screen, wherein the method comprises the following steps: positioning pupil center coordinates in an image coordinate system, and obtaining pupil fixation point positions through a geometric transformation algorithm; inputting the nose point fixation point position, the mouth fixation point position and the pupil fixation point position into a first machine learning model trained in advance to obtain the head gesture of a user; inputting the watching distance, the illumination intensity, the head gesture and the mouth fixation point position into a pre-trained second machine learning model to obtain the optimal adjustment parameters of the parallax barrier array; dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen; based on the above process, the application can track the head gesture of the watching user and the pupil viewpoint of the eyes on the intelligent screen in real time, further, the optimal adjustment parameters of the parallax barrier array are determined according to the parameters such as the head gesture and the pupil viewpoint, and the parallax barrier array is dynamically adjusted, thereby being beneficial to avoiding the abnormal display phenomena such as ghost, distortion or blurring which can be seen by the watching user along with the azimuth change of the watching user; further, the display content that can guarantee that the user watched intelligent screen at different angles is 3D image, is favorable to improving the broadcast display effect of intelligent screen advertisement and user's viewing experience sense to intelligent screen advertisement.

Drawings

FIG. 1 is a flow chart of a naked eye three-dimensional stereoscopic display method based on an intelligent screen;

fig. 2 is a schematic block diagram of a naked eye three-dimensional stereoscopic display system based on an intelligent screen.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

Referring to fig. 1, the disclosure of the present embodiment provides a naked eye three-dimensional stereoscopic display method based on a smart screen, wherein a parallax barrier array and a camera module are arranged on the smart screen, the camera module includes at least 2 infrared cameras, and the method includes:

s101: processing the face image captured by the infrared camera to locate the pupil center coordinate in the image coordinate system, and converting the pupil center coordinate in the image coordinate system by a geometric transformation algorithm to obtain the pupil fixation point position on the intelligent screen; the pupil fixation point positions comprise a left pupil fixation point position and a right pupil fixation point position;

It should be appreciated that: naked eye 3D is a technology capable of viewing 3D images or videos without wearing 3D glasses, and includes three implementations, parallax barrier, lenticular lens, and light field display, respectively; the parallax barrier mode is arranged in front of the display screen in an array mode, so that the left eye and the right eye of a user can see different pixel columns on the screen, each eye of the user receives images sent from different angles, and the space effect of the 3D images is rebuilt in the brain; the method is widely applied because of relatively simple implementation and low cost; however, parallax barriers generally provide optimal 3D effects only within a specific angular range; if the viewing angle is too large, the 3D effect may decrease rapidly, even creating visual discomfort; therefore, the invention provides a solution to the problem that the parallax barrier can only provide the best 3D effect in a specific angle range;

In one specific embodiment, the processing the face image captured by the infrared camera includes:

filtering the noise of the face image, and then carrying out graying treatment to obtain a gray image;

Obtaining a pre-stored pupil template image, setting the step length as 1, placing the pupil template image on a gray level image in a sliding window mode, and performing cross-correlation calculation to obtain the similarity of each overlapping window on the gray level image to obtain Q similarity, wherein Q is an integer larger than zero;

It should be noted that: the basic idea of the cross-correlation calculation is that one image is used as a template, the other image is used as a matching object, the template is placed on the matching object, and the similarity of each overlapped part is calculated by sliding the template on the matching object in a window, so that the same area (namely an overlapped window) of the two images can be obtained; the similarity of each overlapping window is obtained by calculation through a similarity acquisition algorithm; the similarity of each overlapping window comprises, but is not limited to, a cosine similarity algorithm and the like;

Sorting the Q similarities according to the value from large to small, and marking an overlapping window corresponding to the first similarity as a pupil area;

Carrying out Gaussian blur on the gray level image, highlighting a pupil area, determining a circular boundary of the pupil by using Hough transformation, acquiring a center coordinate of the circular boundary, and marking the center coordinate as the center coordinate of the pupil in an image coordinate system;

wherein the pupil center coordinates include a left pupil center coordinate and a right pupil center coordinate;

It should be noted that: through Gaussian blur, image noise can be eliminated and pupil areas are highlighted; the Hough transformation is specifically Hough circle transformation, and the Hough circle transformation can identify a circular structure in an image and determine the center position and the radius of the circular structure;

in another specific embodiment, the processing the face image captured by the infrared camera further includes:

filtering the noise of the face image to obtain a gray image, and carrying out Gaussian blur;

loading a pre-training face detection model in a Dlib library, and inputting the gray level image after Gaussian blur into the pre-training face detection model in a Dlib library to obtain a face boundary region;

It should be noted that: the pre-trained face detection model in the Dlib library is a machine learning model for detecting face regions in images, which is based on directional gradient histogram (Histogram of Oriented Gradients, HOG) features and linear support vector machine (Support Vector Machine, SVM) classifiers; the shape and edge characteristics of the face are identified by analyzing the local gradient direction of the image, so that the area where the face is located is detected;

loading a 68-point facial feature point detection model of a Dlib library, inputting a face boundary region in an image form into the 68-point facial feature point detection model of the Dlib library to obtain coordinates of U facial feature points, wherein U is an integer greater than zero;

Extracting coordinates of pupil feature points in the U facial feature points, and marking the coordinates of the pupil feature points as pupil center coordinates in an image coordinate system;

It should be noted that: the 68-point facial feature point detection model of the Dlib library is a computer vision model for detecting facial feature points; the model can position 68 characteristic points in a given face image, wherein the characteristic points comprise key points of eyes, eyebrows, noses, mouths and facial contours, in a model for detecting the 68-point facial characteristic points in a Dlib library, the characteristic points of the left pupil are 36 th to 41 th points in a model for 68-point in a Dlib library, and the calculation formula of coordinates is as follows: The characteristic points of the right pupil are 42-47 points in a 68-point model in a Dlib library, and the calculation formula of the coordinates is as follows: ; wherein: Is the coordinates of the ith feature point;

in an implementation, the converting pupil center coordinates in the image coordinate system into gaze point positions of the user on the smart screen includes:

Calibrating the infrared camera to obtain internal parameters of the camera; the internal parameters include focal length And image optical center coordinates；

It should be appreciated that: the method comprises the steps of calibrating an infrared camera by using a calibration tool (such as OpenCV);

extracting pupil center coordinates in an image coordinate system of a pupil By an internal reference matrixIs inverse transformed to obtain pupil center coordinates in the camera coordinate system：

；

；

Wherein:；

acquiring camera optical center coordinates According to pupil center coordinates in a camera coordinate systemAnd camera optical center coordinatesCalculating a line-of-sight vectorAnd according to the optical center coordinates of the cameraAnd line of sight vectorSetting a line-of-sight equation: ; wherein: To be a line of sight equation, represent line of sight in Position under parameters; as a parameter, representing the distance from the pupil center to the gaze point location; Usually, is ；

The screen plane equation of the intelligent screen is set as follows: ; wherein: Is a planar normal vector Is a constant term that is used to determine the degree of freedom,Is the coordinates of a point in space on the screen plane;

substituting the line-of-sight equation into the screen plane equation to solve the parameters ：

；

Parameters are setSubstituting into the line of sight equation to calculate the position of the gaze point：

；

Wherein:；

It should be appreciated that: the image optical center is a point on the image plane, representing the point where the optical axis passes through the image plane, and the position of the image optical center is determined by the internal parameters of the camera, and is usually located at the center of the imaging sensor (such as a CMOS sensor); and the camera optical center is the origin of the camera coordinate system through which all light rays will converge during imaging.

S102: acquiring a nose point gazing point position and a mouth gazing point position, and inputting the nose point gazing point position, the mouth gazing point position and the pupil gazing point position into a first machine learning model trained in advance to acquire the head gesture of a user; the head pose comprises a head pitch angle, a yaw angle and a roll angle;

it should be appreciated that: the pitch angle of the head refers to the upward or downward rotation angle of the head, the yaw angle of the head refers to the leftward or rightward rotation angle of the head, and the roll angle of the head refers to the left-right inclination angle of the head;

it should be noted that: the nose point gazing point position refers to the gazing point position of the nose point of the user on the intelligent screen (namely, the projection position of the nose point of the user on the intelligent screen), and the mouth gazing point position refers to the gazing point position of the mouth of the user on the intelligent screen (namely, the projection position of the central point of the mouth on the intelligent screen); the logic for acquiring the nose point fixation point position and the mouth fixation point position is the same as the logic for acquiring the pupil fixation point position, and details can refer to the description content of the pupil fixation point position and are not repeated here;

Also to be described is: before the nose point fixation point position and the mouth fixation point position are obtained, the captured face image is also required to be processed so as to obtain the nose point center coordinate and the mouth nose point center coordinate in an image coordinate system; in the process of processing the captured face image to obtain the center coordinates of the nose tip and the center coordinates of the mouth tip, the 68-point facial feature point detection model of the Dlib library is loaded to collect feature points of the nose tip and feature points of the mouth, in the 68-point facial feature point detection model of the Dlib library, the feature points of the nose tip are 33 points in the 68-point model of the Dlib library, and the calculation formula of the coordinates is as follows: the characteristic points of the mouth are 48 th to 67 th points in a 68-point model of a Dlib library, and the calculation formula of coordinates is as follows: ；

Specifically, the training logic of the pre-trained first machine learning model is as follows:

acquiring historical head posture training data, and dividing the historical head posture training data into a head posture training set and a head posture testing set; the historical head posture training data comprises head posture characteristic data and corresponding head postures thereof;

The head gesture characteristic data comprise a nose point fixation point position, a mouth fixation point position and a pupil fixation point position;

It should be noted that: the head posture feature data in the history head posture training data is obtained by processing and geometrically transforming the face image, and details are described above and are not repeated here; the head gestures in the historical head gesture training data comprise head pitch angle, yaw angle and roll angle, and the head gestures are obtained after real-time acquisition of head gestures of experimenters wearing sensing equipment (such as an IMU sensor) in the experimental process by technicians;

Constructing a first regression network, taking the head posture characteristic data in the head posture training set as input data in the first regression network, taking the head posture in the head posture training set as output data in the first regression network, and training the first regression network to obtain an initial first regression network;

performing model verification on the initial first regression network by using the head posture test set, and outputting the initial first regression network which is smaller than or equal to a preset first test error threshold value as a first machine learning model;

It should be noted that: the first regression network is specifically one of regression algorithm models such as decision trees, random forests, support vector machines, linear regression, polynomial regression or neural networks;

S103: acquiring the watching distance from a user to the intelligent screen and the illumination intensity of the external environment, and inputting the watching distance, the illumination intensity, the head posture and the mouth fixation point position into a second machine learning model trained in advance so as to acquire the optimal adjustment parameters of the parallax barrier array; the optimal adjustment parameters comprise an optimal pitch and an optimal angle of the parallax barrier array;

It should be noted that: the watching distance from the user to the intelligent screen and the illumination intensity of the external environment are acquired through external equipment, wherein the external equipment comprises, but is not limited to, a laser range finder, an illumination sensor and the like;

Specifically, the training logic of the pre-trained second machine learning model is as follows:

Acquiring historical adjustment parameter training data, and dividing the historical adjustment parameter training data into an adjustment parameter training set and an adjustment parameter test set; the history adjustment parameter training data comprises adjustment parameter characteristic data and the corresponding optimal adjustment parameters;

the adjustment parameter characteristic data comprise a viewing distance, illumination intensity, head posture and a mouth fixation point position;

it should be noted that: the adjustment parameter characteristic data in the history adjustment parameter training data is obtained after operations such as sensor acquisition and face image processing, and the details are described above and are not repeated here;

wherein, the optimal adjustment parameters in the history adjustment parameter training data are obtained by optimizing by utilizing a genetic algorithm;

specifically, the logic for obtaining the optimal tuning parameters using the genetic algorithm is as follows:

a1: initializing a population: randomly generating an original population, wherein the original population comprises Y individuals, each individual represents the interval and the angle of a group of parallax barrier arrays, and Y is an integer greater than zero;

a2: and (3) adaptability evaluation: acquiring the 3D picture definition and the stereo parallax of the display content of the intelligent screen under each individual (namely the interval and the angle of each group of parallax barrier arrays in the original population); inputting the definition of the 3D picture and the stereo parallax into a pre-constructed fitness function, and calculating to obtain the fitness of each individual;

it should be noted that: the 3D picture definition is obtained by performing definition calculation on a 3D picture image formed after image shooting of the intelligent screen display content, and the calculation formula of the 3D picture definition is as follows: ; wherein: For the definition of the 3D picture, AndFor being the lateral and longitudinal gradients of the image at the r-th pixel point, N is the total number of pixels in the 3D picture image; and the stereo parallax adopts two bracket cameras to align with the screen of the intelligent screen so as to simulate the visual angles of the left eye and the right eye of a person, captures the views of the left eye and the right eye, extracts the detection characteristic points in the views of the left eye and the right eye through an image processing software application algorithm (such as SIFT, SURF or ORB), adopts a descriptor matching method (such as FLANN or BFMatcher) to pair the characteristic points in the views of the left eye and the right eye so as to obtain paired characteristic points, calculates the position difference of each pair of matched characteristic points in the horizontal direction (or the vertical direction) to obtain the stereo parallax, wherein the calculation formula of the stereo parallax is as follows: ; wherein: And The horizontal coordinates of the corresponding paired feature points in the left and right eye images, and M is the total number of paired feature points;

The calculation formula of the pre-constructed fitness function is as follows: ; wherein: In order to be adaptable to the degree of adaptation, For the definition of the 3D picture,For the reference value of the definition of the picture,For the stereoscopic parallax of the light,Is a reference value for stereoscopic parallax; wherein, the reference value of definition and the reference value of stereo parallax are set by technicians according to experimental data;

a3: selecting: adopting a roulette method to select two individuals with high fitness in an original population as male parents and female parents;

Roulette is a common selection method used in genetic algorithms to select individuals with higher fitness to enter the next generation; the method simulates the roulette process, and each individual obtains a corresponding 'roulette' area according to the fitness of the individual; the higher the fitness, the larger the corresponding area, and the higher the probability of being selected;

a4: crossing: performing crossover operations on the male parent and the female parent to produce a new individual;

It should be noted that: performing crossover operations on male and female parents is based on crossover operations including, but not limited to, one of single-point crossover, uniform crossover, sequential crossover, or the like; wherein two parents (i.e., a male parent and a female parent) produce offspring by combining their genetic information; this process helps introduce diversity in the space of the solution and may create new individuals that are more environmentally friendly;

a5: variation: performing mutation operation on the new individuals to obtain E new individuals, combining the E new individuals into a new population, replacing the original population with the new population, and returning to the step a2;

In the genetic algorithm, mutation operation is used for introducing gene diversity and preventing the algorithm from falling into local optimum; the mutation operation of the new individual is realized by means of uniform mutation or Gaussian mutation and the like;

a6: repeating the steps a 2-a 5 until the fitness of the individuals in the original population or the new population is greater than or equal to a preset fitness threshold value, or the iteration number is greater than or equal to a preset maximum iteration number threshold value, outputting the distance and the angle of the parallax barrier array represented by the corresponding individuals as the optimal adjustment parameters, and obtaining the optimal distance and the optimal angle of the parallax barrier array;

illustratively: assuming that the maximum iteration number is 100 times, after each iteration, recording an individual with the highest fitness in the current population and a fitness value of the individual; if the fitness value is found to be not changed remarkably in a certain generation, the convergence condition is considered to be reached, iteration is stopped, and the distance and the angle of the parallax barrier array represented by the corresponding individual are output as the optimal adjustment parameters;

Constructing a second regression network, taking the characteristic data of the adjustment parameters in the adjustment parameter training set as input data of the second regression network, taking the optimal adjustment parameters in the adjustment parameter training set as output data of the second regression network, and training the second regression network to obtain an initial second regression network;

Performing model verification on the initial second regression network by using the adjustment parameter test set, and outputting the initial second regression network which is smaller than or equal to a preset second test error threshold value as a second machine learning model;

it should be noted that: the second regression network is specifically one of regression algorithm models such as decision trees, random forests, support vector machines, linear regression, polynomial regression or neural networks;

S104: dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen;

It should be noted that: a compact motor or a circuit for controlling the intelligent screen is also arranged in the intelligent screen; when the optimal adjustment parameters are obtained, the parallax barrier array is dynamically adjusted by controlling the compact motor or the circuit, so that the parallax barrier array is at the optimal distance and the optimal angle, and further, the display content (such as advertisements) of the intelligent screen can be ensured to be 3D images when a user views the intelligent screen at different angles, and further, the optimal advertising 3D effect can be ensured to be experienced by the user at different viewing angles.

Example 2

Referring to fig. 2, based on the same inventive concept, the disclosure of the present embodiment provides a naked eye three-dimensional stereoscopic display system based on an intelligent screen, and the details of the embodiment are not described in detail in the description of the relevant part in embodiment 1, where the system includes:

The data acquisition module 210 is configured to process the face image captured by the infrared camera to locate a pupil center coordinate in the image coordinate system, and convert the pupil center coordinate in the image coordinate system by using a geometric transformation algorithm to obtain a pupil fixation point position on the intelligent screen; the pupil fixation point positions comprise a left pupil fixation point position and a right pupil fixation point position;

The gesture determining module 220 is configured to obtain a nose point gaze point position and a mouth gaze point position, and input the nose point gaze point position, the mouth gaze point position, and the pupil gaze point position into a first machine learning model trained in advance to obtain a head gesture of a user; the head pose comprises a head pitch angle, a yaw angle and a roll angle;

The adjustment parameter obtaining module 230 is configured to obtain a viewing distance from the user to the smart screen and an illumination intensity of an external environment, and input the viewing distance, the illumination intensity, the head gesture and the mouth gaze point position into a second machine learning model trained in advance, so as to obtain an optimal adjustment parameter of the parallax barrier array; the optimal adjustment parameters comprise an optimal pitch and an optimal angle of the parallax barrier array;

The stereoscopic display control module 240 is configured to dynamically adjust the parallax barrier array according to the optimal adjustment parameter, so as to implement continuous multi-angle 3D display of the smart display content.

The above formulas are all formulas with dimensionality removed and numerical value calculated, the formulas are formulas with the latest real situation obtained by software simulation by collecting a large amount of data, and preset parameters, weights and threshold selection in the formulas are set by those skilled in the art according to the actual situation.

The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center over a wired network or a wireless network. The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more sets of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the several embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely one, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Finally: the foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims (8)

1. The utility model provides a three-dimensional stereoscopic display method of bore hole based on wisdom screen, be provided with parallax barrier array and camera module on the wisdom screen, the camera module includes 2 at least infrared cameras, its characterized in that, the method includes:

Processing the face image captured by the infrared camera to locate the pupil center coordinate in the image coordinate system, and converting the pupil center coordinate in the image coordinate system by a geometric transformation algorithm to obtain the pupil fixation point position on the intelligent screen; the pupil fixation point positions comprise a left pupil fixation point position and a right pupil fixation point position;

the converting the pupil center coordinates in the image coordinate system includes:

Calibrating the infrared camera to obtain internal parameters of the camera; the internal parameters include focal length And image optical center coordinates；

Extracting pupil center coordinates in an image coordinate system of a pupilBy an internal reference matrixIs inverse transformed to obtain pupil center coordinates in the camera coordinate system：

；

；

Wherein:；

acquiring camera optical center coordinates According to pupil center coordinates in a camera coordinate systemAnd camera optical center coordinatesCalculating a line-of-sight vectorAnd according to the optical center coordinates of the cameraAnd line of sight vectorSetting a line-of-sight equation: ; wherein: To be a line of sight equation, represent line of sight in Position under parameters; Is a parameter; ；

the screen plane equation of the intelligent screen is set as follows: ; wherein: 、、 Is the normal vector of the plane surface and, Is a constant term that is used to determine the degree of freedom,Is the coordinates of a point in space on the screen plane;

substituting the line-of-sight equation into the screen plane equation to solve the parameters ：

；

Parameters are setSubstituting into the line of sight equation to calculate the position of the gaze point：

；

Wherein:；

acquiring a nose point gazing point position and a mouth gazing point position, and inputting the nose point gazing point position, the mouth gazing point position and the pupil gazing point position into a first machine learning model trained in advance to acquire the head gesture of a user; the head pose comprises a head pitch angle, a yaw angle and a roll angle;

Acquiring the watching distance from a user to the intelligent screen and the illumination intensity of the external environment, and inputting the watching distance, the illumination intensity, the head posture and the mouth fixation point position into a second machine learning model trained in advance so as to acquire the optimal adjustment parameters of the parallax barrier array; the optimal adjustment parameters comprise an optimal pitch and an optimal angle of the parallax barrier array;

And dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen.

2. The smart screen-based naked eye three-dimensional stereoscopic display method according to claim 1, wherein the processing of the face image captured by the infrared camera comprises:

filtering the noise of the face image, and then carrying out graying treatment to obtain a gray image;

Obtaining a pre-stored pupil template image, setting the step length as 1, placing the pupil template image on a gray level image in a sliding window mode, and performing cross-correlation calculation to obtain the similarity of each overlapping window on the gray level image to obtain Q similarity, wherein Q is an integer larger than zero;

Sorting the Q similarities according to the value from large to small, and marking an overlapping window corresponding to the first similarity as a pupil area;

Carrying out Gaussian blur on the gray level image, highlighting a pupil area, determining a circular boundary of the pupil by using Hough transformation, acquiring a center coordinate of the circular boundary, and marking the center coordinate as the center coordinate of the pupil in an image coordinate system; the pupil center coordinates include a left pupil center coordinate and a right pupil center coordinate.

3. The smart screen-based naked eye three-dimensional stereoscopic display method according to claim 2, wherein the processing the face image captured by the infrared camera further comprises:

filtering the noise of the face image to obtain a gray image, and carrying out Gaussian blur;

loading a pre-training face detection model in a Dlib library, and inputting the gray level image after Gaussian blur into the pre-training face detection model in a Dlib library to obtain a face boundary region;

loading a 68-point facial feature point detection model of a Dlib library, inputting a face boundary region in an image form into the 68-point facial feature point detection model of the Dlib library to obtain coordinates of U facial feature points, wherein U is an integer greater than zero;

and extracting coordinates of pupil characteristic points in the U facial characteristic points, and marking the coordinates of the pupil characteristic points as pupil center coordinates in an image coordinate system.

4. The smart screen-based naked eye three-dimensional stereoscopic display method according to claim 3, wherein the training logic of the pre-trained first machine learning model is as follows:

acquiring historical head posture training data, and dividing the historical head posture training data into a head posture training set and a head posture testing set; the historical head posture training data comprises head posture characteristic data and corresponding head postures thereof;

The head gesture characteristic data comprise a nose point fixation point position, a mouth fixation point position and a pupil fixation point position;

Constructing a first regression network, taking the head posture characteristic data in the head posture training set as input data in the first regression network, taking the head posture in the head posture training set as output data in the first regression network, and training the first regression network to obtain an initial first regression network;

and performing model verification on the initial first regression network by using the head posture test set, and outputting the initial first regression network which is smaller than or equal to a preset first test error threshold value as a first machine learning model.

5. The smart screen-based naked eye three-dimensional stereoscopic display method according to claim 4, wherein the training logic of the pre-trained second machine learning model is as follows:

Acquiring historical adjustment parameter training data, and dividing the historical adjustment parameter training data into an adjustment parameter training set and an adjustment parameter test set; the history adjustment parameter training data comprises adjustment parameter characteristic data and the corresponding optimal adjustment parameters;

the adjustment parameter characteristic data comprise a viewing distance, illumination intensity, head posture and a mouth fixation point position;

Constructing a second regression network, taking the characteristic data of the adjustment parameters in the adjustment parameter training set as input data of the second regression network, taking the optimal adjustment parameters in the adjustment parameter training set as output data of the second regression network, and training the second regression network to obtain an initial second regression network;

and performing model verification on the initial second regression network by using the adjustment parameter test set, and outputting the initial second regression network which is smaller than or equal to a preset second test error threshold value as a second machine learning model.

6. The naked eye three-dimensional display method based on the intelligent screen according to claim 5, wherein the optimal adjustment parameters in the historical adjustment parameter training data are obtained by optimizing by using a genetic algorithm;

the logic for obtaining the optimal adjustment parameters by using the genetic algorithm is as follows:

a1: initializing a population: randomly generating an original population, wherein the original population comprises Y individuals, each individual represents the interval and the angle of a group of parallax barrier arrays, and Y is an integer greater than zero;

a2: and (3) adaptability evaluation: acquiring 3D picture definition and stereo parallax of intelligent screen display content under each individual; inputting the definition of the 3D picture and the stereo parallax into a pre-constructed fitness function, and calculating to obtain the fitness of each individual;

a3: selecting: adopting a roulette method to select two individuals with high fitness in an original population as male parents and female parents;

a4: crossing: performing crossover operations on the male parent and the female parent to produce a new individual;

a5: variation: performing mutation operation on the new individuals to obtain E new individuals, combining the E new individuals into a new population, replacing the original population with the new population, and returning to the step a2;

a6: repeating the steps a 2-a 5 until the adaptability of the individuals in the original population or the new population is greater than or equal to a preset adaptability threshold value, or the iteration number is greater than or equal to a preset maximum iteration number threshold value, outputting the distance and the angle of the parallax barrier array represented by the corresponding individuals as the optimal adjustment parameters, and obtaining the optimal distance and the optimal angle of the parallax barrier array.

7. The smart screen-based naked eye three-dimensional stereoscopic display method according to claim 6, wherein the calculation formula of the pre-constructed fitness function is:； In order to be adaptable to the degree of adaptation, For the definition of the 3D picture,For the reference value of the definition of the picture,For the stereoscopic parallax of the light,Is a reference value for stereoscopic parallax.

8. Naked eye three-dimensional display system based on wisdom screen, characterized by comprising:

The data acquisition module is used for processing the face image captured by the infrared camera to locate pupil center coordinates in an image coordinate system, and converting the pupil center coordinates in the image coordinate system through a geometric transformation algorithm to obtain pupil fixation point positions on the intelligent screen; the pupil fixation point positions comprise a left pupil fixation point position and a right pupil fixation point position;

the converting the pupil center coordinates in the image coordinate system includes:

Calibrating the infrared camera to obtain internal parameters of the camera; the internal parameters include focal length And image optical center coordinates；

Extracting pupil center coordinates in an image coordinate system of a pupilBy an internal reference matrixIs inverse transformed to obtain pupil center coordinates in the camera coordinate system：

；

；

Wherein:；

acquiring camera optical center coordinates According to pupil center coordinates in a camera coordinate systemAnd camera optical center coordinatesCalculating a line-of-sight vectorAnd according to the optical center coordinates of the cameraAnd line of sight vectorSetting a line-of-sight equation: ; wherein: To be a line of sight equation, represent line of sight in Position under parameters; Is a parameter; ；

the screen plane equation of the intelligent screen is set as follows: ; wherein: 、、 Is the normal vector of the plane surface and, Is a constant term that is used to determine the degree of freedom,Is the coordinates of a point in space on the screen plane;

substituting the line-of-sight equation into the screen plane equation to solve the parameters ：

；

Parameters are setSubstituting into the line of sight equation to calculate the position of the gaze point：

；

Wherein:；

The gesture determining module is used for acquiring the nose point gazing point position and the mouth gazing point position, and inputting the nose point gazing point position, the mouth gazing point position and the pupil gazing point position into a first machine learning model trained in advance so as to acquire the head gesture of a user; the head pose comprises a head pitch angle, a yaw angle and a roll angle;

The adjustment parameter acquisition module is used for acquiring the watching distance from the user to the intelligent screen and the illumination intensity of the external environment, and inputting the watching distance, the illumination intensity, the head gesture and the mouth fixation point position into a second machine learning model trained in advance so as to acquire the optimal adjustment parameters of the parallax barrier array; the optimal adjustment parameters comprise an optimal pitch and an optimal angle of the parallax barrier array;

The three-dimensional display control module is used for dynamically adjusting the parallax barrier array according to the optimal adjustment parameters so as to realize continuous multi-angle 3D display of the display content of the intelligent screen.