CN114596628A - Photoelectric sensor for gesture recognition - Google Patents
Photoelectric sensor for gesture recognition Download PDFInfo
- Publication number
- CN114596628A CN114596628A CN202210139242.3A CN202210139242A CN114596628A CN 114596628 A CN114596628 A CN 114596628A CN 202210139242 A CN202210139242 A CN 202210139242A CN 114596628 A CN114596628 A CN 114596628A
- Authority
- CN
- China
- Prior art keywords
- receiving
- receiving tube
- photoelectric
- sensor circuit
- tubes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 claims abstract description 45
- 238000005286 illumination Methods 0.000 claims description 69
- 230000033001 locomotion Effects 0.000 claims description 16
- 238000005070 sampling Methods 0.000 claims description 12
- 239000003990 capacitor Substances 0.000 claims description 11
- 230000010354 integration Effects 0.000 claims description 11
- 238000003491 array Methods 0.000 abstract description 5
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000000034 method Methods 0.000 description 19
- 230000008569 process Effects 0.000 description 18
- 230000008859 change Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000006854 communication Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 108010076504 Protein Sorting Signals Proteins 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000001678 irradiating effect Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Landscapes
- User Interface Of Digital Computer (AREA)
Abstract
The invention relates to the technical field of photoelectric sensors, in particular to a photoelectric sensor for gesture recognition, which comprises: the receiving tube array is provided with a plurality of receiving tubes which are arranged in a cross shape; the sensor circuit comprises a time sequence unit, a receiving tube array sequentially selects receiving tubes along the horizontal direction and the vertical direction under the control of time sequence signals, photoelectric signals corresponding to the receiving tubes are input into the sensor circuit to generate output signals, and a processing module is connected with the sensor circuit and recognizes gestures according to the output signals. The invention has the beneficial effects that: by arranging the receiving tube arrays in the cross-shaped arrangement, the gesture is effectively recognized, meanwhile, the receiving area of the receiving tube arrays is reduced, and the size of the sensor is reduced; meanwhile, the receiving tube array is controlled by the time sequence signal to sequentially select the receiving tubes in the specific direction on the cross shape to generate output signals, so that the gesture recognition algorithm is simplified, and the accuracy of gesture recognition is improved.
Description
Technical Field
The invention relates to the technical field of photoelectric sensors, in particular to a photoelectric sensor for gesture recognition.
Background
A photoelectric sensor is a device for converting optical signals into electric signals based on photoelectric effect. Based on the physical characteristics of the photoelectric sensor, the photoelectric sensor is widely applied to various position measurement applications of automation equipment, the Internet of things and smart homes. Gesture recognition refers to a technology for recognizing a gesture of a user by acquiring signals through corresponding sensor equipment and processing the signals through a related algorithm. According to different types of sensors, gesture recognition schemes can be divided into gesture recognition technologies depending on image sensors, gesture recognition technologies based on infrared images, gesture recognition technologies based on infrared signals and the like.
In the prior art, there is a technical scheme for recognizing a gesture based on a photoelectric sensor. For example, in a product of a certain automobile manufacturer, recognition of a gesture is realized by projecting infrared light and collecting reflected light of a hand. Because the photoelectric sensor itself does not directly collect the optical image of the user, the privacy of the user can be well protected, and therefore the photoelectric sensor is widely applied.
However, in the actual implementation process, the inventor finds that the photoelectric sensor in the prior art often needs to rely on a receiving tube array composed of a large number of receiving tubes to collect reflected light, which results in a large sensing area of the whole sensor, so that the volume of the photoelectric sensor cannot be reduced, and the requirements of users cannot be met well. Meanwhile, because the receiving tube array in the prior art has a large scale, signals output by a large number of receiving tubes need to be processed to obtain the actual situation of reflected light, which results in that the whole identification algorithm in the prior art is complex and the identification rate is not high.
Disclosure of Invention
In view of the above problems in the prior art, a photoelectric sensor for gesture recognition is provided.
The specific technical scheme is as follows:
a photosensor for gesture recognition, comprising:
the receiving tube array is provided with a plurality of receiving tubes, and the receiving tubes are arranged in a cross shape;
a sensor circuit connected to the array of receiving tubes;
the sensor circuit comprises a timing unit, and the timing unit sends a timing signal to the receiving tube array;
the receiving tube array sequentially selects the receiving tubes along the horizontal direction and the vertical direction under the control of the time sequence signals, photoelectric signals corresponding to the receiving tubes are input into the sensor circuit, and the sensor circuit generates output signals according to the photoelectric signals;
and the processing module is connected with the sensor circuit and identifies gestures according to the output signals.
Preferably, the sensor circuit comprises a timing unit, and the timing unit is connected with the receiving tube array;
the time sequence unit sends a time sequence signal to the receiving tube array, and the receiving tube array enables the photoelectric signal of the specific receiving tube to be input into the sensor circuit according to the time sequence signal.
Preferably, the sensor circuit comprises:
the input end of the switched capacitor amplifier is connected with the output end of the receiving tube array;
the input end of the sampling integration unit is connected with the output end of the switched capacitor amplifier;
and the input end of the signal processing unit is connected with the output end of the sampling integration unit.
Preferably, the sensor circuit is further connected with at least one transmitting tube;
the transmitting tube and the receiving tube array are arranged on the same side and point to the same direction.
Preferably, the receiving tube array comprises:
the first receiving pipes are sequentially arranged along the horizontal direction;
a plurality of second receiving pipes, which are arranged in sequence along the vertical direction;
the third receiving pipe is positioned in the center of the cross shape, and the first receiving pipe, the second receiving pipe and the third receiving pipe form the cross shape;
the third receiving pipe comprises four sub receiving pipes which are arranged in a rectangular shape;
the sub receiving tubes form two first receiving tubes arranged along the horizontal direction under the control of the time sequence signals;
or, the sub receiving tubes form two second receiving tubes arranged along the vertical direction under the control of the time sequence signals.
Preferably, the receiving tube array sequentially inputs the photoelectric signals generated by the first receiving tube to the sensor circuit along a first direction under the control of the timing signal;
and sequentially inputting the photoelectric signals generated by the second receiving tube into the sensor circuit along a second direction.
Preferably, the receiving tube array inputs the photoelectric signal of the first receiving tube to the sensor circuit through a first port, and the receiving tube array inputs the photoelectric signal of the second receiving tube to the sensor circuit through a second port;
the time sequence unit sends a first time sequence signal and a second time sequence signal to the receiving tube array, so that the receiving tube array simultaneously inputs the photoelectric signal of the first receiving tube and the photoelectric signal of the second receiving tube to the sensor circuit;
a preset phase difference exists between the first timing signal and the second timing signal, so that the third receiving tube sequentially forms the first receiving tube and the second receiving tube under the control of the first timing signal and the second timing signal.
Preferably, the processing module acquires the illumination intensity of each receiving tube in the receiving tube array according to the processing signal;
and the processing module identifies the gesture according to the variation trend of the illumination intensity.
Preferably, when the illumination intensity of the first receiving tube increases sequentially along the horizontal direction and the illumination intensity of the second receiving tubes reaches a first peak value at the same time, it indicates that the gesture moves in the horizontal direction;
and when the illumination intensity of the second receiving tube is sequentially increased along the vertical direction and the illumination intensity of the plurality of first receiving tubes reaches a second peak value at the same time, indicating that the gesture moves in the vertical direction.
Preferably, when the illumination intensity of the first receiving tube and the illumination intensity of the second receiving tube are simultaneously increased, the gesture is indicated to be close to the photoelectric sensor;
and when the illumination intensity of the first receiving tube and the illumination intensity of the second receiving tube are simultaneously reduced, indicating that the gesture is far away from the photoelectric sensor.
The technical scheme has the following advantages or beneficial effects: by arranging the receiving tube arrays in a cross-shaped arrangement, the gesture is effectively recognized, meanwhile, the receiving area of the receiving tube arrays is reduced, and the size of the sensor is reduced; meanwhile, the receiving tube array is controlled by the time sequence signal to sequentially select the receiving tubes in the specific direction on the cross shape to generate the output signal, so that the whole output signal sequence is associated with the actual light spot projection condition, the gesture recognition algorithm is simplified, and higher accuracy is realized.
Drawings
Embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings. The drawings are, however, to be regarded as illustrative and explanatory only and are not restrictive of the scope of the invention.
FIG. 1 is an overall schematic diagram of an embodiment of the present invention;
FIG. 2 is a schematic diagram of a sensor circuit according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a launch tube in an embodiment of the invention;
FIG. 4 is a schematic diagram of a photoelectric sensor according to an embodiment of the present invention;
FIG. 5 is a schematic view of a receiver tube array in accordance with an embodiment of the present invention;
FIG. 6 is a schematic view of a third receiving pipe according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating the amplitude variation of the output signal according to an embodiment of the present invention;
FIG. 8 is a diagram illustrating the variation of the amplitude of the output signal according to another embodiment of the present invention;
FIG. 9 is a graph illustrating the output signal amplitude corresponding to each quadrant in an embodiment of the present invention;
FIG. 10 is a diagram illustrating the amplitude variation of the output signal according to an embodiment of the present invention;
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.
The invention comprises the following steps:
a photosensor for gesture recognition, as shown in fig. 1, comprising:
the receiving tube array 1 is provided with a plurality of receiving tubes 2, and the receiving tubes 2 are arranged in a cross shape;
the sensor circuit 3 is connected with the receiving tube array 1;
the sensor circuit 3 comprises a timing unit 31, and the timing unit 31 sends a timing signal to the receiving tube array 1;
the receiving tube array 1 sequentially selects the receiving tubes 2 along the horizontal direction and the vertical direction under the control of the time sequence signals, and inputs photoelectric signals corresponding to the receiving tubes 2 into the sensor circuit 3, and the sensor circuit 3 generates output signals according to the photoelectric signals;
and the processing module 4 is connected with the sensor circuit 3, and the processing module 4 identifies gestures according to the output signals.
Specifically, the sensing area to the photoelectric sensor among the prior art is great, the receiver tube is in large quantity, and then make the great problem of photoelectric sensor's whole volume, through setting up the receiver tube array 1 of cross arrangement in this embodiment, reduced the quantity of receiver tube 2 when effectively discerning the gesture, and then realized less sensing area, thereby make photoelectric sensor's whole size and circuit scale can reduce, realized the smaller photoelectric sensor of volume.
Further, in order to solve the problems that in the prior art, a receiving tube array of a photoelectric sensor is large, the number of receiving tubes is large, and therefore a large number of signals need to be processed by an identification algorithm, and the identification algorithm is complex and a high-performance processing device is needed, in this embodiment, the central portion of a reflected light spot is obtained by arranging the cross-shaped receiving tube array 1, the receiving tube array 1 is controlled by a timing unit, the receiving tubes 2 are sequentially selected along a specific direction to input photoelectric signals into the sensor circuit 3, and finally, the processing module 4 can respectively obtain the distribution conditions of the illumination intensity in the X-axis direction and the Y-axis direction, that is, the distribution conditions of the light spots in the X-axis direction and the Y-axis direction in a single period, and further, effective identification of the gesture is achieved through a simplified algorithm. The gesture recognition method has the advantages that the number of signals needing to be processed and processing steps are reduced while effective gesture recognition is achieved, recognition algorithms are simplified, processing efficiency is improved, requirements for performance and power consumption of the processing module 4 are low, and the overall size of the photoelectric sensor is further reduced.
In the implementation process, the receiving tube array 1 includes a switch matrix, the switch matrix includes a plurality of controllable switches, each controllable switch is connected to a receiving tube, and is sequentially turned on and off under the driving of a control signal, so as to sequentially input photoelectric signals into the sensor circuit 3. The control signal may be generated by a register unit 6, in which register unit 6 corresponding control parameters are fixed, and the control signal is generated according to the control parameters to control the on/off of the controllable switch. Or the control signal is a time sequence signal, and the switch matrix respectively controls each controllable switch to be sequentially switched on or switched off according to the time sequence signal. The sensor circuit 3 mainly includes a signal processing circuit for performing amplification, phase locking, demodulation, and arithmetic processing on the received light intensity, and further generating an output signal for representing the received light intensity. The sensor circuits 3 are connected to the processing module 4 through respective communication ports. The processing module 4 may be understood as one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, Micro Controllers (MCUs), microprocessors (microprocessors), or other electronic components, and is provided with a corresponding computer program for recognizing an operation gesture of a user according to an output signal. In an embodiment, the processing module 4 further has a communication port connected to an external controlled device to output a corresponding operation instruction, so as to control the controlled device. In another embodiment, the processing module 4 may also be connected to the switching tube through a corresponding control circuit, and the switching value control is performed according to the recognition result.
In a preferred embodiment, as shown in fig. 2, the sensor circuit 3 includes a timing unit 31, the timing unit 31 is connected to the receiving tube array 1;
the timing unit 31 sends a timing signal to the receiving tube array 1, and the receiving tube array enables the photoelectric signal of a specific receiving tube to be input into the sensor circuit 3 according to the timing signal.
Specifically, to the problem that the size of the photoelectric sensor for gesture recognition in the prior art is large, in this embodiment, a timing signal is sent to the receiving tube array 1 through the timing unit 31, so that the receiving tube array 1 sequentially selects to access a specific receiving tube 2 to the sensor circuit 3 according to the timing signal, and further, the photoelectric signal of each receiving tube 2 is sequentially input to the sensor circuit 3, thereby achieving receiving of the photoelectric signal of each receiving tube 2, reducing the circuit scale for connecting the receiving tubes 1, and reducing the sensor size.
In the implementation process, each receiving tube 2 is provided with a corresponding serial number, for example, in an embodiment, the receiving tubes 2 are divided into an X-axis receiving tube and a Y-axis receiving tube according to the positions of the receiving tubes in a cross shape; the X-axis receiving pipe is sequentially arranged to be DX 1-DXN along the positive direction of the X axis; the Y-axis receiving tubes are sequentially arranged into DY 1-DYN receiving tubes along the negative direction of the Y axis. The timing unit 31 outputs corresponding timing signals to sequentially turn on the controllable switches in the receiving tube array 11 according to a specific sequence, for example, the horizontal axis direction or the vertical axis direction of the receiving tube array 1, so that the sensor circuit 3 sequentially processes the photoelectric signals output by each receiving tube 2, thereby implementing one-by-one measurement of the illumination intensity of each receiving tube 2. For example, in an embodiment, the receiving tubes 2 are connected one by one along the positive direction of the X axis, and then the receiving tubes 2 are connected one by one along the negative direction of the Y axis, so as to measure the illumination intensity of each receiving tube 2. When the last receiving tube 2 of the Y axis is reached, the first receiving tube 2 of the X axis is returned again to realize the process of circular detection.
In a preferred embodiment, as shown in fig. 2, the sensor circuit 3 further comprises:
the input end of the switched capacitor amplifier 32 is connected with the output end of the receiving tube array 1;
a sampling integration unit 33, wherein the input end of the sampling integration unit 33 is connected with the output end of the switched capacitor amplifier 32;
and the input end of the signal processing unit 34 is connected with the output end of the sampling integration unit 33.
Specifically, in order to achieve a better processing effect on the light intensity signals output by the plurality of receiving tubes 2, in this embodiment, the switched capacitor amplifier 211, the sampling integration unit 222, and the signal processing unit 223 are sequentially arranged to implement the phase locking and demodulation processes according to the time sequence signal, so as to accurately process the light intensity signals output by each receiving tube 2.
In the implementation process, the switched capacitor amplifier 211 is a phase-locked amplifier implemented based on a switched capacitor technology, and performs a phase-locked process based on a timing signal generated by the timing circuit 22, thereby achieving lower power consumption and better noise suppression effect. In one embodiment, the sampling and integrating unit 33 and the signal processing unit 34 are connected to a register unit, and the conversion gain, the sampling times and the integration time of the current-voltage are adjusted by the control parameters recorded in the register unit. The register unit 6 may be connected to an external control device via a bus protocol in the prior art to receive control parameters. According to actual needs, the register unit 6 can rely on a check bit or a check algorithm to check the received control parameter, so as to achieve better interference resistance. The signal processing unit 21 is an arithmetic logic circuit that performs operations such as addition, subtraction, multiplication, division, etc. on the received signal under adjustment of the control parameter, thereby generating an output signal for representing the intensity of illumination.
In a preferred embodiment, as shown in fig. 3, the sensor circuit is also connected to at least one emitter tube 5;
the transmitting tube 5 and the receiving tube array 1 are arranged on the same side and point to the same direction.
Specifically, in order to realize a better gesture recognition effect, the transmitting tube 5 and the receiving tube array 1 are arranged on the same side, and the transmitting tube 5 emits irradiation light upwards. When the user does not perform an operation, the receiving tube array 1 does not receive light. When a user performs gesture operation, a reflecting surface a is formed above the transmitting tube 5 by the hand, light is further reflected to the receiving tube array 1, and the sensor circuit 1 judges the current gesture of the user according to the receiving tube 1 which receives the reflected light.
In the implementation process, the number, size and power of the transmitting tube 5 and the receiving tube 2 can be adjusted adaptively according to actual requirements, such as FOV (Field of View) of the sensor, sensing distance, operation area, etc. For example, in one embodiment, a user is required to perform a large range of operations, so that the number of receiving tubes 2 is increased to form a larger receiving tube array 1, thereby realizing the recognition of a larger range of motions such as swinging arms and waving hands. In another embodiment, the power of the transmitting tube 5 is adjusted, and a lens group is added in front of the receiving tube array 1, so that the operation gesture recognition at a longer distance is realized.
In a preferred embodiment, the sensor circuit 3 further includes a transmitting tube driving circuit 51, the transmitting tube driving circuit 51 is connected to the timing unit 31 and the transmitting tube 5, and the transmitting tube driving circuit drives the transmitting tube 5 to generate the irradiating light under the control of the timing signal, so that the signal processing circuit 21 and the transmitting tube driving circuit complete phase locking through the same timing signal.
As an alternative embodiment, the sensor circuit 3 can be implemented by a programmable sensor circuit, for example, CN202210103335.0 discloses a programmable photoelectric sensor, in which a circuit portion can be used for the sensor circuit 3. The programmable sensor circuit generally includes a register unit 6, and the register unit 6 is connected to an external upper computer through a communication protocol in the prior art, such as I2C, SPI, or a single-wire communication interface, so as to receive control parameters output by the external upper computer, and further perform parameter adjustment on components such as the switch capacitor amplifier 32, the sampling integration unit 33, the signal processing unit 34, the timing unit 21, and the transmitting tube driving circuit 51, so that the sensor is more suitable for actual requirements. According to different actually set environments, a check algorithm can be added in the communication process, check bits can be set in the register unit, and an independent alarm circuit can be set to avoid the fault of register overturn caused by external electromagnetic interference. The upper computer and the processing device may be the same device, which respectively sends the control parameters through independent ports and receives the signals output by the sensor circuit 3, or sends the control parameters and receives the output signals through the same port. The setting mode of each module can be adjusted according to actual needs. For example, as shown in fig. 4, in an embodiment, the receiving tube array 1, the sensor circuit 2, the transmitting tube 5, the transmitting tube driving circuit 51 and the register 6 are integrated in the photoelectric sensor, and are connected to the external processing module 4 through a bus to implement the gesture recognition function. The processing module 4 is used for sending control parameters to the register unit 6 in the photoelectric sensor and receiving output signals output by the sensor circuit 3, so as to realize a gesture recognition process.
In a preferred embodiment, as shown in fig. 5, the receiving tube array 1 comprises:
a plurality of first receiving pipes 2A, the first receiving pipes 2A being arranged in order along a horizontal direction;
a plurality of second receiving pipes 2B, the second receiving pipes 2B being arranged in order in a vertical direction;
the third receiving pipe 2C is positioned in the center of the cross shape, and the first receiving pipe 2A, the second receiving pipe 2B and the third receiving pipe 2C form the cross shape;
as shown in fig. 6, the third receiving pipe 2C includes four sub receiving pipes 2C1, 2C2, 2C3, 2C4 arranged in a rectangular shape;
sub-receiving pipes 2C1, 2C2, 2C3, 2C4 form two first receiving pipes 2A arranged in the horizontal direction under the control of timing signals, wherein sub-receiving pipe 2C1 and sub-receiving pipe 2C3 are connected in parallel to form one first receiving pipe 2A, and sub-receiving pipe 2C2 and sub-receiving pipe 2C4 are connected in parallel to form the other first receiving pipe 2A;
or, the sub receiving pipes 2C1, 2C2, 2C3, 2C4 form two second receiving pipes 2B arranged in the vertical direction under the control of the timing signals, wherein the sub receiving pipe 2C1 and the sub receiving pipe 2C3 are connected in parallel to form one second receiving pipe 2B, and the sub receiving pipe 2C2 and the sub receiving pipe 2C4 are connected in parallel to form another first receiving pipe 2B.
Specifically, in this embodiment, the receiving tube array 1 is set to be a cross shape, and due to the reduction of the number of the receiving tubes 2, an actual processing algorithm is simplified, so that effective gesture recognition is realized, and a smaller sensor volume is realized. Meanwhile, by arranging the first receiving tube 2A and the second receiving tube 2B which are sequentially arranged, the receiving tube array 1 can input photoelectric signals generated by the receiving tubes 2 into the sensor circuit 3 along a specific sequence, and the input circuit scale of the sensor circuit 3 is reduced.
Furthermore, in order to achieve a better recognition effect on the operation gesture, in this embodiment, the X axis and the Y axis are selected to respectively obtain the output signals, so as to completely record the variation trend of the illumination intensity in the X axis direction and the Y axis direction, and further comprehensively obtain the overall situation of the operation gesture. Therefore, by providing third receiver tube 2C composed of four sub-receiver tubes 2C1, 2C2, 2C3, 2C4, and controlling the area of each sub-receiver tube 2C1, 2C2, 2C3, 2C4 to be one-half of the area of first receiver tube 2A or second receiver tube 2B, it can be achieved that in receiver tube array 1, two first receiver tubes 2A are formed by sub-receiver tube 2C1 and sub-receiver tube 2C3, and sub-receiver tube 2C2 and sub-receiver tube 2C4, respectively, in the X-axis direction, or two second receiver tubes 2B are formed by sub-receiver tube 2C1 and sub-receiver tube 2C2, and sub-receiver tube 2C3 and sub-receiver tube 2C4, respectively, as necessary, thereby achieving complete recording of the intensity of received light rays in the X-axis or the Y-axis.
In the implementation process, when the photoelectric signal of the X axis needs to be acquired, the third receiving tube 2C connects the sub receiving tube 2C1 and the sub receiving tube 2C3 in parallel and connects the sub receiving tube 2C2 and the sub receiving tube 2C4 in parallel under the action of the timing signal, so that two receiving tubes with the area equal to that of the first receiving tube 2A are formed in the X axis direction, thereby realizing the continuous recording of the light intensity of the X axis. When the photoelectric signal of the Y axis needs to be acquired, the third receiving tube 2C connects the sub receiving tube 2C1 and the sub receiving tube 2C2 in parallel and connects the sub receiving tube 2C3 and the sub receiving tube 2C4 in parallel under the action of the timing signal, and then two receiving tubes whose areas are equal to those of the second receiving tube 2B are formed in the Y axis direction, so that the continuous recording of the light intensity of the Y axis is realized.
In a preferred embodiment, the receiving tube array 1 sequentially inputs the photoelectric signals generated by the first receiving tube 2A to the sensor circuit along the first direction under the control of the timing signals;
and sequentially inputting the photoelectric signals generated by the second receiving tube 2B to the sensor circuit in the second direction.
Specifically, in order to completely recognize the gesture operation on the cross-shaped sensor array 1, in this embodiment, the photoelectric signals generated by the first receiving tube 2A and the second receiving tube 2B are sequentially input to the sensor circuit 2 by setting a specific direction, and then a serial number-amplitude rectangular coordinate system can be established based on the serial number of the receiving tube 2 on the X axis or the Y axis and the signal intensity of the receiving tube, and the signal intensity of each receiving tube 2 is fitted on the coordinate system to obtain the peak value of the illumination intensity and the distribution of the illumination intensity of the X axis or the Y axis of the receiving tube array 1. Subsequently, by comparing the peak value of the illumination intensity and the distribution of the illumination intensity in the coordinate system along the time sequence, the continuous change of the signal intensity on the receiving tube array 1 on the X axis and/or the Y axis caused by the hand movement of the user can be effectively judged, and then the gesture can be recognized.
In a preferred embodiment, the receiving tube array 1 inputs the photoelectric signal of the first receiving tube 2A to the sensor circuit 3 through a first port, and the receiving tube array 1 inputs the photoelectric signal of the second receiving tube 2B to the sensor circuit 3 through a second port;
the timing unit 31 sends the first timing signal and the second timing signal to the receiving tube array 1, so that the receiving tube array 1 inputs the photoelectric signal of the first receiving tube 2A and the photoelectric signal of the second receiving tube 2B to the sensor circuit 3 at the same time;
a preset phase difference exists between the first timing signal and the second timing signal, so that the third receiving tube 2C forms the first receiving tube 2A and the second receiving tube 2B in sequence under the control of the first timing signal and the second timing signal.
Specifically, to the problem that photoelectric sensor among the prior art identification efficiency is lower, receive the photoelectric signal of horizontal direction and vertical direction output respectively through setting up first port and second port in this embodiment, realized the fast processing to sensor array 1's output signal, reduced the length of time of single cycle, help promoting photoelectric sensor's detection frequency, and then realize better identification efficiency.
In an implementation, the first timing signal and the second timing signal may be the same type of timing signal, which is used to control the receiving tube array 1 to sequentially input the photoelectric signals of the first receiving tube 2A and the second receiving tube 2B into the sensor circuit 3, and to control the third receiving tube 2C to connect the sub receiving tubes 2C1, 2C2, 2C3, 2C4 in parallel to form the first receiving tube 2A or the second receiving tube 2B. The third receiving tube 2C can be switched in the phase difference by setting the phase difference, so as to complete the input process of the photoelectric signal in the specific direction. Because the signals in the X direction and the Y direction are measured and processed simultaneously, the gesture recognition speed is increased.
As an alternative implementation manner, in the above embodiment, the emitter driving circuit 51 may be respectively connected to the two emitters 5, and the emitter driving circuit 51 drives each emitter 5 to form the illumination light under the control of the first timing signal and the second timing signal, so as to generate the same or different illumination light to the first receiving tube 2A and the second receiving tube 2B, respectively, so that the sensor circuit 3 completes sequential phase locking on the first receiving tube 2A and the second receiving tube 2B according to the first timing signal and the second timing signal, thereby improving the processing accuracy of the sensor circuit 3. In the above control process, the emission times of the two emission tubes 5 do not overlap, so that the irradiation light with a small fluctuation range is formed on the receiving tube array 1. Alternatively, in another embodiment, the emitter driving circuit 51 drives one or more emitters 5 to simultaneously emit light under the control of the first timing signal and the second timing signal. It should be understood that, according to different control manners, the illumination intensity on the receiving tube array 1 changes accordingly, so that the output signal changes within a certain range, such as the maximum and minimum values, the kurtosis, the monotonous interval, and the like of the overall output signal, but it does not affect the realization of gesture recognition according to the change rule of the output signal.
In a preferred embodiment, the processing module 4 obtains the illumination intensity of each receiving tube 2 in the receiving tube array 1 according to the processing signal;
the processing module 4 identifies the gesture according to the variation trend of the illumination intensity.
Specifically, in order to completely recognize the gesture operation on the cross-shaped sensor array 1, in this embodiment, the illumination intensity of each receiving tube 2 is obtained, and the operation gesture is accurately recognized according to the variation trend of the illumination intensity on the X axis or the Y axis.
In the implementation process, because the transmitting tube 5 and the receiving tube array 1 are arranged on the same side, a relatively uniform light spot can be formed on the receiving tube array 1 through the reflecting surface A, and the brightness of the central point of the light spot is obviously higher than the brightness of the edge of the light spot, so that the moving condition of the light spot on the receiving tube array 1 can be reflected by recording the sequence number of the receiving tube 2 corresponding to the peak value of the illumination intensity along the time sequence, and the reflecting surface A is further represented, namely the movement of the hand of the user in the directions of the X axis and the Y axis, and further the corresponding gesture is identified. Meanwhile, the distance between the reflecting surface a and the receiving tube array 1 affects the light intensity and distribution of the light spots reflected back to the receiving tube array, so that the size and the size change of the light spots on the receiving tube array 1 can be further judged by judging the distribution of the illumination intensity, and the reflecting surface a, namely the movement of the user's hand on the Z axis (relative to the distance of the photoelectric sensor), is represented.
In a preferred embodiment, when the illumination intensity of the first receiving tube 2A increases sequentially along the horizontal direction and the illumination intensities of the second receiving tubes 2B reach a first peak value at the same time, the gesture is indicated as a horizontal movement;
specifically, in order to achieve a better gesture recognition effect, in this embodiment, the illumination intensity variation trend of the first receiving tube 2A in the X-axis direction and the illumination intensity variation trend of the second receiving tube 2B in the Y-axis direction are compared to comprehensively determine in which direction the gesture moves.
In an implementation process, in an embodiment, the first receiving tube 2A of the receiving tube array 1 specifically includes 8 receiving tubes DX1 to DX8, where DX1 is the leftmost receiving tube in the horizontal direction, and DX8 is the rightmost receiving tube in the horizontal direction. The second receiving pipe 2B includes 8 receiving pipes DY1 to DY8, where DY1 is the uppermost receiving pipe in the vertical direction, and DY8 is the lowermost receiving pipe in the vertical direction. As shown in fig. 7, when the user's hand moves from left to right in the horizontal direction, the output signals of the plurality of receiving pipes DX1 to DX8 in the first receiving pipe 2A are acquired and sorted to obtain: when the hand of the user is positioned at the leftmost side, the receiving tube with the highest illumination intensity is a receiving tube DX1, and the receiving tube with the weakest illumination intensity is a receiving tube DX 8; when the user's hand is located at the central position, the receiving tube with the maximum illumination intensity is the receiving tube DX4 or the receiving tube DX 5; when the user's hand is positioned at the leftmost side, the receiving tube with the highest illumination intensity is DX8, and the receiving tube with the weakest illumination intensity is DX 1. By comparing the above-mentioned variation trends, it can be determined that the hand of the user moves from left to right in the horizontal direction. Meanwhile, as can be seen from comparison of the illumination intensities of the receiving tubes DY1 to DY8, when the user's hand is located on the left side or the right side, the illumination intensities of the plurality of second receiving tubes 2B are substantially the same, and only when the user's hand is located at the center position, that is, the illumination intensity of the receiving tube DX4 or the receiving tube DX5 is the maximum, the illumination intensity of the second receiving tube 2B reaches the peak value, and the illumination intensity of the receiving tube DY4 or the receiving tube DY5 closest to the center position is the maximum. Based on the above comparison, there is no movement of the user's hand in the vertical direction.
And when the illumination intensity of the second receiving tubes is sequentially increased along the vertical direction and the illumination intensity of the first receiving tubes reaches a second peak value at the same time, indicating that the gesture moves in the vertical direction.
In implementation, as shown in fig. 8, when the hand of the user moves from bottom to top in the vertical direction, obtaining and sequencing the output signals of the plurality of receiving pipes DY1 to DY8 in the second receiving pipe 2B may result in: when the hand of the user is positioned at the lowest side, the receiving tube with the highest illumination intensity is the receiving tube DY8, and the receiving tube with the weakest illumination intensity is the receiving tube DY 1; when the hand of the user is positioned at the central position, the receiving tube with the maximum illumination intensity is the receiving tube DY4 or the receiving tube DY 5; when the user's hand is positioned at the uppermost side, the receiving tube with the highest illumination intensity is the receiving tube DY1, and the receiving tube with the weakest illumination intensity is DY 8. By comparing the above change trends, it can be determined that the hand of the user moves from bottom to top in the vertical direction. Meanwhile, as can be seen from comparison of the illumination intensities of the receiving pipes DX1 to DX8, when the user's hand is positioned at the upper side or the lower side, the illumination intensities of the plurality of first receiving pipes 2A are substantially the same, and only when the user's hand is positioned at the central position, that is, the illumination intensity of the receiving pipe DY4 or the receiving pipe DY5 is the maximum, the illumination intensity of the first receiving pipe 2A reaches the peak value, and the illumination intensity of the receiving pipe DX4 or the receiving pipe DX5 closest to the central position is the maximum. Based on the above comparison, there is no movement of the user's hand in the horizontal direction.
Further, based on the above process, the continuous movement of the hand of the user in the whole plane can be obtained by comprehensively comparing the variation trend of the amplitude of the received signal of the first receiving pipe 2A in the horizontal direction and the variation trend of the amplitude of the received signal of the second receiving pipe 2B in the vertical direction.
In one embodiment, as shown in fig. 9, a planar rectangular coordinate system is established with the central point of the receiving tube array 1 as the origin, the central line of the first receiving tube 2A as the X-axis, and the central line of the second receiving tube 2B as the Y-axis. One received signal strength sample for each quadrant.
For example, when the user's hand is located at the first quadrant, the received signal amplitude in the first receiving pipe 2A is DX8 with the largest amplitude and DX1 with the smallest amplitude, the received signal amplitude in the second receiving pipe 2B is DY1 with the largest amplitude and DY8 with the smallest amplitude;
when the hand of the user is located at the second quadrant, the received signal amplitude in the first receiving tube 2A is DX1 with the maximum amplitude, the received signal amplitude is DX8 with the minimum amplitude, the received signal amplitude in the second receiving tube 2B is DY1 with the maximum amplitude, and the received signal amplitude is DY8 with the minimum amplitude;
when the user's hand is located at the third quadrant, at this time, the maximum received signal amplitude in the first receiving tube 2A is DX1, the minimum received signal amplitude is DX8, the maximum received signal amplitude in the second receiving tube 2B is DY8, and the minimum received signal amplitude is DY 1;
when the user's hand is located at the fourth quadrant, the received signal amplitude of the first receiving tube 2A is DX8 with the largest amplitude, and the received signal amplitude of the second receiving tube 2B is DX1 with the largest amplitude, and DY8 with the smallest amplitude, and DY1 with the smallest amplitude.
Based on the four received signal strength samples, the quadrant of the hand of the user in the rectangular coordinate system can be determined.
At this time, if the user wants to make a continuous motion, such as counterclockwise rotation, the motion trajectory should be from the first quadrant to the second quadrant, and after reaching the third quadrant, the user turns to the fourth quadrant and returns to the first quadrant, so that the output signal of the photoelectric sensor sequentially changes according to the received signal intensity samples, and the processing module 4 can recognize the continuous motion as counterclockwise rotation. Similarly, when the user needs to make a diagonal movement, the movement locus is that the first quadrant moves to the third quadrant through the origin, and then in the output signal of the photoelectric sensor, the output amplitude of each receiving tube 2 reaches the maximum value, and then is reduced to the corresponding received signal intensity sample. Based on this process, the diagonal movement of the continuous motion past the origin can be identified. The actions can be simply combined according to actual needs to make operations such as clockwise rotation, L-shaped gestures, J-shaped gestures and the like.
In a preferred embodiment, when the illumination intensity of the first receiving tube 2A and the illumination intensity of the second receiving tube 2B are simultaneously increased, the gesture is indicated to be close to the photoelectric sensor;
and when the illumination intensity of the first receiving tube 2A and the illumination intensity of the second receiving tube 2B are simultaneously reduced, indicating that the gesture is far away from the photoelectric sensor.
In the embodiment, as shown in fig. 10, taking the approach of the user's hand to the photosensor as an example, by comparing the trend of the amplitude change of the output signals of the plurality of receiving tubes DX1 to DX8 in the first receiving tube 2A with the trend of the amplitude change of the output signals of the plurality of receiving tubes DY1 to DY8 in the second receiving tube 2B, when the user's hand approaches the photosensor, the reflected light simultaneously increases the light intensities of the plurality of receiving tubes DX1 to DX8 in the first receiving tube 2A and the plurality of receiving tubes DY1 to DY8 in the second receiving tube 2B, and thus the output signals are increased. It is derived based on this trend of change that the hand of the user is approaching the photosensor. Similarly, when the amplitudes of the output signals of the first receiving tube 2A and the second receiving tube 2B decrease at the same time, it indicates that the hand of the user is far away from the photoelectric sensor; when the amplitudes of the output signals of the first receiving tube 2A and the second receiving tube 2B are not changed, it indicates that the distance between the hand of the user and the photoelectric sensor is not changed.
As an optional implementation manner, a corresponding artificial intelligence model is preset in the processing module 4, and the artificial intelligence model is trained by using the output signals as a training set, so that the artificial intelligence model can classify the actually acquired output signals in application to accurately recognize the gesture of the user.
The invention has the beneficial effects that: through adjusting the mode of arranging of receiving tube array 1, simplifying the algorithm, when improving the rate of accuracy to user's gesture recognition, reduced receiving tube array 1's whole area, and then make photoelectric sensor's whole volume littleer, be convenient for integrate in all kinds of equipment. Through receiving the photoelectric signal of the output of the first receiver tube 2A in the horizontal direction or the second receiver tube 2B in the vertical direction in proper order, form a set of continuously in processing module 4, an output signal amplitude array for representing the illumination intensity in the current horizontal direction or the vertical direction, and then realized effectively recording the facula position that forms on receiver tube array 1 because of the hand reflection of user, be convenient for judge the removal of facula through the change of comparing the array, in order to discern user's gesture and remove. By selecting the programmable switched capacitor amplifier 32, the sampling integration unit 33, the signal processing unit 34, the timing unit 21 and the transmitting tube driving circuit, effective adjustment of parameters such as amplitude, phase and frequency of output signals is realized, the sensor is ensured not to enter a saturated state or a state with smaller output signals, the sensitivity, the detection distance, the detection range and other parameters of the sensor can be changed according to user requirements in practical application, and the sensor further meets the requirements of users.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
Claims (9)
1. A photosensor for gesture recognition, comprising:
the receiving tube array is provided with a plurality of receiving tubes, and the receiving tubes are arranged in a cross shape;
a sensor circuit connected to the array of receiving tubes;
the sensor circuit comprises a timing unit, and the timing unit sends a timing signal to the receiving tube array;
the receiving tube array sequentially selects the receiving tubes along the horizontal direction and the vertical direction under the control of the time sequence signals, photoelectric signals corresponding to the receiving tubes are input into the sensor circuit, and the sensor circuit generates output signals according to the photoelectric signals;
and the processing module is connected with the sensor circuit and identifies gestures according to the output signals.
2. The photosensor circuit of claim 1, wherein the sensor circuit comprises:
the input end of the switched capacitor amplifier is connected with the output end of the receiving tube array;
the input end of the sampling integration unit is connected with the output end of the switched capacitor amplifier;
and the input end of the signal processing unit is connected with the output end of the sampling integration unit.
3. The photosensor of claim 1, wherein the sensor circuit is further connected to at least one emitter tube;
the transmitting tube and the receiving tube array are arranged on the same side and point to the same direction.
4. The photosensor of claim 1, wherein the array of receiving tubes comprises:
the first receiving pipes are sequentially arranged along the horizontal direction;
a plurality of second receiving pipes, which are arranged in sequence along the vertical direction;
the third receiving tube is positioned in the center of the cross shape, and the first receiving tube, the second receiving tube and the third receiving tube form the cross shape;
the third receiving pipe comprises four sub receiving pipes which are arranged in a rectangular shape;
the sub receiving tubes form two first receiving tubes arranged along the horizontal direction under the control of the time sequence signals;
or the sub receiving tubes form two second receiving tubes arranged along the vertical direction under the control of the time sequence signals.
5. The photoelectric sensor of claim 4, wherein the receiving tube array sequentially inputs the photoelectric signals generated by the first receiving tube to the sensor circuit along a first direction under the control of the timing signal;
and sequentially inputting the photoelectric signals generated by the second receiving tube into the sensor circuit along a second direction.
6. The photosensor of claim 4, wherein the array of receiving tubes inputs the photoelectric signal of the first receiving tube to the sensor circuit through a first port, and the array of receiving tubes inputs the photoelectric signal of the second receiving tube to the sensor circuit through a second port;
the time sequence unit sends a first time sequence signal and a second time sequence signal to the receiving tube array, so that the receiving tube array simultaneously inputs the photoelectric signal of the first receiving tube and the photoelectric signal of the second receiving tube to the sensor circuit;
a preset phase difference exists between the first timing signal and the second timing signal, so that the third receiving tube sequentially forms the first receiving tube and the second receiving tube under the control of the first timing signal and the second timing signal.
7. The photoelectric sensor of claim 4, wherein the processing module obtains an illumination intensity of each receiving tube in the receiving tube array according to the processing signal;
and the processing module identifies the gesture according to the variation trend of the illumination intensity.
8. The photoelectric sensor according to claim 7, wherein the gesture is a horizontal movement when the illumination intensities of the first receiving tubes sequentially increase along a horizontal direction and the illumination intensities of the second receiving tubes reach a first peak at the same time;
and when the illumination intensity of the second receiving tube is sequentially increased along the vertical direction and the illumination intensity of the plurality of first receiving tubes reaches a second peak value at the same time, indicating that the gesture moves in the vertical direction.
9. The photosensor according to claim 7, wherein the gesture is indicated as being close to the photosensor when the illumination intensity of the first receiving tube and the illumination intensity of the second receiving tube increase simultaneously;
and when the illumination intensity of the first receiving tube and the illumination intensity of the second receiving tube are simultaneously reduced, indicating that the gesture is far away from the photoelectric sensor.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210139242.3A CN114596628A (en) | 2022-02-15 | 2022-02-15 | Photoelectric sensor for gesture recognition |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210139242.3A CN114596628A (en) | 2022-02-15 | 2022-02-15 | Photoelectric sensor for gesture recognition |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114596628A true CN114596628A (en) | 2022-06-07 |
Family
ID=81806148
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210139242.3A Pending CN114596628A (en) | 2022-02-15 | 2022-02-15 | Photoelectric sensor for gesture recognition |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114596628A (en) |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101799282A (en) * | 2010-04-28 | 2010-08-11 | 东北大学 | Reflection-type angular displacement transducer and measuring method based on optical fiber array |
US20120176345A1 (en) * | 2009-09-30 | 2012-07-12 | Beijing Irtouch Systems Co., Ltd. | Touch screen, touch system and method for positioning a touch object in touch system |
CN204288121U (en) * | 2014-12-15 | 2015-04-22 | 广东省自动化研究所 | A kind of 3D gesture recognition controller |
CN105353876A (en) * | 2015-11-09 | 2016-02-24 | 深圳市云腾智慧科技有限公司 | Multi-point light sensation based spatial gesture identification control system and method |
CN110367946A (en) * | 2019-08-28 | 2019-10-25 | 杭州兆观传感科技有限公司 | A kind of controllable array reflective photoelectric sensor mould group |
CN113495615A (en) * | 2020-03-20 | 2021-10-12 | 代傲阿扣基金两合公司 | Gesture control module, household appliance and application of gesture control module in appliance |
-
2022
- 2022-02-15 CN CN202210139242.3A patent/CN114596628A/en active Pending
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120176345A1 (en) * | 2009-09-30 | 2012-07-12 | Beijing Irtouch Systems Co., Ltd. | Touch screen, touch system and method for positioning a touch object in touch system |
CN101799282A (en) * | 2010-04-28 | 2010-08-11 | 东北大学 | Reflection-type angular displacement transducer and measuring method based on optical fiber array |
CN204288121U (en) * | 2014-12-15 | 2015-04-22 | 广东省自动化研究所 | A kind of 3D gesture recognition controller |
CN105353876A (en) * | 2015-11-09 | 2016-02-24 | 深圳市云腾智慧科技有限公司 | Multi-point light sensation based spatial gesture identification control system and method |
CN110367946A (en) * | 2019-08-28 | 2019-10-25 | 杭州兆观传感科技有限公司 | A kind of controllable array reflective photoelectric sensor mould group |
CN113495615A (en) * | 2020-03-20 | 2021-10-12 | 代傲阿扣基金两合公司 | Gesture control module, household appliance and application of gesture control module in appliance |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108107796B (en) | Range hood, gesture recognition control method and device thereof and readable storage medium | |
CN108459723B (en) | Infrared gesture recognition device and recognition method | |
TWI540462B (en) | Gesture recognition method and electronic apparatus using the same | |
US9268409B2 (en) | Method and system for gesture identification based on object tracing | |
CN105353876A (en) | Multi-point light sensation based spatial gesture identification control system and method | |
US8907264B2 (en) | Motion and simple gesture detection using multiple photodetector segments | |
CN106612399A (en) | Image generating method and image generating apparatus | |
CN102541303A (en) | Proximity sensor with motion detection | |
CN110456913A (en) | Infrared gesture action recognition method and device | |
CN113504847A (en) | FPGA-based capacitive touch screen signal sampling system and method | |
CN111076770A (en) | Multi-mode intelligent sensor with sensing element and memristor combined | |
CN115561826B (en) | Proximity detection circuit and proximity sensor | |
CN111395471B (en) | Intelligent closestool gesture control system and intelligent closestool gesture control method | |
US8941609B2 (en) | Multi-touch sensing system capable of optimizing touch bulbs according to variation of ambient lighting conditions and method thereof | |
WO2021102908A1 (en) | Signal adjustment method and apparatus, and computer storage medium | |
CN110103241B (en) | Lighting robot, lighting robot control method and control device | |
US20150009140A1 (en) | Navigation device including thermal sensor | |
CN114596628A (en) | Photoelectric sensor for gesture recognition | |
CN100442207C (en) | Infrared touch screen performance improving structure and method | |
WO2009130959A1 (en) | Infrared photodetector device and electronic device | |
CN111095163A (en) | Method and apparatus for detecting user input in dependence on gesture | |
CN109584278B (en) | Target identification device and method | |
US20140198025A1 (en) | System and method for discerning complex gestures using an array of optical sensors | |
Yoo et al. | A 137-μW area-efficient real-time gesture recognition system for smart wearable devices | |
US7247834B2 (en) | Photoelectric sensor for detecting presence/absence of object |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |