CN104391569A - Brain-machine interface system based on cognition and emotional state multi-mode perception - Google Patents
Brain-machine interface system based on cognition and emotional state multi-mode perception Download PDFInfo
- Publication number
- CN104391569A CN104391569A CN201410547000.3A CN201410547000A CN104391569A CN 104391569 A CN104391569 A CN 104391569A CN 201410547000 A CN201410547000 A CN 201410547000A CN 104391569 A CN104391569 A CN 104391569A
- Authority
- CN
- China
- Prior art keywords
- brain
- analysis
- system based
- module
- computer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002996 emotional effect Effects 0.000 title claims abstract description 36
- 230000008447 perception Effects 0.000 title claims abstract description 29
- 230000019771 cognition Effects 0.000 title claims abstract description 25
- 238000004458 analytical method Methods 0.000 claims abstract description 56
- 230000004927 fusion Effects 0.000 claims abstract description 36
- 230000001149 cognitive effect Effects 0.000 claims abstract description 27
- 210000004556 brain Anatomy 0.000 claims abstract description 22
- 238000004364 calculation method Methods 0.000 claims abstract description 11
- 230000006399 behavior Effects 0.000 claims description 34
- WHXSMMKQMYFTQS-UHFFFAOYSA-N Lithium Chemical compound [Li] WHXSMMKQMYFTQS-UHFFFAOYSA-N 0.000 claims description 10
- 229910052744 lithium Inorganic materials 0.000 claims description 10
- 238000007726 management method Methods 0.000 claims description 10
- 238000000034 method Methods 0.000 claims description 10
- 230000003542 behavioural effect Effects 0.000 claims description 9
- 230000001360 synchronised effect Effects 0.000 claims description 8
- 238000001228 spectrum Methods 0.000 claims description 7
- 230000008921 facial expression Effects 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 5
- 230000007177 brain activity Effects 0.000 claims description 4
- 238000004891 communication Methods 0.000 claims description 4
- 239000004020 conductor Substances 0.000 claims description 4
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 230000033764 rhythmic process Effects 0.000 claims description 4
- 230000002146 bilateral effect Effects 0.000 claims description 3
- 210000000624 ear auricle Anatomy 0.000 claims description 3
- 238000010223 real-time analysis Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000002360 prefrontal effect Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 claims description 2
- 210000004761 scalp Anatomy 0.000 claims description 2
- 230000008451 emotion Effects 0.000 abstract description 16
- 230000003925 brain function Effects 0.000 abstract description 3
- 230000010354 integration Effects 0.000 abstract description 3
- 238000000926 separation method Methods 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 10
- 238000011156 evaluation Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000011160 research Methods 0.000 description 5
- 230000001953 sensory effect Effects 0.000 description 5
- 230000003993 interaction Effects 0.000 description 4
- 230000006998 cognitive state Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000007659 motor function Effects 0.000 description 2
- 238000000491 multivariate analysis Methods 0.000 description 2
- 230000008560 physiological behavior Effects 0.000 description 2
- 208000006011 Stroke Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 206010008129 cerebral palsy Diseases 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 210000003477 cochlea Anatomy 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 210000003414 extremity Anatomy 0.000 description 1
- 210000001652 frontal lobe Anatomy 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 210000003141 lower extremity Anatomy 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 210000000578 peripheral nerve Anatomy 0.000 description 1
- 230000003304 psychophysiological effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 208000020431 spinal cord injury Diseases 0.000 description 1
- 230000000638 stimulation Effects 0.000 description 1
- 230000007474 system interaction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
- A61B5/024—Measuring pulse rate or heart rate
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
- A61B5/1116—Determining posture transitions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Biophysics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Pathology (AREA)
- Psychiatry (AREA)
- Physiology (AREA)
- Psychology (AREA)
- Cardiology (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Hospice & Palliative Care (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Neurosurgery (AREA)
- Child & Adolescent Psychology (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Dermatology (AREA)
- Human Computer Interaction (AREA)
- General Physics & Mathematics (AREA)
- Social Psychology (AREA)
- Dentistry (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Neurology (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
本发明公开一种基于认知与情绪状态多模态感知的脑机接口系统，包括便携式生理信号采集与分析系统、基于视频分析的行为识别系统、无线网关和数据融合与反馈计算机系统；本发明能够解决认知/情绪等脑高级功能的解析与反馈问题，并突破传统认知和情绪分离的脑特异性理论的局限，构建基于情绪-认知整合框架下的认知与情绪感知与计算的多模态脑机接口系统，揭示认知/情绪层次脑机融合的计算规律，实现脑与机器之间的高效匹配。
The invention discloses a brain-computer interface system based on multimodal perception of cognitive and emotional states, including a portable physiological signal acquisition and analysis system, a behavior recognition system based on video analysis, a wireless gateway, and a data fusion and feedback computer system; the invention It can solve the analysis and feedback problems of advanced brain functions such as cognition/emotion, and break through the limitations of the traditional brain-specific theory of separation of cognition and emotion, and build a system of cognition and emotion perception and calculation based on the framework of emotion-cognition integration. The multi-modal brain-computer interface system reveals the calculation rules of brain-computer fusion at the cognitive/emotional level, and realizes efficient matching between brain and machine.
Description
技术领域technical field
本发明涉及脑机接口领域，具体涉及一种用于认知/情绪层次的脑与信息系统交互的基于认知与情绪状态多模态感知的脑机接口系统。The invention relates to the field of brain-computer interface, in particular to a brain-computer interface system based on multi-modal perception of cognition and emotional states for cognitive/emotional brain-information system interaction.
背景技术Background technique
人脑是世界上最复杂的、威力最强大的系统，在人脑与机器之间建立脑机接口(Brain-Machine Interface，BMI)，即建立直接的信息通道和全新的交互方式，将涉及众多领域创新的前沿，对科技、经济、军事，甚至社会的发展影响巨大。The human brain is the most complex and powerful system in the world. To establish a brain-machine interface (Brain-Machine Interface, BMI) between the human brain and the machine, that is, to establish a direct information channel and a new way of interaction, will involve The forefront of innovation in many fields has a huge impact on the development of science and technology, economy, military, and even society.
而脑机接口技术因其涉及众多领域创新的前沿，对科技、经济、军事，甚至社会的发展影响巨大，但截至目前，国内外脑机接口领域主要关注的是运动/感觉层次脑机接口领域的研究，只涉及到对脑低层次功能的解析和反馈。国内外脑机接口领域主要关注的是运动/感觉层次脑机接口研究，也取得了一些大的进展。现在不仅发展了一些由肌电、脑电、视觉、语音控制的信息系统，以修复一些动作功能(如行走、取物)、以及修复一些感知功能(如耳蜗、视力恢复)；还为脑或脊髓损伤、中风、脑瘫等因素导致的外周神经或肌肉运动功能受损人群开发了机能增强技术和设备。The brain-computer interface technology has a huge impact on the development of science and technology, economy, military, and even society because it involves the frontiers of innovation in many fields. However, up to now, the domestic and foreign brain-computer interface fields have mainly focused on the motor/sensory level brain-computer interface field. The research only involves the analysis and feedback of the low-level functions of the brain. The field of brain-computer interface at home and abroad mainly focuses on the research of brain-computer interface at the motor/sensory level, and some great progress has been made. Now not only some information systems controlled by myoelectricity, EEG, vision, and voice have been developed to restore some motor functions (such as walking, fetching objects) and some perception functions (such as cochlea, vision recovery); Functional enhancement technologies and devices have been developed for people with impaired peripheral nerve or muscle motor function due to factors such as spinal cord injury, stroke, and cerebral palsy.
现有技术中，有大量专利公开了运动/感觉层次脑机接口系统的构建方法，例如：《基于多模态脑机接口的智能轮椅》(CN201110268891.5)、《针对下肢的层级式功能性电刺激康复系统》(CN200910310630.8)、《一种基于运动想象与P300脑电电位的功能键选择方法》(CN201010509550.8)等。虽然运动/感觉层次脑机接口取得了长足进展，但运动/感觉层次脑机接口研究只涉及到对脑低层次功能的解析和反馈。近15年来，在认知神经科学、神经生化、心理生理计算、信息技术等相关学科的推动下，使得涉及认知和情绪等脑高级智能的脑机接口的研究变得可能。认知/情绪层面脑机接口涉及到对人认知、情感和决策行为的感知、解读、识别和反馈控制，直接涉及更广阔的国家战略需求。In the prior art, there are a large number of patents disclosing the construction method of the motor/sensory hierarchical brain-computer interface system, such as: "Intelligent Wheelchair Based on Multimodal Brain-Computer Interface" (CN201110268891.5), "Hierarchical Functionality for Lower Limbs" Electrical Stimulation Rehabilitation System" (CN200910310630.8), "A Function Key Selection Method Based on Motor Imagery and P300 EEG Potential" (CN201010509550.8), etc. Although the motor/sensory level BCI has made great progress, the research on the motor/sensory level BCI only involves the analysis and feedback of the low-level functions of the brain. In the past 15 years, under the impetus of cognitive neuroscience, neurobiochemistry, psychophysiological computing, information technology and other related disciplines, it has become possible to study brain-computer interfaces involving advanced brain intelligence such as cognition and emotion. The brain-computer interface at the cognitive/emotional level involves the perception, interpretation, identification, and feedback control of human cognition, emotion, and decision-making behavior, and directly involves broader national strategic needs.
此外，目前国内外认知与情绪状态分析与识别研究主要还是集中在对6种基本情感(高兴、悲伤、愤怒、恐惧、惊奇、厌恶)的识别及少量认知状态的识别，而采用的方法主要是单模态分析。目前已经发展的单模态认知与情绪评测方法存在一些局限，多模态信息感知与融合方法已经成为国际学术界公认的方案。但众多指标之间的关联关系、各模态之间的关联关系、各个模态对辨识目标的贡献率等多模态分析的关键问题没有得到很好的解决。In addition, the current research on the analysis and recognition of cognitive and emotional states at home and abroad is mainly focused on the recognition of the six basic emotions (happy, sad, angry, fear, surprise, disgust) and the recognition of a small number of cognitive states. Mainly unimodal analysis. The single-modal cognition and emotion evaluation methods that have been developed so far have some limitations, and the multi-modal information perception and fusion method has become a recognized solution in the international academic community. However, the key issues of multimodal analysis, such as the correlation between many indicators, the correlation between various modes, and the contribution rate of each mode to the identification target, have not been well resolved.
发明内容Contents of the invention
发明目的：本发明的目的在于突破传统单模态脑功能感知方法存在的局限，从生理-行为多模态信息融合视角下实现对认知与情绪状态的实时感知与识别，最终提供一种基于认知与情绪状态多模态感知的脑机接口系统。Purpose of the invention: The purpose of the present invention is to break through the limitations of traditional single-modal brain function perception methods, realize real-time perception and recognition of cognition and emotional states from the perspective of physiological-behavior multi-modal information fusion, and finally provide a method based on A brain-computer interface system for multimodal perception of cognitive and emotional states.
技术方案：本发明的一种基于认知与情绪状态多模态感知的脑机接口系统，包括便携式生理信号采集与分析系统、基于视频分析的行为识别系统、无线网关和数据融合与反馈计算机系统；所述便携式生理信号采集与分析系统以及基于视频分析的行为识别系统均通过无线网关分别与数据融合与反馈计算机系统实现双向通讯连接；所述数据融合与反馈计算机系统通过无线网关向便携式生理信号采集与分析系统以及基于视频分析的行为识别系统同步发送数据采集指令，然后二者分别进行数据生理信号和行为数据的同步采集与实时分析，所述数据融合与反馈计算机系统收到所述分析结果后，对所获得的生理数据和行为数据进行融合计算并得到认知与情绪状态的估计结果，所述估计结果将进一步反馈给执行机构从而实现脑与机器的双向交互接口。Technical solution: A brain-computer interface system based on multi-modal perception of cognitive and emotional states of the present invention, including a portable physiological signal acquisition and analysis system, a behavior recognition system based on video analysis, a wireless gateway, and a data fusion and feedback computer system The portable physiological signal collection and analysis system and the behavior recognition system based on video analysis are respectively connected with the data fusion and feedback computer system through the wireless gateway to realize two-way communication connection; the data fusion and feedback computer system transmits the portable physiological signal through the wireless gateway The acquisition and analysis system and the behavior recognition system based on video analysis synchronously send data acquisition instructions, and then the two perform synchronous acquisition and real-time analysis of data physiological signals and behavior data respectively, and the data fusion and feedback computer system receives the analysis results Finally, the obtained physiological data and behavioral data are fused and calculated to obtain the estimation results of cognitive and emotional states, and the estimation results will be further fed back to the actuator to realize the two-way interactive interface between the brain and the machine.
进一步的，所述便携式生理信号采集与分析系统包括微处理器、用于采集原始脑电信号的双导脑电采集模块、用于采集与心率同步的脉冲信号的耳夹式红外心率采集模块、无线收发模块、SD卡读写模块、电源调节与管理模块、锂电池和USB充电模块，所述微处理器对存储于SD卡读写模块中的原始脑电信号和脉冲信号数据进行分析处理，并通过无线收发模块将分析处理结果实时发送至数据融合与反馈计算机系统；所述USB充电模块对锂电池充电。Further, the portable physiological signal collection and analysis system includes a microprocessor, a dual-conductor EEG collection module for collecting raw EEG signals, an ear-clip infrared heart rate collection module for collecting pulse signals synchronized with heart rate, Wireless transceiver module, SD card read-write module, power regulation and management module, lithium battery and USB charging module, the microprocessor analyzes and processes the original EEG signal and pulse signal data stored in the SD card read-write module, And the analysis and processing results are sent to the data fusion and feedback computer system in real time through the wireless transceiver module; the USB charging module charges the lithium battery.
进一步的，所述原始脑电信号中可提取出反映大脑活动节律的脑电波功率谱，所述脑电波功率谱包括δ波、θ波、α波、β波和γ波；所述脉冲信号中可获得产生心率变异性时域和频率指标。Further, the brain wave power spectrum reflecting the brain activity rhythm can be extracted from the original EEG signal, and the brain wave power spectrum includes delta wave, theta wave, alpha wave, beta wave and gamma wave; Time-domain and frequency indicators of heart rate variability can be obtained.
进一步的，所述双导脑电采集模块由干电池供电，在采集脑电信号时，头皮位置的作用电极放置于双侧前额叶，参考电极和地电极分别放置于左右耳垂处。Further, the dual-lead EEG acquisition module is powered by a dry battery. When collecting EEG signals, the active electrodes at the scalp are placed on the bilateral prefrontal lobes, and the reference electrodes and ground electrodes are respectively placed at the left and right earlobes.
进一步的，所述基于视频分析的行为识别系统包括嵌入式平台、摄像机、无线收发模块和SD卡读写模块，所述嵌入式平台通过摄像机实时记录和分析用户的行为和面部表情并存储于SD卡读写模块，同时通过无线收发模块实时将行为和面部表情的数据发送至数据融合与反馈计算机系统。Further, the behavior recognition system based on video analysis includes an embedded platform, a camera, a wireless transceiver module and an SD card reading and writing module, and the embedded platform records and analyzes the user's behavior and facial expressions in real time through the camera and stores them in the SD card. The card reading and writing module sends the behavior and facial expression data to the data fusion and feedback computer system in real time through the wireless transceiver module.
进一步的，所述数据融合与反馈计算机系统包括计算机和执行机构，所述计算机通过无线网关将采样指令同步发送给便携式生理信号采集与分析系统和基于视频分析的行为识别系统，实现生理信号和行为信号同步采集，并同时通过无线网关接受便携式生理信号采集与分析系统以及基于视频分析的行为识别系统的计算结果，该计算结果经过数据融合算法生成控制信号来驱动执行机构实现脑与机器的双向交互接口。Further, the data fusion and feedback computer system includes a computer and an executive mechanism, and the computer synchronously sends sampling instructions to the portable physiological signal collection and analysis system and the behavior recognition system based on video analysis through the wireless gateway, so as to realize the physiological signal and behavior recognition system. The signal is collected synchronously, and at the same time, the calculation result of the portable physiological signal acquisition and analysis system and the behavior recognition system based on video analysis is received through the wireless gateway. The calculation result generates a control signal through a data fusion algorithm to drive the actuator to realize the two-way interaction between the brain and the machine interface.
进一步的，所述数据融合算法通过分析多模态指标与识别目标之间的关联关系获得认知与情绪状态识别的关键生理及行为指标。Further, the data fusion algorithm obtains key physiological and behavioral indicators for cognitive and emotional state identification by analyzing the correlation between multimodal indicators and identification targets.
进一步的，所述执行机构为机械机构(例如假肢和康复机械等)和虚拟3D场景中的任意一种。Further, the actuator is any one of mechanical mechanisms (such as artificial limbs and rehabilitation machinery, etc.) and virtual 3D scenes.
有益效果：与现有技术相比，本发明具有以下优点：Beneficial effect: compared with the prior art, the present invention has the following advantages:
(1)国内外脑机接口技术目前主要聚焦于对脑低层次功能的解析和反馈；本发明从情绪-认知整合视角对认知与情绪状态进行多模态信息感知，建立认知与情绪评测的关键生理及行为指标，实现对认知、情感和决策行为的感知、解读与识别，在此基础上构建认知/情绪层次脑机接口系统。(1) Brain-computer interface technology at home and abroad currently focuses on the analysis and feedback of low-level brain functions; the present invention performs multi-modal information perception on cognition and emotional states from the perspective of emotion-cognition integration, and establishes cognition and emotion The key physiological and behavioral indicators of the evaluation can realize the perception, interpretation and identification of cognition, emotion and decision-making behavior, and build a cognitive/emotional level brain-computer interface system on this basis.
(2)本发明突破传统认知和情绪分离的脑特异性理论的局限，提出基于情绪-认知整合框架下的认知与情绪感知与计算新方法，探索认知/情绪层次脑机融合的感知与认知的计算规律，实现脑与机器之间的高效匹配。(2) This invention breaks through the limitations of the traditional brain-specific theory of cognition and emotion separation, proposes a new method of cognition and emotion perception and calculation based on the emotion-cognition integration framework, and explores the cognitive/emotional level of brain-computer fusion Computational rules of perception and cognition to achieve efficient matching between brain and machine.
(3)本发明建立我国特定人群的核心认知与情绪表征集，并建立多模态指标和认知与情绪评测目标之间的关联关系，在此基础上建立认知与情绪评测的关键指标，并据此通过机器学习的方法建立认知与情绪状态估计模型。(3) The present invention establishes the core cognition and emotion characterization set of specific groups of people in my country, and establishes the correlation between multimodal indicators and cognition and emotion evaluation targets, and establishes key indicators of cognition and emotion evaluation on this basis , and based on this, establish a cognitive and emotional state estimation model through machine learning.
(4)本发明提出了一种新的多元数据分析方法，分析获得多模态指标与识别目标之间的关联关系，在此基础上建立认知与情绪评测的关键生理及行为指标。(4) The present invention proposes a new multivariate data analysis method, analyzes and obtains the correlation between multimodal indicators and recognition targets, and establishes key physiological and behavioral indicators for cognitive and emotional evaluation on this basis.
(5)目前国内外认知与情绪状态分析与识别研究主要集中在对6种基本情感(高兴、悲伤、愤怒、恐惧、惊奇、厌恶)的识别及少量认知状态的识别，本发明实现了对复杂认知与情绪状态的感知与识别。(5) At present, domestic and foreign cognition and emotional state analysis and recognition research mainly focus on the recognition of six basic emotions (happy, sad, angry, fear, surprise, disgust) and a small amount of cognitive states. The present invention realizes Perception and recognition of complex cognitive and emotional states.
(6)本发明突破传统单模态认知与情绪状态感知技术存在的局限，采用生理-行为多模态感知与融合技术，实现对认知与情绪状态的实时感知与估计。(6) The present invention breaks through the limitations of traditional single-modal cognition and emotional state perception technology, and adopts physiological-behavioral multi-modal perception and fusion technology to realize real-time perception and estimation of cognition and emotional state.
附图说明Description of drawings
图1为本发明的整体组成框图；Fig. 1 is the overall composition block diagram of the present invention;
图2为本发明中的便携式生理信号采集与分析系统的组成框图；Fig. 2 is the composition block diagram of portable physiological signal acquisition and analysis system among the present invention;
图3为本发明中的基于视频分析的行为识别系统的组成框图；Fig. 3 is the composition block diagram of the behavior recognition system based on video analysis among the present invention;
图4为本发明中的无线网关的组成框图；Fig. 4 is the composition block diagram of wireless gateway among the present invention;
图5为本发明中的数据融合与反馈计算机系统的组成框图。Fig. 5 is a block diagram of the data fusion and feedback computer system in the present invention.
具体实施方式Detailed ways
下面对本发明技术方案结合附图进行详细说明。The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.
如图1所示，本发明的一种基于认知与情绪状态多模态感知的脑机接口系统，包括便携式生理信号采集与分析系统10、基于视频分析的行为识别系统20、无线网关30和数据融合与反馈计算机系统40；所述便携式生理信号采集与分析系统10以及基于视频分析的行为识别系统20均通过无线网关30分别与数据融合与反馈计算机系统40实现双向通讯连接；所述数据融合与反馈计算机系统40通过无线网关30向便携式生理信号采集与分析系统10以及基于视频分析的行为识别系统20同步发送数据采集指令，然后二者分别进行数据生理信号和行为数据的同步采集与实时分析，所述数据融合与反馈计算机系统40收到所述分析结果后，对所获得的生理数据和行为数据进行融合计算并得到认知与情绪状态的估计结果，所述估计结果将进一步反馈给执行机构从而实现脑与机器的双向交互接口。As shown in Figure 1, a brain-computer interface system based on multi-modal perception of cognitive and emotional states of the present invention includes a portable physiological signal acquisition and analysis system 10, a behavior recognition system 20 based on video analysis, a wireless gateway 30 and Data fusion and feedback computer system 40; the portable physiological signal acquisition and analysis system 10 and the behavior recognition system 20 based on video analysis all realize two-way communication connection with data fusion and feedback computer system 40 respectively through wireless gateway 30; the data fusion The feedback computer system 40 sends data collection instructions synchronously to the portable physiological signal collection and analysis system 10 and the behavior recognition system 20 based on video analysis through the wireless gateway 30, and then the two perform synchronous collection and real-time analysis of data physiological signals and behavior data respectively , after the data fusion and feedback computer system 40 receives the analysis results, it performs fusion calculations on the obtained physiological data and behavioral data and obtains estimation results of cognitive and emotional states, and the estimation results will be further fed back to the executive The mechanism thus realizes the two-way interactive interface between the brain and the machine.
如图2所示，本实施例中的便携式生理信号采集与分析系统10的整体封装于可头戴的外壳中，具体包括微处理器、无线收发模块、SD卡读写模块、耳夹式红外心率采集模块、双导脑电采集模块、电源调节与管理模块、锂电池、USB充电模块等模块，所述微处理器对存储于SD卡读写模块中的原始脑电信号和脉冲信号数据进行分析处理，并通过无线收发模块将分析处理结果实时发送至数据融合与反馈计算机系统40；通过与计算机的USB接口或者与带USB接口的AC-DC器件相连，USB充电模块能够实现对锂电池的充电功能；由于锂电池输出电压范围为3.7-4.2V，电源调节与管理模块可以对本系统所有模块单元提供稳定可靠的3.3V和5V电源；双导脑电采集模块包括干电极、三级放大电路、带通滤波器和A/D转换器等单元组成，两个干电极布置在双侧前额叶，而参考电极和地电极分别位布置在双耳耳垂；由干电极获得的脑电信号经过三级放大电路放大后再经过带通滤波器获得信噪比较高的信号，该信号再经过高精度A/D转换器处理后获得稳定的原始脑电信号；耳夹式红外心率采集模块由耳夹式红外传感器、放大电路、比较器、滤波器等单元组成，由耳夹式红外传感器获得的信号经过放大电路放大后再经过比较器和滤波器处理后获得与心率同步的脉冲信号，该脉冲信号直接与微处理器的I/O口相连，微处理器将通过现有公知技术计算产生心率变异性时域和频率指标；大脑活动节律的δ波、θ波、α波、β波、γ波功率谱、以及心率变异性时域和频率指标，将通过无线收发模块实时发送给无线网关30。As shown in Figure 2, the portable physiological signal acquisition and analysis system 10 in this embodiment is packaged in a head-mountable housing as a whole, specifically including a microprocessor, a wireless transceiver module, an SD card read-write module, an ear clip-type infrared Heart rate acquisition module, dual-lead EEG acquisition module, power regulation and management module, lithium battery, USB charging module and other modules, the microprocessor performs the original EEG signal and pulse signal data stored in the SD card read-write module Analyze and process, and send the analysis and process results to the data fusion and feedback computer system 40 in real time through the wireless transceiver module; by connecting with the USB interface of the computer or with the AC-DC device with the USB interface, the USB charging module can realize the charging of the lithium battery. Charging function; since the output voltage range of the lithium battery is 3.7-4.2V, the power regulation and management module can provide stable and reliable 3.3V and 5V power supplies for all module units of the system; the dual-conductor EEG acquisition module includes dry electrodes and three-stage amplifier circuits , band-pass filter and A/D converter and other units, two dry electrodes are arranged on the bilateral frontal lobes, while the reference electrode and the ground electrode are respectively arranged on the earlobes of both ears; the EEG signals obtained by the dry electrodes are passed through three After being amplified by a high-level amplifier circuit, a signal with a high signal-to-noise ratio is obtained through a band-pass filter, and the signal is processed by a high-precision A/D converter to obtain a stable original EEG signal; the ear-clip infrared heart rate acquisition module is controlled by the ear The clip-type infrared sensor, amplifier circuit, comparator, filter and other units are composed. The signal obtained by the ear clip-type infrared sensor is amplified by the amplifier circuit and then processed by the comparator and filter to obtain a pulse signal synchronized with the heart rate. The signal is directly connected to the I/O port of the microprocessor, and the microprocessor will calculate and generate the heart rate variability time domain and frequency indicators through the existing known technology; the delta wave, theta wave, alpha wave, beta wave, gamma wave of the brain activity rhythm The wave power spectrum, and heart rate variability time domain and frequency indicators will be sent to the wireless gateway 30 in real time through the wireless transceiver module.
从上述原始脑电信号中可提取出反映大脑活动节律的脑电波功率谱，其中脑电波功率谱包括δ波(1-3Hz)、θ波(4-7Hz)、α波(8-13Hz)、β波(14-25Hz)和γ波(25Hz以上)；且从脉冲信号中可获得产生心率变异性时域和频率指标。The brain wave power spectrum reflecting the rhythm of brain activity can be extracted from the above-mentioned original EEG signal, wherein the brain wave power spectrum includes delta wave (1-3Hz), theta wave (4-7Hz), alpha wave (8-13Hz), β wave (14-25Hz) and γ wave (above 25Hz); and the heart rate variability time domain and frequency index can be obtained from the pulse signal.
如图3所示，在本实施例中，基于视频分析的行为识别系统20包括嵌入式平台、摄像机、无线收发模块、SD卡读写模块、液晶显示模块、电源调节与管理模块、锂电池、USB充电模块等模块单元；所述嵌入式平台通过摄像机实时记录和分析用户的行为和面部表情并存储于SD卡读写模块，同时通过无线收发模块实时将行为和面部表情的数据发送至数据融合与反馈计算机系统40。As shown in Figure 3, in the present embodiment, the behavior recognition system 20 based on video analysis includes an embedded platform, a video camera, a wireless transceiver module, an SD card read-write module, a liquid crystal display module, a power regulation and management module, a lithium battery, Module units such as a USB charging module; the embedded platform records and analyzes the user's behavior and facial expressions in real time through the camera and stores them in the SD card reading and writing module, and simultaneously sends the data of the behavior and facial expressions to the data fusion module in real time through the wireless transceiver module and feedback computer system 40 .
其中，通过与计算机USB接口或者与带USB接口的AC-DC器件相连，USB充电模块实现对锂电池的充电功能；由于锂电池输出电压范围为3.7-4.2V，电源调节与管理模块可以对本系统所有功能单元提供稳定可靠的3.3V和5V电源。Among them, by connecting with the computer USB interface or with the AC-DC device with USB interface, the USB charging module realizes the charging function of the lithium battery; since the output voltage range of the lithium battery is 3.7-4.2V, the power regulation and management module can control the power of the system. All functional units provide stable and reliable 3.3V and 5V power supplies.
如图4所示，在本实施例中，无线网关30由微处理器、无线收发模块、串口转USB接口模块、电源调节与管理模块等组成；电源调节与管理模块获得稳定的3.3V电源供无线网关30系统使用；微处理器通过串口转USB接口模块与数据融合与反馈计算机系统40的USB接口相连，并采用无线组网技术与便携式生理信号采集与分析系统10以及基于视频分析的行为识别系统20实现双向无线通讯。As shown in Figure 4, in this embodiment, the wireless gateway 30 is composed of a microprocessor, a wireless transceiver module, a serial port to USB interface module, a power regulation and management module, etc.; the power regulation and management module obtains a stable 3.3V power supply The wireless gateway 30 system is used; the microprocessor is connected to the USB interface of the data fusion and feedback computer system 40 through a serial port to USB interface module, and adopts wireless networking technology and a portable physiological signal acquisition and analysis system 10 and behavior recognition based on video analysis System 20 enables two-way wireless communication.
如图5所示，在本实施例中，数据融合与反馈计算机系统40包括计算机、第一USB接口、第二USB接口、执行机构和电源调节与管理模块，所述计算机通过第一USB接口与无线网关30相连，然后通过无线网关30将采样指令同步发送给便携式生理信号采集与分析系统10和基于视频分析的行为识别系统20，实现生理信号和行为信号同步采集，并同时通过无线网关30接受便携式生理信号采集与分析系统10以及基于视频分析的行为识别系统20的计算结果，该计算结果经过数据融合算法通过第二USB接口生成执行机构的控制信号来驱动执行机构实现脑与机器的双向交互接口；其中电源调节与管理模块将为计算机以及执行机构提供所需稳定电源。As shown in Figure 5, in this embodiment, the data fusion and feedback computer system 40 includes a computer, a first USB interface, a second USB interface, an actuator and a power regulation and management module, and the computer communicates with the computer through the first USB interface. The wireless gateway 30 is connected, and then the sampling instruction is synchronously sent to the portable physiological signal collection and analysis system 10 and the behavior recognition system 20 based on video analysis through the wireless gateway 30, so as to realize the synchronous collection of physiological signals and behavior signals, and at the same time through the wireless gateway 30 Accept The calculation results of the portable physiological signal collection and analysis system 10 and the behavior recognition system 20 based on video analysis, the calculation results are generated through the data fusion algorithm through the second USB interface to generate the control signal of the actuator to drive the actuator to realize the two-way interaction between the brain and the machine interface; among them, the power adjustment and management module will provide the required stable power for the computer and the actuator.
上述的所述数据融合算法通过分析多模态指标与识别目标之间的关联关系获得认知与情绪状态识别的关键生理及行为指标。The aforementioned data fusion algorithm obtains key physiological and behavioral indicators for cognitive and emotional state identification by analyzing the correlation between multimodal indicators and identification targets.
综上所述，本发明能够实时获取对象的生理-行为多模态数据，借助多元数据分析方法分析获得多模态指标与识别目标之间的关联关系，在此基础上建立认知与情绪评测的关键生理及行为指标；且本发明将获取的认知与情绪状态的实时感知与识别结果进行实时反馈控制，在脑与机器之间建立直接双向信息通道和认知/情绪层次交互接口。To sum up, the present invention can acquire the physiological-behavior multimodal data of the subject in real time, analyze and obtain the correlation between the multimodal index and the recognition target by means of the multivariate data analysis method, and establish cognition and emotion evaluation on this basis The key physiological and behavioral indicators; and the present invention performs real-time feedback control on the obtained real-time perception and recognition results of cognitive and emotional states, and establishes a direct two-way information channel and a cognitive/emotional level interaction interface between the brain and the machine.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410547000.3A CN104391569A (en) | 2014-10-15 | 2014-10-15 | Brain-machine interface system based on cognition and emotional state multi-mode perception |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410547000.3A CN104391569A (en) | 2014-10-15 | 2014-10-15 | Brain-machine interface system based on cognition and emotional state multi-mode perception |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104391569A true CN104391569A (en) | 2015-03-04 |
Family
ID=52609481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410547000.3A Pending CN104391569A (en) | 2014-10-15 | 2014-10-15 | Brain-machine interface system based on cognition and emotional state multi-mode perception |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104391569A (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105054928A (en) * | 2015-07-17 | 2015-11-18 | 张洪振 | Emotion display equipment based on BCI (brain-computer interface) device electroencephalogram acquisition and analysis |
| CN105279387A (en) * | 2015-11-17 | 2016-01-27 | 东南大学 | Execution function evaluating and training system for autism spectrum disorder children |
| CN105342569A (en) * | 2015-11-25 | 2016-02-24 | 新乡医学院 | Mental state detection system based on electroencephalogram analysis |
| CN105677020A (en) * | 2015-12-23 | 2016-06-15 | 黄淮学院 | Electronic control device |
| CN105955486A (en) * | 2016-05-16 | 2016-09-21 | 西北工业大学 | Method for assisting teleoperation based on visual stimulation of brainwaves |
| CN106175799A (en) * | 2015-04-30 | 2016-12-07 | 深圳市前海览岳科技有限公司 | Based on brain wave assessment human body emotion and the method and system of fatigue state |
| CN106569346A (en) * | 2016-11-07 | 2017-04-19 | 北京理工大学 | Lenses and Sunglasses |
| CN106970712A (en) * | 2017-05-02 | 2017-07-21 | 臧大维 | A kind of multi-functional people's brain computer interfaces helmet |
| CN107137096A (en) * | 2017-06-22 | 2017-09-08 | 中国科学院心理研究所 | A kind of multi-modal physiology and behavioral data merge acquisition system |
| CN107463874A (en) * | 2017-07-03 | 2017-12-12 | 华南师范大学 | The intelligent safeguard system of Emotion identification method and system and application this method |
| CN109310353A (en) * | 2016-06-06 | 2019-02-05 | 微软技术许可有限责任公司 | Information is conveyed via computer implemented agency |
| CN109471528A (en) * | 2018-10-19 | 2019-03-15 | 天津大学 | A brain-computer interaction system for brain-computer interface system |
| CN109782917A (en) * | 2019-01-21 | 2019-05-21 | 中国联合网络通信集团有限公司 | A kind of consciousness industrial control system and method based on brain-computer interface |
| CN109846582A (en) * | 2019-03-20 | 2019-06-07 | 上海交通大学 | An Electrical Stimulation System Based on Multimodal Perceptual Feedback |
| CN109875579A (en) * | 2019-02-28 | 2019-06-14 | 京东方科技集团股份有限公司 | Emotional health management system and emotional health management method |
| CN112120716A (en) * | 2020-09-02 | 2020-12-25 | 中国人民解放军军事科学院国防科技创新研究院 | Wearable multi-mode emotional state monitoring device |
| US11249951B2 (en) | 2015-07-13 | 2022-02-15 | Samsung Electronics Co., Ltd. | Heuristic interface for enabling a computer device to utilize data property-based data placement inside a nonvolatile memory device |
| CN114237388A (en) * | 2021-12-01 | 2022-03-25 | 辽宁科技大学 | Brain-computer interface method based on multi-mode signal recognition |
| US11461010B2 (en) | 2015-07-13 | 2022-10-04 | Samsung Electronics Co., Ltd. | Data property-based data placement in a nonvolatile memory device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080255949A1 (en) * | 2007-04-13 | 2008-10-16 | Lucid Systems, Inc. | Method and System for Measuring Non-Verbal and Pre-Conscious Responses to External Stimuli |
| CN101872171A (en) * | 2009-04-24 | 2010-10-27 | 中国农业大学 | Driver Fatigue State Recognition Method and System Based on Information Fusion |
| CN103818256A (en) * | 2012-11-16 | 2014-05-28 | 西安众智惠泽光电科技有限公司 | Automobile fatigue-driving real-time alert system |
-
2014
- 2014-10-15 CN CN201410547000.3A patent/CN104391569A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080255949A1 (en) * | 2007-04-13 | 2008-10-16 | Lucid Systems, Inc. | Method and System for Measuring Non-Verbal and Pre-Conscious Responses to External Stimuli |
| CN101872171A (en) * | 2009-04-24 | 2010-10-27 | 中国农业大学 | Driver Fatigue State Recognition Method and System Based on Information Fusion |
| CN103818256A (en) * | 2012-11-16 | 2014-05-28 | 西安众智惠泽光电科技有限公司 | Automobile fatigue-driving real-time alert system |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106175799A (en) * | 2015-04-30 | 2016-12-07 | 深圳市前海览岳科技有限公司 | Based on brain wave assessment human body emotion and the method and system of fatigue state |
| US11989160B2 (en) | 2015-07-13 | 2024-05-21 | Samsung Electronics Co., Ltd. | Heuristic interface for enabling a computer device to utilize data property-based data placement inside a nonvolatile memory device |
| US11461010B2 (en) | 2015-07-13 | 2022-10-04 | Samsung Electronics Co., Ltd. | Data property-based data placement in a nonvolatile memory device |
| US11249951B2 (en) | 2015-07-13 | 2022-02-15 | Samsung Electronics Co., Ltd. | Heuristic interface for enabling a computer device to utilize data property-based data placement inside a nonvolatile memory device |
| CN105054928A (en) * | 2015-07-17 | 2015-11-18 | 张洪振 | Emotion display equipment based on BCI (brain-computer interface) device electroencephalogram acquisition and analysis |
| CN105279387A (en) * | 2015-11-17 | 2016-01-27 | 东南大学 | Execution function evaluating and training system for autism spectrum disorder children |
| CN105279387B (en) * | 2015-11-17 | 2018-08-21 | 东南大学 | A kind of evaluation and test of execution function and training system towards autism-spectrum disorder with children |
| CN105342569A (en) * | 2015-11-25 | 2016-02-24 | 新乡医学院 | Mental state detection system based on electroencephalogram analysis |
| CN105677020A (en) * | 2015-12-23 | 2016-06-15 | 黄淮学院 | Electronic control device |
| CN105955486A (en) * | 2016-05-16 | 2016-09-21 | 西北工业大学 | Method for assisting teleoperation based on visual stimulation of brainwaves |
| CN109310353A (en) * | 2016-06-06 | 2019-02-05 | 微软技术许可有限责任公司 | Information is conveyed via computer implemented agency |
| CN106569346A (en) * | 2016-11-07 | 2017-04-19 | 北京理工大学 | Lenses and Sunglasses |
| CN106970712A (en) * | 2017-05-02 | 2017-07-21 | 臧大维 | A kind of multi-functional people's brain computer interfaces helmet |
| CN106970712B (en) * | 2017-05-02 | 2023-06-16 | 臧大维 | Multifunctional human brain computer interface helmet |
| CN107137096A (en) * | 2017-06-22 | 2017-09-08 | 中国科学院心理研究所 | A kind of multi-modal physiology and behavioral data merge acquisition system |
| CN107463874A (en) * | 2017-07-03 | 2017-12-12 | 华南师范大学 | The intelligent safeguard system of Emotion identification method and system and application this method |
| CN109471528A (en) * | 2018-10-19 | 2019-03-15 | 天津大学 | A brain-computer interaction system for brain-computer interface system |
| CN109782917B (en) * | 2019-01-21 | 2022-03-25 | 中国联合网络通信集团有限公司 | Consciousness industrial control system and method based on brain-computer interface |
| CN109782917A (en) * | 2019-01-21 | 2019-05-21 | 中国联合网络通信集团有限公司 | A kind of consciousness industrial control system and method based on brain-computer interface |
| CN109875579A (en) * | 2019-02-28 | 2019-06-14 | 京东方科技集团股份有限公司 | Emotional health management system and emotional health management method |
| CN109846582B (en) * | 2019-03-20 | 2021-05-28 | 上海交通大学 | An Electrical Stimulation System Based on Multimodal Perceptual Feedback |
| CN109846582A (en) * | 2019-03-20 | 2019-06-07 | 上海交通大学 | An Electrical Stimulation System Based on Multimodal Perceptual Feedback |
| CN112120716A (en) * | 2020-09-02 | 2020-12-25 | 中国人民解放军军事科学院国防科技创新研究院 | Wearable multi-mode emotional state monitoring device |
| CN114237388A (en) * | 2021-12-01 | 2022-03-25 | 辽宁科技大学 | Brain-computer interface method based on multi-mode signal recognition |
| CN114237388B (en) * | 2021-12-01 | 2023-08-08 | 辽宁科技大学 | A brain-computer interface method based on multimodal signal recognition |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104391569A (en) | Brain-machine interface system based on cognition and emotional state multi-mode perception | |
| CN105279387B (en) | A kind of evaluation and test of execution function and training system towards autism-spectrum disorder with children | |
| CN101711709B (en) | Electric artificial hand control method using electro-oculogram and electro-encephalic information | |
| CN103885445B (en) | The brain control method of a kind of brain control animal robot system and animal robot | |
| CN101897640B (en) | Novel movement imagery electroencephalogram control-based intelligent wheelchair system | |
| CN104548347A (en) | Pure idea nerve muscle electrical stimulation control and nerve function evaluation system | |
| WO2020118797A1 (en) | Prosthesis control method, apparatus, system and device, and storage medium | |
| CN104605849A (en) | Electroencephalogram biological feedback diagnosis and treatment instrument and system and using method | |
| CN105022486A (en) | Electroencephalogram identification method based on different expression drivers | |
| Yang et al. | Wearable 3.0: from smart clothing to wearable affective robot | |
| CN107168346A (en) | A kind of asynchronous system brain control UAS based on wearable display | |
| CN112008725B (en) | Human-computer fusion brain-controlled robot system | |
| CN105137830B (en) | A kind of the traditional Chinese Painting mechanical hand and its drawing practice of view-based access control model inducting brain-machine interface | |
| CN111584031B (en) | Brain-controlled intelligent limb rehabilitation system based on portable electroencephalogram acquisition equipment and application | |
| CN106951064A (en) | Introduce the design of stable state vision inducting normal form and discrimination method of object continuous action | |
| CN103169471A (en) | Portable electroencephalogram detection system | |
| CN109009887A (en) | A kind of man-machine interactive navigation system and method based on brain-computer interface | |
| CN107066091A (en) | A kind of brain-computer interface P300 signal characteristic body-sensing inducing devices and its application process | |
| Dai et al. | [Retracted] Application Analysis of Wearable Technology and Equipment Based on Artificial Intelligence in Volleyball | |
| Savadi et al. | Face based automatic human emotion recognition | |
| CN108960023A (en) | A kind of portable Emotion identification device | |
| CN106476281B (en) | Based on blink identification and vision induced 3D printer control method | |
| CN114652328A (en) | Bidirectional closed-loop brain-computer interaction electro-acupuncture parameter intelligent matching system for insomnia | |
| CN109901711B (en) | Asynchronous real-time brain control method driven by weak myoelectricity artifact micro-expression electroencephalogram signals | |
| CN110751060A (en) | Portable motion mode real-time identification system based on multi-source signals |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150304 |