WO2023079729A1 - Biological examination apparatus, biological examination method, and program - Google Patents

Abstract

Provided are a biological examination apparatus capable of efficiently performing an examination concerning deglutition, a biological examination method, and a program.　An information processing apparatus 109 in a biological examination apparatus 100 according to the present invention executes: a process of causing a display 110 to concurrently display a graph based on on detected data acquired by detection units 102, 103 for detecting movement of a thyroid cartilage, the graph indicating movement of a thyroid cartilage when a subject swallows, and a video image wherein the subject when swallowing is captured, the video image being captured by a camera 114; and a process of causing the display 110 to display, on the basis of operation designating an arbitrary point in one of the graph or the video image by an operator in a state wherein the graph and the video image are concurrently displayed, a display wherein a point corresponding to the arbitrary point in the other of the graph or the video image can be grasped.

Description

Biopsy apparatus, biopsy method and program

The present invention relates to a biopsy apparatus, a biopsy method, and a program for performing a test related to swallowing of a living body.

Pneumonia is known to be one of the major causes of death. Among them, aspiration pneumonia induced by dysphagia, which means a disorder related to swallowing, accounts for about 60% or more.

Stroke is the main cause of dysphagia, and it is known that 80% of patients in the acute phase develop dysphagia. It is also known that the percentage of people with dysphagia increases with age, even without a clear causative disease such as stroke. expected to increase.

Therefore, various tests have been attempted to diagnose dysphagia. For example, Videofluoroscopic Examination of Swallowing (VF) is generally known as a method for accurately evaluating and understanding dysphagia. In this VF, a bolus containing a contrast agent such as barium sulfate and an X-ray fluoroscope are used to monitor the movement of the bolus during swallowing and the behavior of the hyoid bone and larynx of the subject. In this case, the swallowing movement is a series of rapid movements and is generally recorded and evaluated on video. However, VF requires caution because it is an examination that has the potential for aspiration and suffocation, and because it requires a large-sized X-ray fluoroscope, exposure and time are limited. , and high cost. Also known is videoendoscopic examination of swallowing (VE), which uses an endoscope to evaluate dysphagia, but has the same problems as VF. In this way, clinical examinations such as VF and VE directly observe the movement of the throat, so they can be diagnosed accurately. It does not mean that

On the other hand, as a simple examination method for dysphagia, palpation (Repetitive Saliva Swallowing Test (RSST)), auscultation (cervical auscultation), observation (water drinking test and food test), or questionnaire Screening tests such as subjective evaluations by MRI are also known, but although they can be performed as routine tests, there are problems in that quantitative evaluation is difficult and reproducibility and objectivity are poor.

In view of the above problems, several methods have been proposed in recent years to share and record the swallowing state. For example, Patent Literature 1 discloses a device that attaches a microphone to the neck, saves voice data corresponding to auscultation as digital data, and detects swallowing by waveform analysis. In Patent Document 2, in addition to a microphone, a magnetic coil is attached to the neck, and in addition to voice data, motion data of the thyroid cartilage during swallowing, which corresponds to palpation, is stored as digital data, and an examination related to swallowing of a living body is performed. and a biopsy apparatus that displays the results. Specifically, this biopsy apparatus has a transmitting coil and a receiving coil arranged so as to sandwich the thyroid cartilage. The lateral displacement of the cartilage is measured as distance information between the coils. According to such an examination form, distance information and voice information corresponding to palpation and auscultation can be acquired noninvasively at the same time, so that the swallowing motion can be evaluated by combining the distance information and voice information.

JP 2013-017694 A JP-A-2009-213592

By the way, in the biopsy apparatus of Patent Document 2, the distance information and the voice information are displayed as a distance waveform and a voice waveform as time-series waveforms, and the swallowing state is evaluated based on these waveforms. However, in these evaluation forms based only on waveforms, there are cases where it is difficult to grasp the actual timing of swallowing (start timing, end timing, etc.) from waveforms, and it takes time to examine swallowing. I could have lost it.

The present invention has been made in view of the above circumstances, and aims to provide a biopsy apparatus, a biopsy method, and a program capable of efficiently performing a swallowing test.

In order to solve the above problems, the biopsy apparatus of the present invention includes:
A biological examination apparatus comprising an input device that receives an operation by an operator, a display device, and an information processing device,
The information processing device is
A graph showing the movement of the thyroid cartilage when the subject swallows based on the detection data acquired by the detection unit that detects the movement of the thyroid cartilage, and an image of the subject when the subject swallows taken by a camera. and a process of simultaneously displaying on the display device,
In a state in which the graph and the moving image are displayed at the same time, based on an operation by the operator to designate an arbitrary point on one of the graph or the moving image, the arbitrary point on the other of the graph or the moving image and a process of causing the display device to display a display that enables the user to grasp the points corresponding to .

In addition, the biopsy method of the present invention is
A biopsy method performed by a biopsy apparatus including an input device for receiving an operation by an operator, a display device, and an information processing device,
A graph showing the movement of the thyroid cartilage when the subject swallows based on the detection data acquired by the detection unit that detects the movement of the thyroid cartilage, and an image of the subject when the subject swallows taken by a camera. and a step of simultaneously displaying on the display device;
In a state in which the graph and the moving image are displayed at the same time, based on an operation by the operator to designate an arbitrary point on one of the graph or the moving image, the arbitrary point on the other of the graph or the moving image and a step of causing the display device to display a display that makes it possible to grasp the points corresponding to the.

Further, the program of the present invention is
A graph showing the movement of the thyroid cartilage when the subject swallows based on the detection data acquired by the detection unit that detects the movement of the thyroid cartilage, and an image of the subject when the subject swallows taken by a camera. and a process of simultaneously displaying on a display device,
In a state in which the graph and the moving image are displayed at the same time, based on an operation by the operator to designate an arbitrary point on one of the graph or the moving image, the arbitrary point on the other of the graph or the moving image and a process of causing the display device to display a display that enables the user to grasp the points corresponding to the above.

According to the present invention, the graph showing the movement of the thyroid cartilage and the moving image of the subject during swallowing are displayed on the same screen. The graph (detection data) can be checked while looking at it, and the swallowing test can be efficiently performed. Further, according to the present invention, based on the operation of designating an arbitrary point in the graph, a display is displayed that allows the point corresponding to the arbitrary point in the moving image to be grasped, or the arbitrary point in the moving image is displayed. At least one of displaying a display that makes it possible to grasp the point corresponding to the arbitrary point in the graph can be displayed based on the specified operation. That is, by specifying an arbitrary point on the graph, searching for a video frame in the video corresponding to that arbitrary point, or by specifying an arbitrary time (point) in the video, Finding the corresponding point on the graph becomes possible. Therefore, it becomes easy to perform work such as extracting a range in which analysis should be performed intensively in the graph, and it is possible to efficiently perform a swallowing examination. In addition, it becomes easy to grasp the relationship between each point on the graph and the actual swallowing motion shown in the moving image, and the examination of swallowing can be performed efficiently.

Further, in the configuration of the present invention, the graph includes a first coordinate axis corresponding to the forward and backward movement of the thyroid cartilage, a second coordinate axis corresponding to the vertical movement of the thyroid cartilage and intersecting with the first coordinate axis, may have

With such a configuration, it is easier to grasp how the actual swallowing action is from the graph itself. In addition, by linking such graphs and moving images, it becomes easier to understand the relationship between the measured data and the actual swallowing motion, and the examination of swallowing can be performed efficiently.

Further, in the configuration of the present invention, the graph may be a single graph in which the movement of the thyroid cartilage and the swallowing sound acquired by the microphone are temporally associated with each other.

According to such a configuration, the movement of the thyroid cartilage and the change in the swallowing sound can be integrated and visualized in one graph, so that the swallowing dynamics such as the timing of the swallowing motion and the swallowing sound can be visualized non-invasively. You can tell at a glance. In addition, by linking such graphs and moving images, it becomes easier to understand the relationship between the measured data and the actual swallowing motion, and the examination of swallowing can be performed efficiently.

According to the present invention, examinations related to swallowing can be performed efficiently.

It is a functional block diagram showing an example of composition of a biopsy apparatus concerning one embodiment of the present invention. FIG. 4 is a schematic perspective view showing an example of a flexible holder that holds the laryngeal displacement detector of the biopsy apparatus. It is a functional block diagram which shows an example of a structure of the computer of a biopsy apparatus. 4 is a flow chart showing an example of processing of a motion analysis unit of a processing unit of a computer; 4 is a flow chart showing an example of processing of a speech analysis unit of a processing unit of a computer; 4 is a flow chart showing an example of processing of an analysis unit of a processing unit of a computer; FIG. 4 is a diagram showing an example of a distance waveform based on typical distance information detected by a laryngeal displacement detection unit of a biopsy apparatus; (a) is a diagram showing an example of distance information based on detection data detected by a laryngeal displacement detection unit of a biopsy apparatus and an operating waveform (fitting waveform) fitted from the distance information; ) is a diagram showing an example of component waveforms individually showing temporal behavior trajectories of the thyroid cartilage in the vertical direction and the anteroposterior direction. FIG. 4 is a diagram showing an example of a swallowing sound waveform based on typical voice information detected by the swallowing sound detection unit of the biopsy apparatus; It is a figure which shows an example of the locus|trajectory graph displayed based on the two-dimensional locus|trajectory data obtained by the process part of a biopsy apparatus. It is a figure which shows an example of a display by a display apparatus. It is a figure which shows an example of a display by a display apparatus. It is a figure which shows an example of a display by a display apparatus. It is a figure which shows an example of a display by a display apparatus.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. This embodiment contributes to the development of medical care and the realization of a healthy society with highly advanced technology by providing the technology described below. By realizing this inspection device and inspection method, we will contribute to "9. Build a foundation for industry and technological innovation" in the Sustainable Development Goals (SDGs) advocated by the United Nations.
FIG. 1 is a functional block diagram showing a configuration example of a biopsy apparatus 100 according to this embodiment. As shown in the figure, the biopsy apparatus 100 measures the larynx (thyroid gland) of a subject 101 caused by vertical and anteroposterior behavior of the thyroid cartilage (commonly known as the larynx) when the subject (subject) 101 swallows. A transmitter coil 102 and a receiver coil 103 as a laryngeal displacement detector that detects a change in the distance between two positions in the body part surrounding the cartilage), and a swallowing sound that detects the swallowing sound when the subject 101 swallows. and a microphone 106 as a detector, these coils 102, 103 and microphone 106 are held by a flexible holder 113 which will be described later with reference to FIG.

The transmission coil 102 and the reception coil 103 are arranged facing each other so as to sandwich the thyroid cartilage from both sides, the transmission coil 102 is connected to the transmitter 104 and the reception coil 103 is connected to the receiver 105 . Further, the microphone 106 is placed near the thyroid cartilage of the subject 101 and is electrically connected to a detection circuit 107 for detecting swallowing sounds captured by the microphone 106 during swallowing, and power is supplied from the detection circuit 107. etc., and operates. The microphone 106 is preferably a microphone using, for example, a piezo element (piezoelectric element) so as not to pick up ambient sounds other than swallowing sounds as much as possible, but may be a condenser type microphone or the like.

Moreover, the biopsy apparatus 100 further has a control device 108 , a computer 109 , a display device 110 , an external storage device 111 , an input device 112 and a camera 114 . The control device 108 controls the operations of the transmitter 104, the receiver 105, the detection circuit 107, the computer 109, the camera 114, and the external storage device 111, and controls power supply, signal transmission/reception timing, and the like. Further, the computer 109 is an information processing device including a CPU, a memory, an internal storage device, etc., and performs various arithmetic processing. The control and calculations performed by the computer 109 are realized by the CPU executing a predetermined program. However, some of the calculations are based on ASICs (Application Specific Integrated Circuits) and FPGAs (Field
It can also be implemented by hardware such as a programmable gate array). A display device 110 , an external storage device 111 and an input device 112 are electrically connected to the computer 109 .

Also, the camera 114 photographs the subject 101 . Specifically, the camera 114 captures the state of the subject 101 while the swallowing test (measurement) is being performed by the coils 102 and 103 and the microphone 106, and acquires a moving image 950 described later (FIGS. 11 to 11). See Figure 14). More specifically, the camera 114 captures movement of the throat of the subject 101 during the examination. Also, the movement of the mouth of the subject 101 during the examination may be photographed by the camera 114 .

In addition, the display device 110 is an interface that displays the measured waveform, analysis information by the computer 109, and the like. The display device 110 may be, for example, a liquid crystal display, an EL display, a plasma display, a CRT display, a projector, or the like, but is not limited to these. Moreover, the display device 110 may be mounted on a tablet terminal, a head-mounted display, a wearable device, or the like. Note that a specific function may be notified by an LED, sound, or the like. In addition to the internal storage device, the external storage device 111 stores data used for various arithmetic processing executed by the computer 109, data obtained by the arithmetic processing, images acquired (photographed) by the camera 114 (video 950), It holds conditions, parameters, and the like that are input via the input device 112 . The input device 112 is an interface for an operator to input conditions and the like necessary for measurement and arithmetic processing performed in the present embodiment.

In such a configuration, a high frequency magnetic field is emitted from the transmission coil 102 by transmitting a high frequency signal generated by the transmitter 104 to the transmission coil 102, and a signal received by the reception coil 103 is transmitted to the receiver 105. you will be able to receive it. Also, the signal received by the receiver 105 is sent to the computer 109 as an output voltage measurement value of the inter-coil voltage. On the other hand, the swallowing sound captured by the microphone 106 is detected by the detection circuit 107 and converted into a voltage signal, which is input from the detection circuit 107 to the computer 109 as an output voltage measurement value.

FIG. 2 shows a flexible holder 113 that holds the transmitting and receiving coils 102, 103 and the microphone 106. FIG. The flexible holder 113 is made of any flexible material such as various resins, and is attached to the neck of the subject 101 using its open end as shown in the figure. and a pair of arc-shaped sensor holding members 203a and 203b positioned inside the neck-mounted member 202 along substantially the same arc. One ends of a pair of sensor holding members 203a and 203b are integrally connected to each other so as to hold one ends of the pair of sensor holding members 203a and 203b on the inside thereof, and the other ends of the sensor holding members 203a and 203b are opened and positioned near the larynx of the subject 101. It seems to be Sensor portions 204a and 204b are arranged at the other ends of the pair of sensor holding members 203a and 203b, respectively. Together with the sensor holding members 203a and 203b positioned without contacting the neck of the body 101, the movement of swallowing (the movement of the thyroid cartilage, etc.) can be followed independently of the neck mounting member 202.

The transmission coil 102 is fixedly arranged inside one of the sensor portions 204a and 204b, and the reception coil 103 is fixedly arranged inside the other of the sensor portions 204a and 204b. It is arranged in Particularly in this embodiment, the transmitting coil 102 and the receiving coil 103 are attached to the sensor sections 204a and 204b so as to be arranged in directions that are likely to face each other (close to the vertical direction of the neck surface of the subject 101). , thereby enabling detection with a high signal-to-noise (SN) ratio. Therefore, the microphone 106 and the transmission coil 102 or the reception coil 103 can be arranged at positions substantially perpendicular to each other, and magnetic field noise generated from the microphone 106 can be reduced from entering the transmission and/or reception coils 102 and 103. can be done. However, the corresponding positions of the transmitting coil 102 and the receiving coil 103 and the positions orthogonal to the microphones are not limited to the described arrangement, and may be any position that can realize detection with a sufficiently high SN ratio.

In addition, a pressing portion 205 a to be applied to the neck of the subject 101 is provided at the opposing end portion (the portion of the neck attachment member 202 positioned on the back side of the neck of the subject 101 ) forming the open end of the neck attachment member 202 . , 205b are formed in a shape suitable for pressing, such as cylindrical or spherical. The neck size of the subject 101 is determined by four pressing points, which are the two pressing portions 205a and 205b and the two sensor portions 204a and 204b provided at the other ends of the sensor holding members 203a and 203b. This allows the flexible retainer 113 to be easily worn around the neck without any need. Electrical wires 201a and 201b extending from transmitting/receiving coils 102 and 103 incorporated in sensor units 204a and 204b and microphone 106 are electrically connected to transmitter 104, receiver 105 and detection circuit 107 shown in FIG. connected

A functional block diagram of the computer 109 is shown in FIG. As illustrated, the calculator 109 includes a swallowing measurement unit 410 , an image acquisition unit 415 , a processing unit 420 and a display unit 430 . The swallowing measurement unit 410 uses the transmitting coil 102, the receiving coil 103, the transmitter 104, the receiver 105, the microphone 106, the detection circuit 107, and the control device 108 described with reference to FIG. Sounds are measured (larynx displacement detection step and swallowing sound detection step). In addition, the image acquisition unit 415 acquires a moving image 950 of the subject 101 during swallowing motion and swallowing sound measurement using the camera 114 (moving image acquisition step). Also, the image acquisition unit 415 stores the acquired moving image 950 (moving image data) in the internal storage device of the computer 109 and/or the external storage device 111 (that is, memory). Further, the processing unit 420 includes a motion analysis unit 421 that analyzes distance information, a voice analysis unit 422 that analyzes swallowing sounds that are voice information, and an analysis unit 423 that analyzes a combination of the distance information and the swallowing sounds. The data measured by the swallowing measurement unit 410 is processed by these (processing step). Specifically, as will be described later, processing unit 420 applies a model function (in the present embodiment, equation (1) described later) that models the swallowing motion to detection data detected by transmission/reception coils 102 and 103. Based on the distance information (in the present embodiment, data indicating changes over time in the distance between the coils 102 and 103 arranged so as to sandwich the thyroid cartilage of the subject 101 (distance waveform shown in FIG. 7 to be described later) 701)) is fitted to (in the present embodiment, a fitted waveform 1103 shown in (a) of FIG. 8 to be described later) is obtained, and from this fitting result, an anterior-posterior movement associated with the anteroposterior movement of the thyroid cartilage is obtained. A dynamic component (in this embodiment, a back-and-forth dynamic component waveform 1105 shown in (b) of FIG. 8 to be described later or data values forming it) and a vertical movement component associated with vertical movement of the thyroid cartilage (this embodiment Then, the vertical motion component waveform 1106 shown in (b) of FIG. 8 (to be described later) or the data values forming it) is extracted, and the vertical motion component of the thyroid cartilage is extracted based on these extracted vertical motion components and longitudinal motion components. Two-dimensional trajectory data (in the present embodiment, data for forming a trajectory graph 901 shown in FIG. 10, which will be described later) representing behavior trajectories in a direction and a longitudinal direction are generated. In addition, as will be described later, the processing unit 420 generates a swallowing sound waveform (in the present embodiment, shown in FIG. 9 to be described later) that indicates a temporal change in the amplitude of the swallowing sound based on detection data detected by the microphone 106. In addition to generating the swallowing sound waveform 801), the swallowing sound waveform is temporally associated with the trajectory graph, and the plot of each trajectory data value on the trajectory graph is identified and displayed according to the magnitude of the amplitude of the swallowing sound. to generate identification data for In addition, the display unit 430 causes the display device 110 to display information (data) measured and processed by the swallowing measurement unit 410 and the processing unit 420 and moving images (moving image data) acquired by the image acquisition unit 415 (display step). Note that the swallowing measurement unit 410, the processing unit 420, and the display unit 430 operate independently.

FIG. 4 shows the processing flow of the motion analysis unit 421 of the processing unit 420 of the computer 109 of FIG. The motion analysis unit 421 processes the detection data detected by the transmission/reception coils 102 and 103. Specifically, first, in step S501, the data measured by the swallowing measurement unit 410 is smoothed. make a change. In particular, in the present embodiment, smoothing is performed using piecewise polynomial approximation using a Savitzky-Golay filter. The smoothing in this case is performed by setting the number of windows and the degree of the polynomial to, for example, 5, 51, etc., respectively. Note that the smoothing method may be, for example, a simple moving average, and the present invention is not limited by these.

Subsequently, in step S502, fitting is performed on the measurement signal smoothed in step S501. In this regard, FIG. 7 shows a typical example of a range waveform 701 showing the variation over time of the distance between transmit and receive coils 102, 103, which is the distance between two locations in the larynx of subject 101. . Such a measured distance waveform 701 is the result of one-dimensional (horizontal) observation of the two-dimensional movement (forward and backward movement and vertical movement) of the thyroid cartilage (hyoid bone). Therefore, it exhibits a W-shaped waveform as shown in the figure. Specifically, the thyroid cartilage is lifted as the bolus is sent into the esophagus from the start point (time T0) 702 where the subject 101 starts swallowing the bolus in the mouth, thereby causing the transmission/reception coil to The distance between 102 and 103 narrows from D0 to D1 and the distance waveform 701 reaches a first trough (first lower peak value; time T1) 703. FIG. In this bolus feeding process, the epiglottis of the subject 101 moves downward to block the passage from the nasal cavity to the respiratory tract. Thereafter, when the bolus passes through the esophagus, the thyroid cartilage moves forward (in the direction in which the subject's face is facing) to open the esophagus, thereby increasing the distance between the transmitting and receiving coils 102 and 103 from D1 to D1. D2, distance waveform 701 transitions from first valley 703 to peak (upper limit peak value; time T2) 704 . Then, when the bolus has completely passed through the esophagus (epiglottis) and is sent into the stomach, the thyroid cartilage moves backward as the epiglottis moves upward, thereby increasing the distance between the transmitting and receiving coils 102 and 103. narrows from D2 to D3, and the distance waveform 701 transitions from peak 704 to second valley (second lower peak value; time T3) 705 . The thyroid cartilage then descends so that the epiglottis and thyroid cartilage return to their original positions, thereby increasing the distance between the transmit and receive coils 102, 103 from D3 to D4 and causing the distance waveform 701 to enter a second trough 705. to the end point (time T4) 706.

As can be seen from the above, in such a distance waveform 701, a downwardly convex waveform component is generated in a series of behaviors of the thyroid cartilage from ascending to descending. , an upwardly convex waveform component is generated. Therefore, in the present embodiment, the W-shaped distance waveform 701 is replaced with a gentle downwardly convex waveform 710 ((b )) and a sharp upwardly convex waveform 720 (corresponding to the front-back motion component waveform 1105 shown in FIG. 8B). It is modeled as shown in Equation (1).

Here, t is the time, y(t) is the measured distance waveform, rAP(t) is the component in the front-back direction, rHF(t) is the component in the vertical direction, d(t) is body movement, etc. Offset from the initial value caused by individual differences such as thickness), and e indicates measurement noise.

In the present embodiment, the longitudinal and vertical components rAP and rHF are modeled by a normal distribution, and the trend component d(t) is modeled by a linear equation, but these models may be autoregressive models or nonlinear models. , the present invention is not limited by these. Moreover, in such modeling of the present embodiment, each component is obtained by parameter fitting using a mathematical optimization technique. In this embodiment, parameter fitting is performed using the nonlinear least-squares method, but the present invention is not limited to this. Further, when performing parameter fitting, for example, a constraint may be set such that the variance value of rAP is smaller than the variance value of rHF.

The result of fitting the model function (formula (1)) using such a normal distribution is shown in (a) of FIG. In the figure, a waveform 1102 formed by data values represented by dots corresponds to the distance waveform 701 shown in FIG. It is an operating waveform (fitting waveform) that has been fitted. Here, the horizontal axis is time and the vertical axis is normalized amplitude based on the distance between the coils shown in FIG.

After completing the signal fitting step S502 as described above, parameters are extracted from the fitted model function in step S503. In the present embodiment, the anteroposterior and vertical behaviors of the thyroid cartilage are modeled by independent normal distributions. to extract The "amplitude" corresponds to the magnitude of the movement of the thyroid cartilage, the "average value" corresponds to the time when the movement occurred, and the "variance" corresponds to the duration of the movement.

In relation to this, FIG. 8(b) shows only the longitudinal and vertical components of the thyroid cartilage that are individually extracted from the operating waveform (fitted waveform) 1103 shown in FIG. 8(a). waveforms (upwardly convex front-back motion component waveform 1105 and downwardly convex vertical motion component waveform 1106) are shown. As described above, the processing unit 420 including the motion analysis unit 421 of the biopsy apparatus 100 of the present embodiment performs temporal movement of the thyroid cartilage in the vertical direction and the anteroposterior direction based on the vertical motion component and the longitudinal motion component. It is possible to generate two-dimensional trajectory data that individually indicates behavior trajectories.

After the component extraction step S503 is finished, in step S504, the feature points of the W-shaped waveform, that is, the distance waveform 701 in FIG. The feature points corresponding to the peak points 702 to 706 (data values of D0 to D4 and T0 to T4) are extracted. Specifically, in the present embodiment, since the measurement signal is modeled and separated into components as shown in Equation (1), feature points are easily extracted without considering noise and trend components. More specifically, as an example, T2 is taken as the average value of rAP, T1 and T3 are taken as the minimum values before and after T2, respectively, and T0 and T4 are the negative and positive values of the average rHF, respectively. Obtained as the time at the point advanced by the variance value in the direction of . D0 to D4 are obtained as values corresponding to times T0 to T4, respectively.

After such peak value detection step S504 is completed, in step S505, the waveforms, parameters, characteristic points, etc. calculated in steps S501 to S504 are stored in the internal storage device and/or external storage device of computer 109. 111. Note that each of steps S501 to S505 described above may be performed while the swallowing motion and swallowing sound are being measured by the swallowing measurement unit 410, or may be performed multiple times.

FIG. 5 shows the processing flow of the speech analysis unit 422 of the processing unit 420 of the computer 109 of FIG. As shown in the figure, in step S601, rectification processing is performed on audio information (generally, an audio signal including both positive and negative values) measured through the swallowing measurement unit 410 from the microphone 106 . Here, the rectification process means a process of taking an absolute value and converting a negative value into a positive value. FIG. 9 shows a swallowing sound waveform 801 obtained by rectifying typical voice information.

In step S602, the rectified signal obtained in step S601 is logarithmically transformed. This processing can reduce the influence of spike-like signals mixed in the swallowing sound.

In step S603, the logarithmically transformed signal obtained in step S602 is smoothed. Especially in the present embodiment, smoothing processing is performed using a moving average, and the window width of the moving average is set to 400 points. The present invention is not limited by this smoothing technique.

In step S604, exponential transformation is applied to the smoothed signal obtained in step S603. As a result, it is possible to obtain a waveform representing the envelope of the initially measured audio information. In FIG. 9, an envelope curve 802 obtained from such typical speech information (swallowing sound waveform 801) is indicated by a dashed line.

In step S605, the envelope signal obtained in step S604 is resampled. Specifically, in this embodiment, since the sampling frequencies of the voice information and the distance information in the swallowing measurement unit 410 shown in FIG. 3 are 4000 Hz and 100 Hz, respectively, the envelope signal is resampled to 1/40 is used to match the sampling frequency of the distance information.

In step S606, the maximum value as a feature point is obtained for the resampled envelope signal obtained in step S605. This is because the section where the maximum amplitude is obtained in the swallowing sound signal (the swallowing sound waveform 801) is considered to indicate the flow of the ingested material, and is an important feature of the swallowing sound. Therefore, in this step S606, the time S2 corresponding to the peak point 803 indicating the maximum amplitude with respect to the envelope curve 802 shown in FIG. 9 is obtained.

In step S607, the swallowing sound section of the resampled envelope signal obtained in step S605 is obtained. That is, in order to obtain the time interval Ts in which the swallowing sound occurs in the envelope 802, the times at both ends of the swallowing sound interval are obtained. Specifically, an amplitude threshold value 804 indicated by a dashed line in FIG. 9 is set, and a point crossing the threshold value 804 downward from the maximum value (peak point 803) obtained in step S606, that is, temporally Time points S1 and S3 corresponding to an early start point 805 and a temporally late end point 806, respectively, are acquired as feature points. Also, in the present embodiment, a value obtained by adding the normalized median absolute deviation to the median is used as the threshold value 804 . The method of setting the threshold 804 does not limit the present invention, and a value obtained by adding the standard deviation to the average value may be used.

Finally, in step S608, the waveforms and feature values calculated in steps S601 to S607 are stored in the internal storage device of the computer 109 and/or the external storage device 111. Note that each of steps S601 to S608 described above may be performed while the swallowing motion and swallowing sound are being measured by the swallowing measurement unit 410, or may be performed multiple times.

FIG. 6 shows the processing flow of the analysis unit 423 of the processing unit 420 of the computer 109 of FIG. As shown, in step S1001, the maximum longitudinal and vertical maximum displacements (maximum values) of the motion waveform 1103 (or the distance waveform 701), which is the fitted waveform, are calculated.

In step S1002, the signed curvature of each point on the trajectory graph 901 described in detail below with reference to FIG. 10 is calculated. In this step S1002, the time progress direction (transition direction) of the trajectory graph 901 is extracted, and the signed curvature at each point on the trajectory graph 901 is calculated in order to extract the point with the maximum displacement.

In step S1003, the sign is obtained for the signed curvature obtained in step S1002. Specifically, in the trajectory graph 901, the amplitude of curvature is maximized at the point farthest from the coordinate origin. get. By determining the sign of the coordinate system such that the counterclockwise rotation is positive and the clockwise rotation is negative, the time progress direction can be uniquely determined. The factor that determines whether the sign is positive or negative is the magnitude of the average value of the longitudinal component rAP and the vertical component rHF. It shows that the average value of the displacement in the direction (that is, the time to take the maximum value) is faster than that in the vertical direction.

In step S1004, the geometric distance from the coordinate origin of the point at which the maximum signed curvature calculated in step S1002 is obtained is obtained. In the trajectory graph 901, since the amplitude of curvature is maximum at the point farthest from the origin of coordinates, the geometric distance from the point where the amplitude of curvature is maximum to the origin of coordinates is calculated. This makes it possible to acquire the time point (time) at which the displacement is the largest when the vertical and longitudinal components of the thyroid cartilage are synthesized.

In step S1005, the time difference between the time at which the maximum value of the voice information is obtained and the time at which the maximum value of the distance information is obtained in the front-rear direction is acquired. This is especially because the time difference between the maximum values is an important parameter in characterizing the swallowing state. In the present embodiment, as can be seen from the display form of the trajectory graph 901, which will be described later, this parameter can not only be grasped visually, but can also be displayed as a quantitative value. The present invention is not limited by these quantitative values, and for example, the area of the region surrounded by the trajectory graph may be displayed as a feature amount.

In step S1006, the ratio (variance value time difference). In the healthy human model, the swallowing sound is generated at the timing when the thyroid cartilage advances, so in this step S1006, the ratio is calculated in order to display how much the swallowing sound generation deviation is within the individual.

Finally, in step S1007, the waveforms and feature values calculated in steps S1001 to S1006 are saved in the internal storage device of the computer 109 and/or the external storage device 111. Note that each of the above steps S1001 to S1007 may be performed while the swallowing motion and swallowing sound are being measured by the swallowing measurement unit 410, or may be performed multiple times.

Based on the above-described processing steps, the processing unit 420 further calculates the behavior of the thyroid cartilage in the vertical direction and the longitudinal direction based on the vertical motion component and the longitudinal motion component described above. ) to generate two-dimensional trajectory data. Specifically, such two-dimensional trajectory data is generated as coordinate data shown on a coordinate plane defined by two mutually orthogonal coordinate axes, one coordinate axis corresponding to the trajectory data value of the longitudinal motion component. , the other coordinate axis corresponds to the trajectory data value of the vertical motion component. More specifically, as shown in FIG. 10, based on the signal fitting (step S502 in FIG. 4) and component extraction (step S503 in FIG. 4) by the motion analysis unit 421 described above, the vertical motion component waveform 1106 The above data values and the data values on the longitudinal motion component waveform 1105 are associated with each other in time, and the horizontal axis is the trajectory data value of the longitudinal motion component (displacement in the longitudinal direction; normalized amplitude in the longitudinal motion component waveform 1105). ), and the vertical axis is plotted as the trajectory data value of the vertical motion component (vertical displacement; normalized amplitude in the vertical motion component waveform 1106). That is, the horizontal axis indicates the value of the normal distribution having the parameters extracted for rAP of formula (1) in step S503 in FIG. values of a normal distribution with parameters

Such a trajectory graph 901 shown in FIG. 10 is displayed on the display device 110 via the display unit 430 of the computer 109. Particularly, in this embodiment, the plot of each trajectory data value on the trajectory graph 901 is Identification display, for example, color-coded display is performed according to the magnitude of the amplitude of the swallowing sound. In order to realize such an identification display, the processing unit 420 generates a swallowing sound waveform 801 and an envelope curve 802 representing temporal changes in the amplitude of the swallowing sound based on detection data detected through the microphone 106 as described above. In addition to temporally associating the swallowing sound waveform 801 or the envelope curve 802 with the trajectory graph 901, the plot of each trajectory data value on the trajectory graph 901 is identified and displayed according to the magnitude of the amplitude of the swallowing sound. to generate identification data for In relation to such identification display, in the present embodiment, which is color-coded display, a band graph for reference showing how the color changes according to the magnitude of the swallowing sound amplitude value along the vertical axis 909 is displayed adjacent to the trajectory graph 901 . For example, here, the larger the amplitude of the swallowing sound, the more yellowish it becomes, and the smaller the amplitude, the more blueish it becomes. Alternatively, an identification display form may be used in which the color is divided into black and white, and the color becomes lighter as the amplitude increases. Note that the identification display form is not limited to this, and trajectory data values with different amplitudes of swallowing sounds, such as changing the size or shape of the plot (mark) of each trajectory data value according to the magnitude of the amplitude of the swallowing sound. Any display mode may be used as long as it is a display mode that allows identification of each other.

Such a trajectory graph 901, in which the trajectory data values are plotted as a time-series scatter diagram, displays the behavior of the thyroid cartilage in the front-rear direction and in the vertical direction by separating them on two coordinate axes. Make behavior visible at a glance. In addition to the behavior of the swallowing motion, by displaying the features of the swallowing sound information in one trajectory graph 901 in this way, it is possible to visually confirm at what point in time the swallowing sound occurred with respect to the behavior of the thyroid cartilage. Not only can the swallowing motion be grasped quantitatively, but also the deviation of the swallowing sound from the normal state and the power of the swallowing sound can be grasped at a glance.

In addition, various supplementary information is added to the trajectory graph 901 and displayed. For this purpose, in the present embodiment, the processing unit 420 generates a predetermined feature point associated with the motion waveform 1103 (or the distance waveform 701), a predetermined feature point associated with the swallowing sound waveform 801 (or the envelope curve 802). , and supplementary display data for superimposing and displaying on the trajectory graph 901 supplementary information including the time of occurrence of the trajectory data values plotted on the trajectory graph 901, and the transition direction of the trajectory graph 901 and the trajectory graph. Reference display data for displaying reference information including a predetermined feature amount calculated from 901 together with the trajectory graph 901 is also generated.

Specifically, regarding such an auxiliary display, 902 in FIG. 10 is an arrow indicating in which direction the trajectory has progressed (transition direction of trajectory graph 901). In this embodiment, the trajectory starts from the coordinate origin, rotates counterclockwise, and then returns to the coordinate origin. 903 denotes a feature amount calculated from the trajectory graph 901 . Specifically, the maximum amount of displacement in the longitudinal direction, the maximum amount of displacement in the vertical direction, the maximum amount of displacement from the coordinate origin indicated by 904, and the time difference (σ) between the maximum values of the motion information and the voice information. , and the ratio of the time difference based on the variance of the displacement in the longitudinal direction (rAP), respectively, are shown as feature amounts. These pieces of information are obtained by the processing by the analysis unit 423 described above. As a method of displaying the feature amount, instead of displaying above the coordinate area of the trajectory graph 901 as in the present embodiment, it may be displayed within the coordinate area of the trajectory graph 901, or may be displayed in another coordinate area. It may be displayed in a drawing, and the present invention is not limited by these.

Also, in FIG. 10, 905 indicates the generation time of the trajectory data value plotted on the trajectory graph 901, which is displayed every 0.1 seconds in the present embodiment. 906 indicates a peak point in the distance information obtained in step S504 of FIG. 907 indicates the point in time when the maximum value of the voice information obtained in step S606 of FIG. 5 is obtained. With this display, it is possible to confirm the time lag between the point of time when the maximum value of the voice information is indicated and the point of time when the distance information indicates the maximum value of the component in the front-rear direction of the thyroid cartilage. Reference numeral 908 denotes the start point 805 and end point 806 (see FIG. 9) of the audio information obtained in step S607 of FIG.

As described above, according to the present embodiment, the model function modeling the swallowing motion is fitted to the distance information based on the detection data detected by the transmitting/receiving coils 102 and 103 to obtain the fitting result. Therefore, it is possible to non-invasively reproduce the movement of the thyroid cartilage (hyoid bone) two-dimensionally (modeling of the swallowing movement), and at the same time, the behavioral components related to all the movement directions of the thyroid cartilage during swallowing, that is, the vertical direction Two anteroposterior motion components and two vertical motion components corresponding to the movement in the forward and backward directions are extracted from the fitting results, and two-dimensional trajectory data showing the trajectories of the thyroid cartilage in the vertical direction and the anteroposterior direction based on these two components. is generated, it is possible to grasp the two-dimensional movement of the thyroid cartilage (hyoid bone) up and down and back and forth as swallowing dynamics at a glance without the need for comprehensive estimation of swallowing behavior.

Although the various processes (steps) related to the generation of the trajectory graph 901 are applied to the motion of the thyroid cartilage in the above, the various processes can also be applied to the motion of body parts other than the thyroid cartilage. That is, the various processes can be applied to analysis of movements of body parts other than the laryngeal region as long as they move in the same or similar manner as the thyroid cartilage (hyoid bone). Specifically, if the change in distance detected by a predetermined detection unit is decomposed into motions in a plurality of directions and can be analyzed, trajectory data is generated based on the motions, and a trajectory graph 901 is generated. can draw Further, the detection unit is not limited to detecting movement of a predetermined part (acquiring data indicating movement) using a coil. For example, the laryngeal displacement detection unit is not limited to detecting the movement by the coils 102 and 103 as long as it can detect the movement of the laryngeal (thyroid cartilage) (capable of acquiring data indicating the movement).

Next, display by the display device 110 will be described in detail.
Calculator 109 (display unit 430) can cause display device 110 to display a graph showing movement of the thyroid cartilage (larynx) and a graph relating to swallowing sounds. Further, the computer 109 (display unit 430) displays a moving image 950 captured by the camera 114 (a moving image of the subject 101 during the swallowing motion and swallowing sound measurement, which captures the movement of the thyroid cartilage), It can be displayed on the display device 110 .
An example will be described below in which a distance waveform 710 and a trajectory graph 901 are displayed as graphs representing the movement of the thyroid cartilage, and a swallowing sound waveform 801 and a trajectory graph 901 are displayed as graphs related to swallowing sounds. However, the graph showing the movement of the thyroid cartilage and the graph relating to swallowing sounds are not limited to these. Further, in the following description relating to display, "distance waveform 701" may be read as "motion waveform 1103", "backward motion component waveform 1105", or "vertical motion component waveform 1106". Also, the “swallowing sound waveform 801” may be read as “envelope 802” or the like.

The distance waveform 701 is displayed as a graph in which the first coordinate axis (horizontal axis) corresponds to time and the second coordinate axis (vertical axis) intersecting with the first coordinate axis corresponds to the distance between the coils 102 and 103. It is a display, and can be said to be a display of a graph showing the movement of the thyroid cartilage. In the display of the trajectory graph 901, the first coordinate axis (horizontal axis) corresponds to the trajectory data value of the anteroposterior motion component of the thyroid cartilage, and the second coordinate axis (vertical axis) intersecting with the first coordinate axis is up and down. This is a display of a graph corresponding to the trajectory data value of the dynamic component, and can be said to be a display of a graph showing the movement of the thyroid cartilage. In the display of the swallowing sound waveform 801, the first coordinate axis (horizontal axis) corresponds to time, and the second coordinate axis (vertical axis) intersecting with the first coordinate axis corresponds to the magnitude (amplitude) of the swallowing sound. It is a display of the corresponding graph, and can be said to be a display of a graph relating to swallowing sounds. Note that the trajectory graph 901 may be, for example, a three-dimensional graph or the like having a time axis as a third coordinate axis intersecting the first coordinate axis and the second coordinate axis.

As shown in FIGS. 11 and 12, the calculator 109 can execute processing for displaying the distance waveform 701, the swallowing sound waveform 801, and the moving image 950 simultaneously (on the same screen) on the display device 110. there is A screen on which the distance waveform 701 , the swallowing sound waveform 801 , and the moving image 950 are displayed is called a first display screen 2001 . Further, as shown in FIGS. 13 and 14, the calculator 109 causes the display device 110 to display the trajectory graph 901, the distance waveform 701, the swallowing sound waveform 801, and the moving image 950 at the same time (on the same screen). processing can be executed. A screen on which the trajectory graph 901 , the distance waveform 701 , the swallowing sound waveform 801 , and the moving image 950 are displayed is called a second display screen 2002 .
Note that some of the items displayed on the same screen here may not be displayed on the same screen.

11 and 12 are diagrams showing display examples of the first display screen 2001. FIG. FIG. 11 shows the distance waveform 701 and the swallowing sound waveform 801 without markers 964 (964a, 964b), which will be described later. Further, FIG. 12 shows a state in which a marker 964, which will be described later, is displayed for the distance waveform 701 and the swallowing sound waveform 801, and the image at the time (6.39 seconds) when the display of the moving image 950 is indicated by the marker 964. Indicates that the frame is displayed. 13 and 14 are diagrams showing display examples of the second display screen 2002. FIG. FIG. 13 shows the distance waveform 701 and the swallowing sound waveform 801 in a state in which a marker 964, which will be described later, is not displayed, and the entire trajectory graph 901 is plotted. Further, FIG. 14 shows a state in which a marker 964, which will be described later, is displayed for the distance waveform 701 and the swallowing sound waveform 801, and the image at the time (4.74 seconds) when the display of the moving image 950 is indicated by the marker 964. Indicates that the frame is displayed. Also, FIG. 14 shows a state in which a part of the trajectory graph 901 is plotted, and shows a state in which the trajectory graph 901 is plotted up to the portion corresponding to the point in time (4.74 seconds). is.

The operator can use the input device 112 to specify any point on the distance waveform 701, the swallowing sound waveform 801, or the trajectory graph 901 displayed on the first display screen 2001 or the second display screen 2002. It has become. Specifically, for example, using a mouse as the input device 112, by aligning the cursor 960 with an arbitrary point on the displayed distance waveform 701, swallowing sound waveform 801, or trajectory graph 901 (by aligning the cursor 960 by clicking), the point may be specified (see FIG. 12). Further, for example, a touch panel integrated with the display device 110 serves as the input device 112, and by touching an arbitrary point on the distance waveform 701, the swallowing sound waveform 801, or the trajectory graph 901, the point is specified. It can be like this.

The computer 109 causes the display device 110 to display an image corresponding to the specified point in the video 950 displayed on the same screen based on the operation of specifying the point on the input device 112 . In other words, the computer 109 displays a display that enables the user to grasp the point corresponding to the specified point in the moving image 950 displayed on the same screen based on the operation of specifying the point on the input device 112. Display on device 110 . Specifically, for example, when an arbitrary point on the distance waveform 701, in other words, data at a specific point in time on the distance waveform 701 is specified, the computer 109 calculates A video (video frame) is extracted (searched) from the video 950 (video data), and the extracted video is displayed on the display device 110 . Further, for example, when an arbitrary point of the swallowing sound waveform 801, in other words, data at a specific point in time on the swallowing sound waveform 801 is specified, the computer 109 calculates the image captured at the same time as the acquisition of the data. The (video frame) is extracted (searched) from the video 950 (video data), and the extracted video is displayed on the display device 110 . Further, for example, when an arbitrary point on the trajectory graph 901, in other words, specific trajectory data on the trajectory graph 901 is specified, the computer 109 calculates the generation time of the trajectory data (the time associated with the trajectory data). A video (video frame) captured at the same time as the time) is extracted (searched) from the video 950 (video data), and the extracted video is displayed on the display device 110 . For example, as shown in FIG. 12, if the data at 6.39 seconds (for example, 6.39 seconds after the start of measurement) in the distance waveform 701 is specified, the data at 6.39 seconds in the moving image 950 (for example, 6.39 seconds after the start of measurement) is displayed.

Note that the computer 109 calculates the specified point (the A marker 964, which will be described later, may be displayed on the point portion).

Also, the operator can use the input device 112 to specify an arbitrary time point (point) in the moving image 950 displayed on the first display screen 2001 or the second display screen 2002 . Specifically, in the present embodiment, a seek bar 962 indicating the playback position of moving image 950 is displayed on first display screen 2001 or second display screen 2002 . The operator can use a mouse, a touch panel, or the like as the input device 112 to specify the playback position of the moving image 950 (an arbitrary time point (point) in the moving image 950) on the seek bar 962. FIG. It should be noted that the designation of an arbitrary time point in the moving image 950 may be performed by directly inputting the playback time (playback position) using, for example, a keyboard as the input device 112. 2002 may be provided with a button for fast-forwarding or fast-reversing the moving image 950 by a predetermined frame (predetermined time), and this may be performed by operating the button.

Calculator 109 calculates the distance waveform 701, swallowing sound waveform 801, or trajectory graph 901 displayed on the same screen based on the operation of specifying an arbitrary time point (point) of the moving image 950 on the input device 112. The display device 110 is caused to display a display that enables the point corresponding to the point in time to be grasped. In other words, in a state where a specific video frame in the moving image 950 is displayed, the computer 109 displays a point corresponding to the video frame in the distance waveform 701, the swallowing sound waveform 801, or the trajectory graph 901. is displayed on the display device 110 . Specifically, for example, the calculator 109 may display a predetermined marker 964 at a location corresponding to a specified point in the distance waveform 701, swallowing sound waveform 801, or trajectory graph 901. Note that the marker 964 may be, for example, an icon 964a or the like, or a straight line 964b or the like perpendicular to the time axis (predetermined coordinate axis) (see FIGS. 12 and 14). Further, for example, the calculator 109 may display a distance waveform 701, a swallowing sound waveform 801, or a trajectory graph 901 that is plotted up to a specified point in time and not plotted beyond the specified point in time. (See locus graph 901 in FIG. 14). In addition, for example, the calculator 109 can generate a distance waveform 701, a swallowing sound waveform 801, or a trajectory graph 901 in which plots (lines) have different colors, thicknesses, etc. up to a specified time point and beyond the specified time point. may be displayed. In FIG. 14, the time point of 4.74 seconds (for example, 4.74 seconds after the start of measurement) is specified in the moving image 950, and the time point of 4.74 seconds (for example, 4.74 seconds after the start of measurement) is specified in the distance waveform 701 and the swallowing sound waveform 801 (for example, 4.74 seconds after the start of measurement). A state in which a marker 964 is displayed in the portion of 0.74 seconds has elapsed is shown. Also, in FIG. 14, the time point of 4.74 seconds (for example, 4.74 seconds after the start of measurement) of the moving image 950 is designated, and the trajectory graph 901 is set at 4.74 seconds (for example, 4.74 seconds after the start of measurement). The plotted state is shown until the elapsed time).

Note that the marker 964 indicates the position corresponding to the current playback position (displayed video frame) of the moving image 950, and may move along with the playback (progress) of the moving image 950. Also, the display of the distance waveform 701, the swallowing sound waveform 801, or the trajectory graph 901 is plotted up to the point in time corresponding to the current playback position (displayed video frame) of the moving image 950 (before the specified point in time). is not plotted), and the plot may proceed in accordance with the playback (progress) of the moving image 950 . In addition, the display of the distance waveform 701, the swallowing sound waveform 801, or the trajectory graph 901 is displayed up to the time point corresponding to the current playback position (displayed video frame) of the moving image 950. state (the plot (line) before the designated point is not in the predetermined state), and the portion in the predetermined state is advanced in accordance with the playback (progress) of the moving image 950. You can do it. That is, in accordance with the playback (progress) of the video 950, a display (a display or plot in which the marker 964 moves) that makes it possible to grasp the point corresponding to the current video playback position in the distance waveform 701, the swallowing sound waveform 801, or the trajectory graph 901 ) may be displayed on the display device 110 .

Note that in the present embodiment, based on the operation of specifying an arbitrary point on the graph (distance waveform 701, swallowing sound waveform 801, or trajectory graph 901), or the operation of specifying an arbitrary point in the video 950 An image (image frame) at a specific point in time and a display (marker 964 or the like) indicating a point on the graph corresponding to the image (image frame) are displayed on the same screen. One video (video frame) may be displayed on the same screen, or a plurality of video frames may be displayed. That is, for example, by performing an operation to specify a plurality of points on the graph on the input device 112, images (image frames) corresponding to the plurality of specified points, that is, a plurality of images (image frames) are generated. ) may be displayed on the same screen. In addition, each point specified at this time is displayed so that the corresponding relationship between each image (image frame) can be understood (for example, by color coding, attaching an identification code, or aligning the positional relationship). You may let Further, for example, by performing an operation of designating a plurality of times (an operation of designating a plurality of time points of the moving image 950) on the input device 112, points on the graph corresponding to each of the designated time points can be displayed. display (that is, display of a plurality of markers 964, etc.) may be displayed on the same screen.

Based on the operation of specifying an arbitrary point on the graph (distance waveform 701, swallowing sound waveform 801, or trajectory graph 901), an image at a time point corresponding to the specified point in the moving image 950 (that is, the specified point in the moving image 950) is displayed. display on the display device 110, or based on the operation of designating a predetermined time point in the moving image 950, the point corresponding to the designated time point in the graph is grasped. The processing for displaying the display that enables display on the display device 110 is performed by combining the distance information, the audio information, and the moving image 950 with mutual time information (for example, obtaining the acquisition time of the distance information, the acquisition time of the audio information, and the acquisition of each frame of the moving image 950). It can be realized by storing in association with the time period). In the present embodiment, acquisition of distance information by the transmitting/receiving coils 102 and 103, acquisition of audio information by the microphone 106, and shooting of the moving image 950 are performed in synchronism. Time information and time information about the moving image 950 are associated. Specifically, in the present embodiment, a measurement start operation on the input device 112 (for example, an operation of clicking (selecting) a measurement start button displayed on the display device 110, an operation of a predetermined physical button for instructing the start of measurement, etc.) etc.), the measurement of the swallowing motion (larynx displacement detection step), the measurement of the swallowing sound (the swallowing sound detection step), and the shooting of the video 950 (video acquisition step) are simultaneously started. ing. As a result, the distance information, the audio information, and the moving image 950 stored in the predetermined memory are associated with each other by their time information (distance information acquisition time, sound information acquisition time, and acquisition time of each frame of the moving image 950). It will be stored as it is. Here, "storing in a state in which the time information is associated with each other" means that distance information (the distance between the coils 102 and 103) and audio information (the amplitude of the swallowing sound) are stored for one piece of time information. ) and the moving image 950 (each frame image forming the moving image 950) are associated with each other.

Further, in this embodiment, as shown in FIG. 11, the distance waveform 701 and the swallowing sound waveform 801 are displayed in parallel on the first display screen 2001 . Also, in the first display screen 2001 , the display area of the moving image 950 is outside the display area of the distance waveform 701 and the display area of the swallowing sound waveform 801 . Further, as shown in FIG. 13, a distance waveform 701 and a swallowing sound waveform 801 are displayed in an overlapping manner on the second display screen 2002 . In other words, the distance waveform 701 and the swallowing sound waveform 801 are displayed on one graph. Also, in the second display screen 2002 , the display area of the moving image 950 is inside the display area of the distance waveform 701 and the display area of the swallowing sound waveform 801 . In other words, the moving image 950 is displayed superimposed on the graph of the distance waveform 701 and the graph of the swallowing sound waveform 801 . Thus, by displaying the distance waveform 701 and the swallowing sound waveform 801 in an overlapping manner, it is possible to display the trajectory graph 901 and the moving image 950 in a large size. Also, by displaying the moving image 950 so as to overlap a predetermined graph, it is possible to display the predetermined graph and other graphs in a larger size. Note that the predetermined graph may be the trajectory graph 901, for example.

Further, in the present embodiment, the measured distance waveform 701 and swallowing sound waveform 801 are displayed over the entire measurement time (first range), and a part of the measurement time (first range) is displayed. It is possible to display the second range as a partial range of the range). For example, here, it is assumed that the measurement (and the shooting of the moving image 950) is performed for 15 seconds. Note that the measurement (and shooting of the moving image 950) is started based on a measurement start operation on the input device 112, and a measurement end operation on the input device 112 (for example, by clicking a measurement end button displayed on the display device 110). The measurement may be ended based on an operation, an operation on a predetermined physical button for instructing the end of measurement, or the like. Also, the measurement (and the shooting of the moving image 950) may be started based on a measurement start operation on the input device 112, and may end after a predetermined time (for example, 15 seconds) has elapsed.

In the present embodiment, it is possible to select (specify) which time range of the measured time the distance waveform 701 and the swallowing sound waveform 801 are to be displayed. The selection may be made by, for example, specifying (drag, etc.) an arbitrary range of the displayed distance waveform 701 or swallowing sound waveform 801 using a mouse, touch panel, or the like as the input device 112. This may be performed by inputting the start point and end point of the time range to be displayed using a keyboard as the input device 112 . 11 and 12 show the distance waveform 701 and the swallowing sound waveform 801 displayed over the entire measurement time. 13 and 14 show a part of the time range (3.94 seconds to 5.37 seconds) of the measurement time (the time range (first range) displayed in FIGS. 12 and 13). A distance waveform 701 and a swallowing sound waveform 801 displayed for a part of the time range (second range) of . That is, in the present embodiment, based on the operation of specifying a partial range (time range) of the distance waveform 701 and the swallowing sound waveform 801, the specified range of the distance waveform 701 and the swallowing sound waveform 801 can be displayed. It is possible. In addition, on the screen (second display screen 2002) on which the distance waveform 701 and the swallowing sound waveform 801 in a partial time range are displayed, a portion of the captured moving image 950 corresponding to the partial time range is displayed. (only) is to be played. That is, in a state in which a partial time range of the distance waveform 701 and the swallowing sound waveform 801 is selected and displayed (a state in which the second display screen 2002 is displayed), the video 950 is reproduced on the input device 112. When the playback of the moving image 950 is started based on a start operation (for example, an operation of clicking a playback button displayed on the display device 110, an operation of a predetermined physical button for instructing the start of playback, etc.), Of the moving image 950, the moving image 950 from the start point to the end point of the partial time range is reproduced.
Note that the distance waveform 701 and swallowing sound waveform 801 shown in FIGS. 11 and 12 may be based on part of the measured data.

Note that, for example, the operation of selecting the time range to display the distance waveform 701 (swallowing sound waveform 801) also serves as the operation of specifying the analysis range of the distance information (audio information) (operation related to starting analysis). good too. That is, for example, when the motion analysis unit 421 analyzes the distance information (analysis of the swallowing sound by the voice analysis unit 422) (in other words, when generating the trajectory graph 901), When specifying which time range to analyze, the analysis result (for example, the trajectory graph 901) is displayed on the display device 110 based on the operation of specifying the time range to be analyzed, and the distance waveform of the analyzed time range 701 and the swallowing sound waveform 801 may be displayed on the display device 110 . Further, based on the operation of designating the time range to be analyzed, it may be possible to reproduce the moving image of the analyzed time range. In other words, the second display screen 2002 may be displayed based on the operation of designating the analysis range (operation related to starting analysis).

The biopsy apparatus 100 of this embodiment is
A biopsy apparatus 100 including an input device 112 that receives an operation by an operator, a display device 110, and an information processing device (computer 109),
The information processing device 109 is
A graph (distance waveform 701 or trajectory graph 901, etc.) showing the movement of the thyroid cartilage when the subject swallows, based on the detection data acquired by the detection units 102 and 103 that detect the movement of the thyroid cartilage, and the camera 114. A process of simultaneously displaying on the display device 110 a moving image 950 of the subject when swallowing, which is shot by
In a state in which the graph and the moving image 950 are displayed simultaneously, based on the operator's operation of designating an arbitrary point in either the graph or the moving image 950, the corresponding point in the other of the graph or the moving image 950 is displayed. and a process of causing the display device 110 to display a display that enables the point corresponding to the arbitrary point to be grasped.
Specifically, for example, when the graph and the moving image 950 are displayed on the same screen, the information processing device performs , causes the display device 110 to display an image (image frame) at a time point corresponding to the arbitrary point (specified point) in the moving image 950 . This makes it possible to grasp the point (video frame) on the moving image 950 corresponding to the specified point.
Alternatively, the information processing device, in a state in which the graph and the moving image 950 are displayed on the same screen, based on the operator's operation of designating an arbitrary time point (point) in the moving image 950, The display device 110 may be caused to display a point corresponding to the arbitrary time point (designated time point) in the graph so that the point can be grasped. This makes it possible to grasp the point on the graph corresponding to the designated point.
It should be noted that "on the basis of the operator's operation of designating an arbitrary point in one of the graph or the moving image, a display that enables the user to grasp the point corresponding to the arbitrary point in the other of the graph or the moving image" is described above. "Display on a display device" means that, based on an operation of specifying an arbitrary point on at least one of a graph or a moving image, it is sufficient to display the corresponding point on the other so that the other can be grasped, and the specifying operation on the other It is not necessary to make a display that allows the user to grasp the corresponding points on the one side based on the above.

According to such a configuration, the graph showing the movement of the thyroid cartilage and the moving image of the subject during swallowing are displayed on the same screen. The graph (detection data) can be confirmed while observing the movement, and the examination of swallowing can be efficiently performed. In addition, according to such a configuration, based on the operation of designating an arbitrary point in the graph, a display that enables the user to grasp the point corresponding to the arbitrary point in the moving image is displayed, or an arbitrary point in the moving image is displayed. At least one of displaying a display that makes it possible to grasp the point corresponding to the arbitrary point in the graph based on the operation of designating the point. That is, by specifying an arbitrary point on the graph, searching for a video frame in the video corresponding to that arbitrary point, or by specifying an arbitrary time (point) in the video, Finding the corresponding point on the graph becomes possible. Therefore, it becomes easy to perform work such as extracting a range in which analysis should be performed intensively in the graph, and it is possible to efficiently perform a swallowing examination. In addition, it becomes easy to grasp the relationship between each point on the graph and the actual swallowing motion shown in the moving image, and the examination of swallowing can be performed efficiently.

Note that the processing by each device described in the present embodiment may be realized by any of software, hardware, and a combination of software and hardware. Programs that make up software are, for example, non-transitory computer-readable media (non-transitory
computer readable medium). Also, the program may be distributed via a network, for example.

It should be noted that the present invention is not limited to the above-described embodiments, and it is possible to freely combine each component, or to modify or omit any component without departing from the scope of the invention.

100 biopsy device 102 transmission coil (detection unit)
103 Receiving coil (detection unit)
106 microphone 109 computer (information processing device)
110 display device 112 input device 114 camera

Claims (5)

1. A biological examination apparatus comprising an input device that receives an operation by an operator, a display device, and an information processing device,
   The information processing device is
   A graph showing the movement of the thyroid cartilage when the subject swallows based on the detection data acquired by the detection unit that detects the movement of the thyroid cartilage, and an image of the subject when the subject swallows taken by a camera. and a process of simultaneously displaying on the display device,
   In a state in which the graph and the moving image are displayed at the same time, based on an operation by the operator to designate an arbitrary point on one of the graph or the moving image, the arbitrary point on the other of the graph or the moving image and a process of causing the display device to display a display that enables the user to grasp a point corresponding to the biopsy apparatus.
2. 2. The graph as claimed in claim 1, wherein said graph has a first coordinate axis corresponding to the longitudinal movement of the thyroid cartilage and a second coordinate axis corresponding to the vertical movement of the thyroid cartilage and intersecting said first coordinate axis. 2. The biopsy device according to 1.
3. The biopsy apparatus according to claim 1 or 2, characterized in that the graph is a single graph in which the movement of the thyroid cartilage and the swallowing sound acquired by the microphone are associated with each other in time.
4. A biopsy method performed by a biopsy apparatus including an input device for receiving an operation by an operator, a display device, and an information processing device,
   A graph showing the movement of the thyroid cartilage when the subject swallows based on the detection data acquired by the detection unit that detects the movement of the thyroid cartilage, and an image of the subject when the subject swallows taken by a camera. and a step of simultaneously displaying on the display device;
   In a state in which the graph and the moving image are displayed at the same time, based on an operation by the operator to designate an arbitrary point on one of the graph or the moving image, the arbitrary point on the other of the graph or the moving image and a step of causing the display device to display a display that makes it possible to grasp a point corresponding to .
5. A graph showing the movement of the thyroid cartilage when the subject swallows based on the detection data acquired by the detection unit that detects the movement of the thyroid cartilage, and an image of the subject when the subject swallows taken by a camera. and a process of simultaneously displaying on a display device,
   In a state in which the graph and the moving image are displayed simultaneously, based on an operator's operation to specify an arbitrary point on either the graph or the moving image, the arbitrary point on the other of the graph or the moving image is displayed. A program for causing a computer to execute a process of causing the display device to display a display that makes it possible to grasp a corresponding point.

Images