JP7514521B2 - HEART RATE MEASUREMENT SYSTEM, HEART RATE MEASUREMENT METHOD, AND HEART RATE MEASUREMENT PROGRAM - Google Patents

Description

The present invention relates to a heart rate measurement system, a heart rate measurement method, and a heart rate measurement program.

Conventionally, measuring heart rate has been widely practiced for the purpose of preventing heart disease, etc. It is desirable to measure heart rate as simply as possible. To meet this demand, for example, a small wearable device capable of measuring heart rate is used. However, even if the wearable device is small, there are cases where the wearer feels burdened by simply wearing the device.
Therefore, there are techniques for measuring the heart rate of a person to be measured without contacting the person. For example, in the technique disclosed in Patent Literature 1, an image of the person to be measured is taken. Then, the heart rate is measured based on the change in luminance in the image.

JP 2020-092817 A

However, it is believed that there is room for improvement in the general technology for measuring heart rate based on images, such as that disclosed in the above-mentioned Patent Document 1, in order to improve the accuracy of the measurement.

The present invention has been made in light of these circumstances. The objective of the present invention is to perform image-based heart rate measurements with greater accuracy.

In order to solve the above problem, a heart rate measuring system according to an embodiment of the present invention comprises:
an image acquisition means for acquiring a plurality of images generated by continuously photographing a subject;
a pulse wave signal detection means for detecting a pulse wave signal which is a signal indicative of a pulse wave component of the subject based on a change in a value expressing a color between the plurality of images;
a measuring means for detecting a timing of a heartbeat in the pulse wave signal by analyzing a cross-correlation between the pulse wave signal and a model signal which is a model of a waveform indicative of a heartbeat, and for measuring the heartbeat of the subject based on the detected timing of the heartbeat;
The present invention is characterized by comprising:

The present invention makes it possible to perform image-based heart rate measurements with greater accuracy.

1 is a diagram showing a system configuration of a heart rate measuring system according to an embodiment of the present invention; 1 is a block diagram showing an example of a configuration of an imaging device according to an embodiment of the present invention; 1 is a block diagram showing an example of a configuration of a heart rate measuring device according to an embodiment of the present invention. 11 is a graph showing correction of the waveform of a pulse wave signal. 1 is a graph showing an example of a model signal used in a cross-correlation analysis. 11 is a graph showing an example of distribution of detection counts to explain measurement of heartbeat timing. 11 is a graph showing an example of a measurement result of a measurement involving a cross-correlation analysis in the present embodiment. 5 is a flowchart showing the flow of a photographing process executed by the photographing device according to one embodiment of the present invention. 4 is a flowchart showing the flow of a heart rate measurement process executed by a heart rate measurement device according to an embodiment of the present invention.

An example of an embodiment of the present invention will now be described with reference to the attached drawings.

[System configuration]
Fig. 1 is a diagram showing a system configuration of a heartbeat measurement system S according to this embodiment. As shown in Fig. 1, the heartbeat measurement system S includes a photographing device 10 and a heartbeat measurement device 20. Fig. 1 also shows a measurement subject whose heartbeat is to be measured by the heartbeat measurement system S.

The photographing device 10 and the heart rate measuring device 20 are connected so that they can communicate with each other. This communication may be performed in accordance with any communication method, and the communication method is not particularly limited. The connection in this communication may be a wired connection or a wireless connection. Furthermore, this communication may be performed directly between each device, or may be performed via a network including a relay device. In this case, the network is realized by, for example, a network such as a LAN (Local Area Network), the Internet, or a mobile phone network, or a network that combines these.

The heart rate measurement system S is a system that uses a photographing device 10 to capture an image of a person being measured, and uses a heart rate measurement device 20 to perform more accurate measurements of the heart rate based on this image.

The photographing device 10 generates multiple images by continuously photographing the subject as a subject. The photographing device 10 then transmits image data corresponding to the multiple images generated to the heart rate measurement device 20. The photographing device 10 can be realized by, for example, an information processing device equipped with a photographing function, such as a smartphone, digital camera, or video camera.

The heartbeat measuring device 20 acquires the image data transmitted by the photographing device 10. Then, the heartbeat measuring device 20 measures the heartbeat of the subject based on the acquired image data. The heartbeat measuring device 20 can be realized by an information processing device having a calculation processing capacity, such as a personal computer or a server device.
When measuring the heartbeat, the heartbeat measuring device 20 detects a pulse wave signal, which is a signal indicating the pulse wave component of the subject, based on the change in the value expressing the color between a plurality of images. The heartbeat measuring device 20 also detects the timing of the heartbeat in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and a model signal, which is a model of the waveform indicating the heartbeat, and measures the heartbeat of the subject based on the detected heartbeat timing.

In this way, in the heart rate measurement system S, each device works together to detect a pulse wave signal from an image of the subject. Then, in the heart rate measurement system S, the timing of the heart rate is detected by analyzing the cross-correlation between the pulse wave signal and the model signal, and the heart rate is measured based on this.
As a result, the heart rate measurement system S can perform measurement of the heart rate based on the image with higher accuracy. Therefore, the heart rate measurement system S can measure the heart rate of the measurement subject even if there is an event that affects the measurement, such as a change in the orientation of the measurement subject (e.g., a change in the face orientation) or a change in brightness due to the surrounding environment. In addition, the heart rate measurement system S can measure the heart rate of the measurement subject without contacting the measurement subject, and the measurement subject does not need to wear a wearable device or the like, so that it is possible to more easily realize measurement of the heart rate.

Next, we will explain in more detail the configuration and functions of the imaging device 10 and the heart rate measurement device 20 to realize this processing.

[Configuration of the imaging device]
Next, the configuration of the image capturing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the image capturing device 10.
2, the photographing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication unit 14, a storage unit 15, an input unit 16, an output unit 17, and an imaging unit 18. These units are connected by signal lines and transmit and receive signals between each other.

The CPU 11 executes various processes (for example, a photographing process described below) according to a program recorded in the ROM 12 or a program loaded from the storage unit 15 to the RAM 13 .
The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes.

The communication unit 14 performs communication control for the CPU 11 to communicate with other devices (for example, the heart rate measuring device 20).
The storage unit 15 is configured with a semiconductor memory such as a dynamic random access memory (DRAM), and stores various data.

The input unit 16 is composed of various buttons and the like, and inputs various information in response to user's instruction operations. The output unit 17 is composed of a display, a speaker, and the like, and outputs images and sounds. Note that, when the photographing device 10 is realized by, for example, a smartphone or a digital camera, the input unit 16 can be composed of a touch sensor and placed over the display of the output unit 17, thereby providing a configuration with a touch panel.
The imaging unit 18 is configured with an imaging device including a lens, an imaging element, etc., and captures a digital image of a subject.

In the photographing device 10, these various parts work together to perform the "photographing process." Here, the photographing process is a series of processes that generate multiple images by continuously photographing the subject as a subject, and transmit image data corresponding to the multiple images thus generated to the heart rate measuring device 20.

When the photographing process is executed, as shown in FIG. 2, in the CPU 11, a photographing control unit 111 and an image data transmission unit 112 function.
An image data storage section 251 is provided in one area of the storage section 15 .
Including cases not specifically mentioned below, data required to realize processing is transmitted and received between these functional blocks at appropriate times as appropriate.

The photographing control unit 111 controls continuous photographing of the subject by the imaging unit 18 based on an instruction operation from the photographer (the subject may be the photographer himself or another person) received by the input unit 16, or instruction information from the photographer received via the communication unit 14. For example, the photographing control unit 111 controls outputting a live view screen or a predetermined user interface to assist the photographer in photographing from the output unit 17, and reflecting the operation instruction received by the input unit 16. Here, in this embodiment, it is assumed that the photographing control unit 111 takes continuous photographs of the subject's face as a subject, but this is merely an example for explanation. The photographing device 10 may be configured to photograph other parts of the subject other than the face (for example, the neck, hands, or parts of the upper body near the heart).

The shooting control unit 111 also generates multiple images by continuous shooting using the imaging unit 18. Then, the shooting control unit 111 stores image data corresponding to the multiple images generated in the image data storage unit 251. That is, the image data storage unit 251 functions as a storage unit that stores image data. Note that the image data corresponding to the multiple images may be in a general-purpose video format. For example, it may be in a 30 FPS (Frames Per Second) video format expressed in RGB.

The image data transmission unit 112 transmits the image data that the imaging control unit 111 has stored in the image data storage unit 151 to the heart rate measurement device 20. Note that the transmission may be performed by storing the generated image data in the image data storage unit 151 and transmitting the image data stored in the image data storage unit 151 all at once after imaging is completed, or may be performed in real time as the imaging control unit 111 generates the image data.

[Configuration of Heart Rate Measurement Device]
Next, the configuration of the heartbeat measuring device 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the heartbeat measuring device 20. As shown in Fig. 3, the heartbeat measuring device 20 includes a CPU 21, a ROM 22, a RAM 23, a communication unit 24, a storage unit 25, an input unit 26, an output unit 27, and a drive 28. These units are connected by signal lines and transmit and receive signals between each other.

The CPU 21 executes various processes (for example, a heart rate determination process described later) according to a program recorded in the ROM 22 or a program loaded from the storage unit 25 to the RAM 23 .
The RAM 23 also stores data and the like necessary for the CPU 21 to execute various processes.

The communication unit 24 performs communication control for the CPU 21 to communicate with other devices (for example, the image capturing device 10).
The storage unit 25 is composed of a semiconductor memory such as a DRAM, and stores various data.

The input unit 26 is composed of various buttons, a touch panel, or an external input device such as a mouse or a keyboard, and inputs various information in response to user instructions.
The output unit 27 is composed of a display, a speaker, etc., and outputs images and sounds.

Removable media (not shown) such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached to the drive 28. Programs and various data read from the removable media by the drive 28 are installed in the storage unit 25 as necessary.

In the heart rate measuring device 20, these units cooperate with each other to perform a "heart rate measuring process."
The heart rate measurement process is a series of processes for acquiring image data transmitted by the image capture device 10 and measuring the heart rate of the subject based on the image data.

When this heart rate measurement process is executed, as shown in FIG. 3, an image data acquisition unit 211, an area extraction unit 212, a pulse wave signal detection unit 213, a waveform correction unit 214, and a heart rate measurement unit 215 function in the CPU 21.
In addition, an image data storage section 251, a pulse wave signal storage section 252, and a model signal storage section 253 are provided in one area of the storage section 25.
Including cases not specifically mentioned below, data required to realize processing is transmitted and received between these functional blocks at appropriate times as appropriate.

The image data acquisition unit 211 acquires image data corresponding to multiple images in which the subject is the subject, by receiving the image data transmitted from the imaging device 10. The image data acquisition unit 211 then stores the acquired image data in the image data storage unit 251. In other words, the image data storage unit 251 functions as a storage unit that stores image data.

The imaging device 10 may store image data in a removable medium. The image data acquisition unit 211 may then acquire the image data from this removable medium inserted in the drive 28, rather than acquiring the image data through communication.

The area extraction unit 212 performs a process of extracting a predetermined area from each of a plurality of images corresponding to the image data so that the pulse wave signal detection unit 213 described later can detect a pulse wave signal, which is a signal indicating the pulse wave component of the subject, from the image data. Specifically, the area extraction unit 212 extracts a face area, which is an area excluding areas in which background, clothing, and parts other than the face are photographed, which are unnecessary when detecting the pulse wave component, from each of a plurality of images corresponding to the image data stored in the image data storage unit 251. Furthermore, the pulse wave signal detection unit 213 extracts a skin area, which is an area excluding areas in which hair, eyeballs, and the inside of the mouth are photographed, which are unnecessary when detecting the pulse wave component, from each of the extracted face areas. Then, the pulse wave signal detection unit 213 outputs the image after extracting this skin area to 313. This skin area becomes the target area for detecting the pulse wave signal by the pulse wave signal detection unit 213 described later.

Here, the extraction of the face region and skin region by the region extraction unit 212 can be realized by any method. For example, the extraction of the face region by the region extraction unit 212 can be realized by a method called the Viola-Jones method, which is widely used as a face region extraction method. In the Viola-Jones method, a Haar Cascade classifier is generated based on Haar Like features that indicate features in the face region (for example, features such as the region around the eyes being darker than the cheeks, or the regions on either side of the nose being darker than the bridge of the nose). Then, based on this Haar Cascade classifier, the face region can be extracted from the image with high accuracy.

Next, the region extraction unit 212 further extracts a skin region from each of the extracted face regions. For example, the region extraction unit 212 can extract the skin region by using a classifier such as OC-SVM (One Class Support Vector Machine). In this case, the region extraction unit 212 first converts the image data expressed in RGB to be expressed in YCrCb, which is expressed by a luminance signal Y and two color difference signals Cr and Cb. The region extraction unit 212 then separates the luminance signal Y and extracts the skin region based only on the two color difference signals Cr and Cb. In this way, the region extraction unit 212 can extract the skin region by reducing the influence of the skin shadow by once separating the lightness component indicated by the luminance signal Y.

Then, the region extraction unit 212 performs outlier detection using OC-SVM. In this case, for example, the pulse wave signal detection unit 213 assumes that the center of the face region is skin, and performs learning for one class using the Cr and Cb values of the pixels in this center region as normal data, thereby determining the discrimination boundary for outlier detection. The region extraction unit 212 then detects the region corresponding to the outlier (corresponding to a non-skin region) based on the obtained discrimination boundary. This makes it possible to separate the skin region from the non-skin region. The region extraction unit 212 then reconverts the skin region back into the original image expressed in RGB. Meanwhile, the region extraction unit 212 replaces the RGB values of the non-skin region with zero. In this way, the region extraction unit 212 can extract the skin region from the image.

By performing such processes of extracting face areas and skin areas, the area extraction unit 212 can generate images from which skin areas have been extracted (hereinafter referred to as "skin area images") from each of the multiple images corresponding to the image data (i.e., from each frame in the video). The area extraction unit 212 then outputs the multiple skin area images corresponding to each of the multiple images to the pulse wave signal detection unit 213.

The pulse wave signal detection unit 213 detects a pulse wave signal, which is a signal indicating the pulse wave component of the subject, based on the change in the value expressing the color between the multiple skin region images input from the region extraction unit 212. For example, detection of the pulse wave signal by the region extraction unit 212 can be achieved by a method called rPPG (Remote Photo-Plethysmography), which is widely used as a method for detecting pulse wave signals. rPPG focuses on the fact that blood components (e.g., hemoglobin) absorb more light than surrounding tissues, and therefore the pulse wave component associated with blood flow is superimposed on the diffuse reflection of the skin.

Specifically, rPPG detects a pulse wave signal, which is a signal indicating the pulse wave components of the subject, based on changes in RGB values in a region of interest in an image (here, the skin region in the face region). In this case, if the image contains areas such as hair, eyeballs, or the inside of the mouth, the movements of these can become disturbances. However, in this embodiment, the region extraction unit 212 excludes these non-skin regions and extracts the skin region, so the effects of such disturbances can be reduced.

In this case, the pulse wave signal detection unit 213 may perform a predetermined pre-processing on the skin area image, taking into account the individual differences in skin color of each measurement subject and the color of the light source, before detecting the pulse wave signal by rPPG. For example, from each of a plurality of skin area images (i.e., from each frame of a video consisting of skin area images), calculate C(t), which is the spatial average RGB value for each frame. Then, calculate _Cn (t) for each frame by dividing C(t) for each frame by the average value of the values C(t) for all frames. Then, the pulse wave signal detection by rPPG is performed on the _Cn (t) after this pre-processing. This makes it possible to suppress the influence of the individual differences in skin color of each measurement subject and the color of the light source.

Furthermore, pulse wave signal detection unit 213 may further apply a method called POS (Plane-Orthogonal-to-Skin) in detecting the pulse wave signal by rPPG. POS is a method that focuses on the fact that changes in RGB values contain a mixture of pulse wave components and components derived from brightness changes, and projects C(t) (or C _n (t) after the above-mentioned preprocessing), which is the spatial average RGB value, to an axis perpendicular to the direction of the vector in which the brightness change occurs in the RGB space, in order to extract the pulse wave component. This allows pulse wave signal detection unit 213 to detect the pulse wave signal with higher accuracy. Details of POS are disclosed, for example, in Non-Patent Document 1 below.

<Non-Patent Document 1>
W. Wang, et al., "Algorithmic Principles of Remote PPG", [online], IEEE Transactions on Biomedical Engineering, Vol. 64, No. 7, 2017, [Retrieved July 13, 2020], Internet <URL: https://pure.tue.nl/ws/portalfiles/portal/31563684/TBME_00467_2016_R1_preprint.pdf>

The pulse wave signal detection unit 213 detects the pulse wave signal in this manner and outputs the detected pulse wave signal to the waveform correction unit 214.

The waveform correction unit 214 corrects the pulse wave signal input from the pulse wave signal detection unit 213 for measurement of the heart rate by the heart rate measurement unit 215. FIG. 4 is a graph showing correction of the waveform of the pulse wave signal by the pulse wave signal detection unit 213. As shown in FIG. 4(A), the pulse wave signal has a baseline fluctuation due to changes in illumination during shooting, etc., as a premise. Therefore, the waveform correction unit 214 performs trend removal using a moving average filter. For example, the waveform correction unit 214 converts the pulse wave signal so that the average is 0 and the standard deviation is 1, thereby converting it into a Z-score. Next, the waveform correction unit 214 extracts a moving average line with a time window of a predetermined length (for example, 1 second width) from the pulse wave signal as a baseline signal. Then, the waveform correction unit 214 subtracts this baseline signal from the pulse wave signal. Then, the pulse wave signal shown in FIG. 4(A) becomes a trend-removed pulse wave signal as shown in FIG. 4(B). Furthermore, the waveform correction unit 214 may further perform processing such as passing the pulse wave signal through a bandpass filter that takes into account the frequency characteristics of the pulse wave (for example, a bandpass filter of 0.5 [Hz]-5 [Hz]).

The waveform correction unit 214 corrects the pulse wave signal in this manner. The waveform correction unit 214 then stores the corrected pulse wave signal in the pulse wave signal storage unit 252. In other words, the pulse wave signal storage unit 252 functions as a storage unit that stores the pulse wave signal.

The heart rate measurement unit 215 detects the timing of the heart rate in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and a model signal, which is a model of a waveform indicating the heart rate, and measures the heart rate of the subject based on the detected heart rate timing.

The model signal is a signal that corresponds to a model of a waveform that shows an ideal heart rate in rPPG. An example of a waveform that shows an ideal heart rate is a waveform that corresponds to a heart rate in rPPG, and can be a typical heart rate waveform commonly seen in subjects, or a heart rate waveform that is an average of multiple heart rate waveforms.

Figure 5 is a graph showing an example of a model signal used in cross-correlation analysis by the heart rate measurement unit 215. The model signal, which is such a waveform, is created in advance by a user who uses the heart rate measurement system S and stored in the model signal storage unit 253. In other words, the model signal storage unit 253 functions as a storage unit that stores the model signal. The user can refer to the measurement results regarding the heart rate using this model signal and appropriately modify this model signal so that measurements can be performed with greater accuracy.

When performing cross-correlation analysis with this model signal, the heartbeat measurement unit 215 first reads out the pulse wave signal from the pulse wave signal storage unit 252 and cuts it out into a time window of a predetermined length (for example, 2 seconds wide) to be analyzed. The heartbeat measurement unit 215 also converts the cut-out pulse wave signal to be analyzed into a Z-score, and calculates a cross-correlation coefficient by performing cross-correlation analysis with the model signal stored in the model signal storage unit 253. The heartbeat measurement unit 215 then detects the point where the calculated correlation coefficient is high as the timing of the heartbeat. However, at this point, the heartbeat measurement unit 215 merely counts this detected heartbeat timing as a candidate for the timing of the heartbeat.

The heartbeat measuring unit 215 then repeatedly extracts a new analysis target from the pulse wave signal, detects the timing of the heartbeat in this manner, and counts it as a candidate for the timing of the heartbeat. Here, the analysis target is extracted by shifting it by a time window of a predetermined length (e.g., 0.1 second width) so that it overlaps with the part previously analyzed (i.e., partially overlaps with the part previously analyzed). In other words, the analysis target is repeatedly shifted by a time window of a predetermined length (e.g., 0.1 second width) and cross-correlation analysis with the model signal is performed. Then, by repeating such shifts, when all of the pulse wave signals stored in the heartbeat measuring unit 215 have been extracted as analysis targets and cross-correlation analysis has been performed, the repeated cross-correlation analysis ends.

Next, the heartbeat measurement unit 215 measures the heartbeat of the subject based on the number of times the timing of the heartbeat is detected in the repetition of this cross-correlation analysis (i.e., the number of times counted as a candidate for the timing of the heartbeat). FIG. 6 is a graph showing an example of a distribution of the number of detections to explain the measurement of the timing of the heartbeat by the heartbeat measurement unit 215. As an example, as shown in FIG. 6(A), it is assumed that the heartbeat measurement unit 215 has created a distribution of the number of times the timing of the heartbeat is detected in the repetition of the cross-correlation analysis (i.e., the number of times counted as a candidate for the timing of the heartbeat). In this case, the heartbeat measurement unit 215 searches for a peak (e.g., a local maximum value) by performing a peak search on a waveform showing such a distribution.

In Figure 6 (B), the peaks found in this way are shown as "dashed circles." The heartbeat measurement unit 215 selects the peaks found in this way as the measured heartbeat timing. On the other hand, even if a timing has been counted as a candidate heartbeat timing several times, timings that have not been found as peaks and have been counted a small number of times are not selected as measured heartbeat timings and are discarded. This makes it possible to select a timing with a higher likelihood and measure the heartbeat timing of the person being measured with greater accuracy.

The heartbeat measuring unit 215 also outputs the measurement results. For example, the heartbeat measuring unit 215 outputs the measured heartbeat timing of the subject as the measurement result. Alternatively, the heartbeat measuring unit 215 outputs the interval between the measured heartbeat timing of the subject as the measurement result. This output may be, for example, a display on a display included in the output unit 27, an audio output from a speaker included in the output unit 27, printing on paper media from a printing device (not shown) via the communication unit 24, or transmission to another device (not shown) via the communication unit 24. A user who uses the heartbeat measuring system S can use the output measurement results as useful information for applications such as the prevention of heart disease.

Figure 7 is a graph showing an example of the measurement result of the measurement involving the cross-correlation analysis in this embodiment. For comparison, FIG. 7(A) shows the heartbeat interval measured by an electrocardiograph that measures the heartbeat by contacting electrodes with the body of the measurement subject as the correct value. Also, FIG. 7(B) shows the heartbeat interval measured when a simple peak search is performed on the pulse wave signal measured by rPPG as a conventional example. On the other hand, FIG. 7(C) shows the heartbeat interval measured when various correction processes, cross-correlation analysis with a model signal, and selection based on the detection count distribution are performed as described above in this embodiment. Note that the measurement by the electrocardiograph corresponding to FIG. 7(A) and the photography to obtain the image data corresponding to FIG. 7(B) and (C) were performed simultaneously in parallel on the same measurement subject. Therefore, ideally, the measured heartbeat intervals should be the same.

However, when comparing Fig. 7(A) with Fig. 7(B), a very large error occurs. In other words, compared to an electrocardiograph that can measure with high accuracy, it can be seen that a very large measurement error occurs when simply searching for peaks in a pulse wave signal measured by rPPG. On the other hand, when comparing Fig. 7(A) with Fig. 7(C), the results are very close and there is almost no error. In other words, compared to an electrocardiograph that can measure with high accuracy, it can be seen that according to this embodiment, measurement can be performed with similar high accuracy.
In this way, according to the present embodiment, it is possible to perform image-based heartbeat measurement with higher accuracy, similar to an electrocardiograph. In addition, according to the present embodiment, it is possible to perform heartbeat measurement without contacting the subject, and the subject does not need to wear an electrocardiograph, so that it is possible to more easily realize heartbeat measurement.

[Shooting processing]
Next, the flow of the photographing process executed by the photographing device 10 will be described with reference to Fig. 8. Fig. 8 is a flowchart for explaining the flow of the photographing process executed by the photographing device 10. The photographing process is executed in response to an instruction operation to start photographing from a photographer (the subject himself may be the photographer, or another person may be the photographer).

In step S11, the photographing control unit 111 starts generating a plurality of images by controlling the image capturing unit 18 to continuously photograph the measurement subject as a subject.
In step S12, the imaging control unit 111 causes the image data storage unit 251 to store image data corresponding to the multiple images generated in step S11.

In step S13, the shooting control unit 111 judges whether or not to end shooting by the imaging unit 18. Here, shooting ends when a predetermined time has elapsed since shooting in step S11, or when the photographer issues an instruction to end shooting. If shooting is to end, step S13 is judged as Yes, and the process proceeds to step S14. On the other hand, if shooting is not to end, step S13 is judged as No, and the process returns to step S11 and is repeated.

In step S14, the image data transmission unit 112 transmits the image data that the imaging control unit 111 has stored in the image data storage unit 151 to the heart rate measurement device 20. This ends the process. Note that in the figure, it is assumed that the generated image data is stored in the image data storage unit 151, and that the image data stored in the image data storage unit 151 is transmitted all at once after imaging is completed, but as described above, it is also possible to transmit the image data in real time as it is generated by the imaging control unit 111.

By using the photographing process described above, the photographing device 10 can generate multiple images by continuously photographing the subject as a subject, and can transmit image data corresponding to the multiple images generated to the heart rate measuring device 20.

[Measurement model construction process]
Next, the flow of the heartbeat measurement process executed by the heartbeat measurement device 20 will be described with reference to Fig. 9. Fig. 9 is a flowchart illustrating the flow of the heartbeat measurement process executed by the heartbeat measurement device 20. The heartbeat measurement process is executed in response to an instruction operation to start the heartbeat measurement process from a user of the heartbeat measurement system S.

In step S21, the image data acquisition unit 211 acquires image data corresponding to multiple images of the subject being measured, by receiving the image data, transmitted from the imaging device 10.

In step S22, the image data acquisition unit 211 causes the image data storage unit 251 to store the image data acquired in step S21.
In step S23, the area extraction unit 212 extracts a face area from the image data stored in the image data storage unit 251 in step S22.

In step S24, the area extraction unit 212 extracts a skin area from the face area extracted in step S23, and generates a plurality of skin area images.
In step S25, the pulse wave signal detection unit 213 detects a pulse wave signal, which is a signal indicating the pulse wave component of the subject, based on the change in values expressing the color between the multiple skin area images generated in step S24.

In step S26, waveform correction section 214 corrects the pulse wave signal detected in step S25 for measurement related to the heart rate.
In step S27, waveform correcting unit 214 stores the pulse wave signal corrected in step S26 in pulse wave signal storage unit 252.

In step S28, the heartbeat measurement unit 215 detects the timing of the heartbeat in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal stored in the pulse wave signal storage unit 252 in step S27 and a model signal that is a model of a waveform indicating the heartbeat.

In step S29, the heartbeat measurement unit 215 measures the heartbeat of the subject based on the number of times the heartbeat timing was detected in step S28 (i.e., the number of times counted as a candidate for the heartbeat timing).

In step S30, the heart rate measurement unit 215 outputs the measurement results obtained in step S29. This ends the process.

According to each of the processes described above, as explained with reference to FIG. 7, it is possible to perform image-based heart rate measurements with a higher degree of accuracy, similar to an electrocardiograph. In addition, according to each of the processes described above, it is possible to perform heart rate measurements without contacting the subject, and the subject does not need to wear an electrocardiograph, making it possible to more easily perform heart rate measurements.

[Modification]
Although the embodiment of the present invention has been described above, this embodiment is merely an example and does not limit the technical scope of the present invention. The present invention can take various other embodiments and can be modified in various ways, such as omissions and substitutions, without departing from the gist of the present invention. In this case, these embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents.

As an example, the device configuration of the heart rate measurement system S in the above-described embodiment is merely an example and can be modified as appropriate. For example, in the above-described embodiment, the image capture device 10 and the heart rate measurement device 20 are realized as separate devices, but the image capture device 10 and the heart rate measurement device 20 may be realized as an integrated device. In addition, for example, the heart rate measurement device 20 may be realized in a distributed manner using multiple computers, such as a cloud system.

[Configuration example]
As described above, the heart rate measuring system S according to this embodiment includes the image data acquiring unit 211 , the pulse wave signal detecting unit 213 , and the heart rate measuring unit 215 .
The image data acquisition unit 211 acquires a plurality of images generated by continuously photographing the subject as a subject.
Pulse wave signal detection section 213 detects a pulse wave signal, which is a signal indicating the pulse wave component of the subject, based on the change in values expressing the colors between a plurality of images.
The heart rate measurement unit 215 detects the timing of the heart rate in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and a model signal, which is a model of a waveform indicating the heart rate, and performs measurements of the subject's heart rate based on the detected timing of the heart rate.
In this way, the heart rate measurement system S detects a pulse wave signal from an image of the measurement subject. The heart rate measurement system S then detects the timing of the heart rate by analyzing the cross-correlation between the pulse wave signal and a model signal, and performs measurements related to the heart rate based on this.

As a result, the heart rate measurement system S can perform more accurate measurements of the heart rate based on the image. Therefore, the heart rate measurement system S can measure the heart rate of the subject even if there is an event that affects the measurement, such as a change in the orientation of the subject (e.g., a change in the face direction) or a change in brightness due to the surrounding environment. Furthermore, the heart rate measurement system S can measure the heart rate of the subject without contacting the subject, and the subject does not need to wear a wearable device, etc., making it possible to more easily measure the heart rate.

The heartbeat measurement unit 215 repeatedly detects the timing of the heartbeat by shifting the portion of the pulse wave signal to be analyzed for cross-correlation so that it overlaps with the portion previously analyzed, and then analyzing the cross-correlation.
The heartbeat measuring section 215 measures the heartbeat of the subject based on the number of times the heartbeat timing is detected in the repetition.
This allows multiple analyses to be performed on the same portion of the pulse wave signal, improving the likelihood of the measurement and enabling more accurate measurement of the subject's heart rate.

The heart rate measurement unit 215 is
A distribution along a time series of the number of times the heartbeat timing is detected is obtained, and a peak search is performed on the obtained distribution, so that heartbeat timings that are detected infrequently are excluded from the heartbeat timings used when performing measurement.
This makes it possible to discard heartbeat timings that are detected infrequently, thereby improving the likelihood of measurement and enabling more accurate measurement of the subject's heartbeat.

The plurality of images are images including a face portion of the measurement subject as a subject,
The heart rate measuring section 215 measures the interval between the heart rates of the subject as a measure related to the heart rate.
This makes it possible to measure the heartbeat interval of a person who is the subject based on the face image of that person.

[Realization of functions through hardware and software]
The function of executing the series of processes according to the above-mentioned embodiment can be realized by hardware, software, or a combination of these. In other words, it is sufficient that the function of executing the series of processes described above is realized in any one of the heartbeat measurement systems S, and there is no particular limitation on how this function is realized.

For example, when the function of executing the above-mentioned series of processes is realized by a processor that executes arithmetic processing, the processor that executes this arithmetic processing includes not only those that are composed of various processing devices alone, such as single processors, multiprocessors, and multicore processors, but also those that combine these various processing devices with processing circuits such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays).

For example, when the function of executing the above-mentioned series of processes is realized by software, the program constituting the software is installed on a computer via a network or a recording medium. In this case, the computer may be a computer with dedicated hardware built in, or a general-purpose computer (e.g., a general-purpose electronic device such as a general-purpose personal computer) that can execute a specified function by installing a program. The steps of writing the program may include only processes that are performed chronologically according to the order, but may also include processes that are executed in parallel or individually. The steps of writing the program may be executed in any order within the scope of the gist of the present invention.

A recording medium on which such a program is recorded may be provided to a user by being distributed separately from the computer main body, or may be provided to a user in a state in which it is already installed in the computer main body. In this case, the storage medium distributed separately from the computer main body is composed of a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. The optical disk is, for example, a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc), or a Blu-ray (registered trademark) Disc. The magneto-optical disk is, for example, an MD (Mini Disc). These storage media are, for example, loaded into drive 28 in FIG. 3 and installed in the computer main body. The recording medium provided to the user in a state where it is preinstalled in the computer main body is composed of, for example, ROM 12 in FIG. 2, ROM 22 in FIG. 3, storage unit 15 in FIG. 2, or a hard disk included in storage unit 25 in FIG. 3, on which the program is recorded.

10 Photographing device, 20 Heart rate measuring device, 11, 21 CPU, 12, 22 ROM, 13, 23 RAM, 14, 24 Communication unit, 15, 25 Memory unit, 16, 26 Input unit, 17, 27 Output unit, 18 Imaging unit, 28 Drive, 111 Photographing control unit, 112 Image data transmission unit, 151, 251 Image data memory unit, 211 Image data acquisition unit, 212 Region extraction unit, 213 Pulse wave signal detection unit, 214 Waveform correction unit, 215 Heart rate measuring unit, 216 Measurement unit, 252 Pulse wave signal memory unit, 253 Model signal memory unit, S Heart rate measuring system

Claims (4)

an image acquisition means for acquiring a plurality of images generated by continuously photographing a subject;
a pulse wave signal detection means for detecting a pulse wave signal which is a signal indicative of a pulse wave component of the subject based on a change in a value expressing a color between the plurality of images;
a measuring means for detecting a timing of a heartbeat in the pulse wave signal by analyzing a cross-correlation between the pulse wave signal and a model signal which is a model of a waveform indicative of a heartbeat, and for measuring the heartbeat of the subject based on the detected timing of the heartbeat ;
Equipped with
The measuring means is
a portion of the pulse wave signal to be analyzed for cross-correlation is shifted so as to overlap with a portion previously analyzed, and then the cross-correlation is analyzed to detect the timing of the heartbeat, and the process is repeated;
A distribution of the number of times the heartbeat timing is detected in the repetition is obtained along a time series.
A peak search is performed on the distribution obtained, and the heartbeat timings that have not been detected as peaks are discarded because the number of detections is relatively small compared to the heartbeat timings that have been detected as peaks;
measuring the heartbeat of the subject based on the timing of the heartbeat detected as a peak;
A heart rate measuring system comprising: the plurality of images are images including a face portion of the measurement subject as a subject;
The measurement means measures an interval between heartbeats of the subject as the measurement related to the heartbeat.
2. The heart rate measuring system according to claim 1 . A method for measuring a heart rate performed by a computer, comprising:
an image acquiring step of acquiring a plurality of images generated by continuously photographing a person to be measured as a subject;
a pulse wave signal detection step of detecting a pulse wave signal which is a signal indicating a pulse wave component of the subject based on a change in a value expressing a color among the plurality of images;
a measuring step of detecting a timing of a heartbeat in the pulse wave signal by analyzing a cross-correlation between the pulse wave signal and a model signal which is a model of a waveform indicative of a heartbeat, and measuring the heartbeat of the subject based on the detected timing of the heartbeat;
Including ,
In the measuring step,
a portion of the pulse wave signal to be analyzed for cross-correlation is shifted so as to overlap with a portion previously analyzed, and then the cross-correlation is analyzed to detect the timing of the heartbeat, and the process is repeated;
A distribution of the number of times the heartbeat timing is detected in the repetition is obtained along a time series.
A peak search is performed on the distribution obtained, and the heartbeat timings that have not been detected as peaks are discarded because the number of detections is relatively small compared to the heartbeat timings that have been detected as peaks;
measuring the heartbeat of the subject based on the timing of the heartbeat detected as a peak;
A method for measuring heart rate comprising: An image acquisition function that acquires a plurality of images generated by continuously photographing a subject;
a pulse wave signal detection function that detects a pulse wave signal that is a signal indicative of a pulse wave component of the subject based on a change in a value expressing a color between the plurality of images;
a measurement function that detects the timing of a heartbeat in the pulse wave signal by analyzing the cross-correlation between the pulse wave signal and a model signal that is a model of a waveform indicative of a heartbeat, and measures the heartbeat of the subject based on the detected timing of the heartbeat;
This is realized on a computer .
The measurement function is
a portion of the pulse wave signal to be analyzed for cross-correlation is shifted so as to overlap with a portion previously analyzed, and then the cross-correlation is analyzed to detect the timing of the heartbeat, and the process is repeated;
A distribution of the number of times the heartbeat timing is detected in the repetition is obtained along a time series.
A peak search is performed on the distribution obtained, and the heartbeat timings that have not been detected as peaks are discarded because the number of detections is relatively small compared to the heartbeat timings that have been detected as peaks;
measuring the heartbeat of the subject based on the timing of the heartbeat detected as a peak;
A heart rate measuring program comprising:

Images