Keywords

1 Introduction

Due to the conflict between the image theory of stereoscopic display and the human vision system, the discrepancy between vergence and accommodation also called vergence-accommodation conflict (VAC) is inevitable. Viewer feels visual discomfort and pain due to functional overload of brain which is an important cause driven by VAC [1]. Along with strong depth disparity and other external factors, visual fatigue will always exit. Subjective evaluation is an effective way to detect and determine the extent of different symptoms for visual fatigue [2]. However, the sensitivity of subjects to visual fatigue is variable and the results may be biased. Objective methods are utilized to measure a precisely level of the fatigue, which can be roughly divided into two methods: (1) analyzing 3D videos and (2) measuring biological signals.

Firstly, analyzing characteristics of current 3D display and the external environment to estimate visual fatigue [3,4,5,6]. Analyzing the inherent characteristics of 3D techniques show stable and objective results of visual fatigue measurement. However, the amount of visual fatigue could be changed depending on the viewer and the viewers status. Secondly, biological signals are used for the reliable assessment of products by providing objective measurements, even for slight changing external stimuli. So it is suitable for measuring factors of the visual fatigue level of individual viewers. Some previous researches focused on blinking rate, Electroencephalogram (EEG), cognitive functions (Event-related potential: ERP), Punctum Maximum Accommodation (PMA), electrocardiogram (ECG) and photoplethysmogram (PPG) etc. [7,8,9]. Zou et al. assessed three EEG activities, \(\theta \), \(\alpha \) and \(\beta \) during a monotonous and repetitive random dot stereogram (RDS) based task in a conventional stereoscopic 3D display. Results of EEG data showed stable of \(\theta \) activity and a significant increase of \(\alpha \) activity, and a significant decrease of \(\beta \) activity over time [10]. But the results have a little conflict with CX Chen’s results in his experiments [11], which show the energy in \(\alpha \) and \(\beta \) frequency bands significantly decreased. It can be concluded that biological signals have unstable results and strongly affected by task type.

In our research, we chose gravity frequency of power spectrum (GF), power spectral entropy (PSE) and ratio algorithms (\(\alpha +\theta )/\beta \) (R) of EEG as biological features, which showed remarkable correlation with fatigue status after compared between time domain, frequency domain and non-linear dynamic analysis. We use the portable EEG acquisition equipment, focused on EEG measurement at the posterior and frontal sites, the former is supposed to be related to visual processing, and the latter is to cognitive functions (such as attention and executive functions). Using three electrodes at the posterior and frontal (P8, F3, F7 with p-value  0.05) to detect visual fatigue is enough and exact by correlation test and t-test. The portable device overcome limitations of poor portability, operational complexity and high cost.

Based on the above advantages and disadvantages, it is suitable to combine two kinds of objective features (physical feature of 3D display: VAC, biological features of EEG: GF, PSE and R) with one subjective feature (Subjective evaluation: questionnaire score (QS) parameters) to design a Fuzzy Fusion of Visual Fatigue (FFVF) model. The multi-modalities FFVF by using fuzzy theory as fusion method is a novel way to measure visual fatigue more accurately and easily, the simple and convenient system could provide elicitation to commercialization and availability.

Fig. 1.
figure 1

Human vision system

Full size image

2 Methods and Experiment

2.1 Visual Fatigue Prediction

Parameters of the 3D display are measured before experiment, thus we defined analyzing 3D videos as visual fatigue prediction (VFP). Prior researches show effect factors of disparity magnitude such as display methods, viewing distance, display size, video resolution, both are closely related to the accuracy of visual system [12]. As following equation, the values of \(Disp_{cm}\) will change if the resolution and the size of showing device is different.

$$\begin{aligned} Disp_{cm} = Disp_{pixel}\times \frac{Display\quad width}{Resolution\quad width} \end{aligned}$$
(1)

The amount of VAC was obtained by the disparity of viewpoint, as depicted in Fig. 1, \(\alpha \) represents the viewpoint angle on the display screen. \(\beta \) and \(\gamma \) represent viewpoint angles in the positive and negative region separately. Then the VFP is defined as follows:

$$\begin{aligned} VAC = VFP = \left\{ \begin{matrix} \alpha -\gamma , &{} \mathrm {if\, crossed \,disparity} \\ \beta - \alpha , &{} \mathrm {if \,uncrossed \,disparity} \end{matrix}\right. \end{aligned}$$
(2)

We use the disparity magnitude \(Disp_{pixel}(x,y)\) to measure viewpoint angles \(\alpha \), \(\beta \) and \(\gamma \). By using (1) we transform \(Disp_{pixel}\) to \(Disp_{cm}\), then there are following relationship:

$$\begin{aligned} \left\{ \begin{matrix}d_{p}:\frac{l}{2}=d_{p}-d_{v}:\frac{D_{cm}(x,y)}{2} &{} \\ d_{n}:\frac{l}{2}=d_{v}-d_{n}:\frac{D_{cm}(x,y)}{2} &{} \end{matrix}\right. \end{aligned}$$
(3)

where \(d_{v}\) is the viewing distance, \(d_{p}\) is the positive perceived distance and \(d_{n}\) is negative perceived distance, and l is inter-ocular distance. Based on the trigonometric ratio, we substitute parameters in (2) as follows:

$$\begin{aligned} VAC=VFP=\left\{ \begin{matrix} \tan ^{-1}\left( \frac{2d_{v}}{l} \right) - \tan ^{-1}\left( \frac{2d_{v}}{l+D_{cm}(x,y)} \right) , &{} \mathrm {if\, crossed \,disparity} \\ \tan ^{-1}\left( \frac{2d_{v}}{l-D_{cm}(x,y)} \right) -\tan ^{-1}\left( \frac{2d_{v}}{l} \right) ,&{} \mathrm {if \,uncrossed \,disparity} \end{matrix}\right. \end{aligned}$$
(4)

2.2 Visual Fatigue Obtain

Subjects. 15 adults took part in the experiment: 7 females, 8 males; mean age 25.31 (SD = 2.81). All subjects were in good physical health and none of them were reported of any visual disease. These subjects were notified have an excellent sleep (6–8 h) before the experiment. For insure good physically and mentally condition, wine, tea, coffee and drugs were prohibited. Before beginning experiment all of them gave their informed written consent to take part in and were briefed on the purpose of the experiment and experimental procedures. The test during 9:00–11:00 AM in order to control the potentially physiological rhythms and this period shown more awareness.

Apparatus. Stereoscopic images were shown in full HD resolution (1080p) on a 67.57 cm LG D2743 (Width * High = 59 cm * 33 cm), an active display, subjects wear matching shuttered glasses. Subjects sit in a soft chair and quite environment with comfortable temperature, the distance between subjects and screen is 130–150 cm. The setup and environment of the experiment are show in Fig. 2. EEG signals are acquired using portable device Emotive EPOC on the micro-voltage levels, this headset device has two reference nodes (CMS and DRL) and 14 other electrodes.

Fig. 2.
figure 2

Experiment environment

Full size image

Measures. In order to make the system has a wide range of adaptability, the experiment include three trails which show the different types of 3D movies randomly: documentary (Ocean Wonderland), action (Avengers) and animation (Frozen). In each trail, subjects watching time is 20 min, before and after watching could have 2 min as relax. At the beginning of the experiment, subjects need completed the questionnaire, then, operator help subject wear Emotive, and subjects close eyes and measured brain signals in 5 min if everything is ready, as well as after finished watching the 3D movie. During the recorded, all subjects tried to relax and avoid unnecessary movements. The proposed experiment process of data acquisition and analysis are show in Fig. 3.

Fig. 3.
figure 3

The experiment procedure

Full size image

The subjective evaluation questionnaire consisted of five questions on eye fatigue and five questions on body fatigue, and participants answered on a five-point rating scale: 1: Very severe, 2: Severe, 3: Moderate, 4: Comfortable, 5: Very comfortable. Table 1 shows the fatigue symptoms in the questionnaire. Our proposed experiment composition related to participants and the questionnaire was designed following ITU recommendation [13].

Table 1. The fatigue symptoms of questionnaire
Full size table

2.3 Data Processing

The raw EEG data of each trial for each subject was processed with a 50 Hz notch filter and 0.5–30 Hz band filter (IIR digital filter). After that choose 1 min EEG data without obvious interference and discontinuity in the close eyes period to the following analysis [14]. Then the gravity frequency of power spectrum (GF) was calculated by:

$$\begin{aligned} GF=\sum _{w=w_{1}}^{w_{2}}(p(\hat{w})\cdot w)/\sum _{w=w_{1}}^{w_{2}}p(\hat{w}) \end{aligned}$$
(5)

where \(p(\hat{w})\) is power spectrum, \(w_{1}\), \(w_{2}\) represent frequency upper and lower limits respectively, namely from 0.5–30 Hz. Then the power spectral entropy (PSE) is given by:

$$\begin{aligned} PSE=-\sum _{i}p_{n}(i)\cdot \log _{2}(p_{n}(i)) \end{aligned}$$
(6)

where \(p_{n}(i)\) represents the probability density distribution of power spectrum at \(w_{i}\). \(\sum _{i}p_i=1\). Every segment data was used band-pass filter separately, then acquired four frequency wavebands. The ratio algorithm (\(\alpha +\theta )/\beta \) (R) could be calculated by:

$$\begin{aligned} R=(p_{\alpha } +p_{\theta })/ +p_{\beta } \end{aligned}$$
(7)

where \(p_{\alpha }\), \(p_{\beta }\) and \(p_{\theta }\) represent the power spectrum of \(\alpha \), \(\beta \) and \(\theta \) respectively.

2.4 Fuzzy-Fusion Method

The fundamental purpose of fuzzy theory method is to define an uncertain state based on the relationship between the characteristic of datasets [15]. Since a defined uncertain state includes information on the relationship between input datasets, the results of defuzzification can be used as a weight value. The entire process of obtaining a weight value to each input factors using fuzzification and defuzzification has five steps as depicted in Fig. 4.

Fig. 4.
figure 4

Process flow of fuzzy theory

Full size image

The first step, drawing a fuzzy rule table. The fuzzy rule table describes the characteristic of input data sets and defines the level of fuzzy output depending on the relationship between the input and output. In this study, we perform a quality measurement for five factors: GF, PSE, R, VFP and QS. The two features (\(F_{1}\) and \(F_{2}\)) are extracted from the corresponding values of each factor, they are used as the inputs for the fuzzy system to produce the quality (weight) values. Each factor indicates the variation of EEG, VAC and Questionnaire score (QS) before and after watching 3D display. For the EEG feature we choose the difference value of the first three electrodes as \(F_{1}\) and \(F_{2}\) after using Pearson correlation analysis results. The membership function and rules is given by operators control action and knowledge, we adopted the triangular membership function according to processing speed and complexity of problem [16] and applied center of gravity (COG) as defuzzification methods. After acquired weight values of each modality, we normalized the weight values as follows:

$$\begin{aligned} W_{i}=\frac{w_{i}}{\sum _{k=1}^{n}w_{EEG_{k}}+w_{VFP}+w_{QS}} \end{aligned}$$
(8)

where i is \(EEG_{k}\), VFP and QS, k is GF, PSE and R. Finally, we proposed visual fatigue evaluation system obtain one final output by combining EEG, VFP and QS with a corresponding weight is given as follows:

$$\begin{aligned} FFVF=\sum _{k=1}^{n}(EEG_{k}\times W_{k})+VFP\times W_{VFP}+QS\times W_{QS} \end{aligned}$$
(9)

3 Results and Analysis

3.1 VFP Analysis

In the above mentioned formulation (1), by programming in Visual Studio 2013, we executed sum of squared differences (SSD) algorithm to calculate the disparity map in each frame of video segments. After equalizing acquired VFP factors, the linear relevant fitting methods are applied to verify our proposed VFP. As shown in Fig. 5, VFP shows a high fitting coefficient (\(R^{2}\) = 0.8964, adjusted \(R^{2}\) = 0.8816) in linear relevant fitting with QS.

Fig. 5.
figure 5

Linear relevant fitting of Questionnaire Score (QS) and Visual Fatigue Prediction (VFP)

Full size image
Table 2. The Pearson correlation of GF, PSE, R
Full size table

VFP produces more stable results than biological signals due to the results of a certain value are affected by only one parameter. On the other hand, the results of each biological signal composite all possibility of viewers condition. Thus, the proposed combine VFP factors in FFVF method will increase reliability and maintain stability.

3.2 EEG Signal Analysis

GF and PSE show significantly decreases in several brain regions after long time of watching 3D display, meanwhile R values show significantly increase. In order to select the best representative channel of each EEG feature, we calculated the Pearson correlation between each variation of factors and Questionnaire Score (QS) respectively. Table 2 shows the top three r-value electrodes of GF, PSE and R respectively, strength attachment electrodes of the posterior and the frontal have higher variation than others, except temporal cortex (T7, T8) which is irrelevant with visual processing and prefrontal cortex (AF3, AF4) which is too close to the eyes and therefore easily influenced by eye movements.

3.3 Fusion Process and Results

We conduct a quality measurement for three features to detect a more accurate visual fatigue level. For EEG signals, if the tendency of difference value between before and after watching 3D display in the three nodes are consistently similar, that means we acquired brain signal is stable and representative, so the data is acceptable as consequence of \(F_{1}\) and \(F_{2}\). Instead, if the quality of the acquired EEG signal is poor or variation, which is frequently caused by EEG signal noise related to the movement of the head or facial muscle. For example, the differences among the values of the P8, F3 and FC5 nodes are considerable, which makes the influence of the \(F_{1}\) and \(F_{2}\) of the EEG signal significant. Thus, the two input of GF, PSE and R factors are negative proportion with output data.

For the VAC features, VFP is leveraged for penalizing the values over the threshold of a visual comfortable zone, the stronger stereoscopic effect the higher visual fatigue level, so the maximum value of VFP (\(F_{1}\)) is positive proportion with output data, whereas the minimum value of the VFP (\(F_{2}\)) is negative proportion.

For the subjective evaluation features, in the case of a higher users preference and greater number of users watching 3D movies, we can assume that the user is more accustomed to 3D content and he (or she) can perform a more accurate and objective QS. That is because body and mind will fight it with procrastination due to dislike. So the two input of QS factors is both positive proportion with output data. The detail information of two input features as indicated in Table 3.

Table 3. Two input features for calculate the weight values
Full size table

After acquired all weight of each factor, utilizing function (9) then obtain the finally visual fatigue level of the FFVF model. The proposed method for measuring eye fatigue was implemented by MATLAB program (MATLAB R2014b). The final values of visual fatigue using proposed FFVF were calculated Pearson correlation with subjective evaluation, the results showed r = 0.9676 and p = \(3.8\times 10^{-9}\).

4 Discussion

Our experiment demonstrated that EEG power spectrum will change with the difference of alertness and vigilant state, the subjects’ before state shows a more sober and vigilant state than after watching 3D display. The comparisons shown in Fig. 6(a–c) by paired-t-test.

Fig. 6.
figure 6

Comparison of each channel of GF, PSE, R and subjective evaluation between before and after watching (*\(p\le 0.05\), **\(p\le 0.005\))

Full size image

Figure 6(a) indicate that GF parameters significantly decreased after watching 3D display, especially in FC5, O1, P8 channel. Similarity, as shown in Fig. 6(b), the PSE factors also decreased which represent the complexity of the disorder time sequence signals and the level of chaos of multi-frequency components. However, as shown in Fig. 6(c), the R factors have clearly increased which related to the mental alertness level, that was more reliable fatigue indicator since it showed a distinct indication of increasing fatigue as the scale between the slow wave and fast wave activities increased. Figure 6(d) is the comparison of subjects answer to their condition between before and after watching 3D movie, which indicates that after 1 h watching 3D movie can lead to fatigue on mental and physiology.

Figures 5 and 6 demonstrate that validity of type and number of proposed parameters necessity to build this system. Furthermore, in terms of performance, our proposed model verified \(R^{2}\) = 0.9362, adjusted \(R^{2}\) = 0.9313 (\(p< 0.001\)) showed better results than multiple linear regression analysis as Table 4.

Table 4. Comparison of multiple linear regression and fuzzy fusion theory
Full size table

5 Conclusion

In this paper, we proposed a novel visual fatigue evaluation system by using fuzzy theory to combine EEG signals and physical features of 3D display. EEG signals can reflect human physical condition more accurately than other biological signals, we chose three EEG features which change obviously during watching 3D display. Analysis of the production theory of 3D display and vergence-accommodation conflict (VAC) indicated a direct relationship to the fundamental cause of visual fatigue, then we acquired objective physical features by equalizing VAC parameters in our experiment video. Comparing with prior researches, our work has following advantages: (1) With analyzing the brain regions associated with visual fatigue, we verified 3 most relevant electrodes to visual fatigue, which are chosen as the system input is enough. Thus our work reduced the computational complexity and improved the accuracy. (2) Exploration of the source of watching fatigue, obtaining one final value for the variation of visual fatigue level by two types features fusion, meanwhile, considering the subjective opinion, the system also includes subjects evaluation. (3) Proposed weight values of each feature based on the fuzzy theory, the dynamic evaluation overcame external and internal factors change, adjusted to different subjects and different watching conditions.

The evaluation system give visual fatigue level with high accuracy and stability, but in the way of fusion method of fuzzy theory, we determined membership function and rules by expert experience and prior knowledge. For future work, we should consider set system parameters automatically by combine neurol control to fuzzy control. Moreover, considering the integration of a variety of patterns, further optimize the FFVF system performance for real-time visual fatigue monitoring.