Keywords

1 Introduction

Present-day society relies very much upon AI (artificial intelligence) and HCI (human-computer interaction). We are asked to prepare suitable environments so that such present-day science and technology can be properly applied to increase the quality of human life. It is difficult but interesting how we can adjust the differences among individual characters, personalities and abilities when we develop some systems and/or devices relied on AI and HCI.

BCI (brain-computer interface) is one of the most important fields of HCI. In the design of BCI systems, brain signals distributed in a certain and proper range are often regarded as the identical output from the brains. However, there exist some cases where such rough measurement of signals and data-collection are not good enough. Phenomena called “Gestaltzerfall” are examples of such cases. For example, Gestaltzerfall in the recognition of a letter is a phenomenon that a human does not recognize the meaning of the letter when he/she keeps watching the letter for a while. For such a case as a Gestaltzerfall phenomenon, the output signals from the brain may often go beyond their expected range. It is usually difficult for a BCI system to cope properly such a case.

In this paper we examine the following hypotheses:

  • Hypo1: It is possible to measure the occurrence of Gestaltzerfall in the letter recognition by ERPs.

  • Hypo2: It is possible to grasp the features of Gestaltzerfall and to understand the process getting to Gestalzerfall in the letter recognition by observing the changes of ERPs.

From our investigation we predict that Gestaltzerfall in the letter recognition occurs when high dimensional brain functions, which are related to the pattern recognition mechanism in the brains, do not work well [2, 3]. Furthermore, we confirm that ERPs [4] reflect somewhat significantly the recognition and judgement potentials as well as the visual evoked potentials for stimuli. Especially, ERPs are useful signals to measure the brain status caused by the high dimensional brain functions.

If the hypotheses listed above are affirmatively verified, then we would be able to design better BCI systems by considering Gestaltzerfall.

2 Methods

2.1 Experimental Methods

Since individual differences of ERPs are relatively large, the experiments were carried out by “single-case experimental design [5]”.

The subject is a right-handed and 22 years old man. The experiments were carried out in the laboratory of the first author at Hakuoh University. We use 92 kinds of stimuli in the experiments. Each stimulus is one of 46 “Hiraganas” (Japanese phonetic symbols) in one of the two different fonts as shown in Fig. 1 (Ms-gothic and MS-Mincho). Each stimulus is displayed 40 times in a CRT display of 19 inches for one second. The time interval between two consecutive stimuli is 10 ms and the white screen is displayed during each interval.

Fig. 1.
figure 1

Stimuli in the experiment

Full size image

A subject watches a stimulus, and inputs “0” or “1” by the keyboard. If here cognizes the meaning of the letter displayed in the screen, then he inputs “0”, and otherwise inputs “1”.

The single polar and eight channels of the “International 10 – 20 method” are used for the measurement of EEGs (electroencephalograms). The positions of the measurement are at C3, C4, Cz, and Pz. The base is A1 that is connected to A2. The sampling frequency for the A/D converter is 1 kHz.

2.2 Preprocess of EEGs

The recorded EEGs are sent to an adaptive filter [11, 12]. For the proper function of the adaptive filter, we need to determine its cut-off frequencies as a function of time. This means that the numbers of data supplied to the adaptive filter are given as a function g(t) of time (0 ≦ t ≦ 000 [ms]). From the previous research, g(t) is in the form k(1 – e – t/c). As shown in Fig. 2, we adopt that k = 50 and c is 100 [ms]. The data after the adaptive filter are normalized by taking the standard deviation of the averaged waveforms of EEGs. In this way, we obtain 100 repetitious data (EEGs) D = {x 1(t), x 2(t), …, x 40(t)} (t = 1, 2, …, 1000). Then ERPs are derived from the normalized data of EEGs by the AM (Averaging Method) and the m-DSAM (multi-Data Selecting Averaging Method) [6].

Fig. 2.
figure 2

The numbers of data sent to the adaptive filter, g(t)

Full size image

2.3 The m-DSAM

Let x i (t) be a filtered and normalized data, where i is a data index and t is the latency (the time delay from the time when a letter is displayed). Let D be a data set of x i (t), and D before and D after be data subsets of D, which denotes “before Gestaltzerfall” and “after Gestaltzerfall” respectively. These subsets are determined by the inputs of the subject. In the following steps, D denotes D after or D before. The m-DSAM is a sequence of the calculation of the following steps:

  • Step 1: For each i, x i (t) is transformed into two binary sequences \( b_{i}^{+} (t)\) and \( b_{i}^{-} (t)\) by the following formulae with threshold value L (in this paper, from the signal-noise ratios we determine L = 0.5):

$$ b_{i}^{ + } (t) = \left\{ \begin{aligned} & 1\,\,\,\,\,if\,\,\,x_{i} (t)\,\, > \,\,L\, \\ & 0\,\,\,\,if\,\,\,x_{i} (t)\,\, \le \,\,L \\ \end{aligned} \right.,\,\,\,\,\,\,\,\,\,\,b_{i}^{ - } (t) = \left\{ \begin{aligned} & 0\,\,\,\,\,if\,\,\,x_{i} (t)\,\, \ge \,\, - L\, \\ & 1\,\,\,if\,\,\,x_{i} (t)\,\, < \,\, - L \\ \end{aligned} \right.. $$

  • Step 2: The sum B +(t) of all \( b_{i}^{+} (t)\) and the sum B −(t) of all \( b_{i}^{-} (t)\) are calculated.

  • Step 3: The maximum value MB + of B +(t) around the latency of a positive peak P is found. The minimum value MB − of B −(t) around the latency of a negative peak N is found, too. Let T P and T N be the latencies such that B +(T P) = MB + and B −(T N) = MB −, respectively.

  • Step 4: We find the subsets D P and D N of D such that

$$ D_{\text{P}} = \left\{ {x_{i} (t)|x_{i} (T_{\text{P}} ) > L,\,\,x_{i} (t) \in D} \right\},\,\,\,\,D_{\text{N}} = \left\{ {x_{i} (t)|x_{i} (T_{\text{N}} ) < - L,\,\,x_{i} (t) \in D} \right\} $$

  • Step 5: We calculate

$$ ERP_{\text{P}} (t) = \frac{1}{{n_{\text{P}} }}\sum\limits_{{x_{i} (t) \in D_{\text{P}} }}^{{}} {x_{i} (t)} ,\quad ERP_{\text{N}} (t) = \frac{1}{{n_{\text{N}} }}\sum\limits_{{x_{i} (t) \in D_{\text{N}} }}^{{}} {x_{i} (t)} , $$

where n p and n N are the numbers of elements in D p and D N, respectively.

  • Step 6: For each peak P or N, we estimate ERP P(t) or ERP N(t), by the following formulae (a normal distribution determined by a certain algorithm [2]):

$$ ERP_{\text{Q}} (t) \approx f_{\text{Q}} (t) = ( - 1)^{h(Q)} w_{\text{Q}} \frac{1}{{\sqrt {2\pi } s_{\text{Q}} }}\exp \left( { - \frac{{(t - T_{\text{Q}} )^{2} }}{{2s_{\text{Q}}^{2} }}} \right) $$
(1)

where Q is a P or an N, h(Q) is a function such that if Q = P then h(Q) = 0, and otherwise, h(Q) = 1, w Q is the amplitude of peak Q, s Q is the standard deviation, and T Q is the latency of Q.

  • Step 7: We calculate

\( ERP_{\text{m - DSAM}} (t) = \frac{1}{{\bar{n}}}\sum\limits_{\text{Q}}^{{}} {n_{\text{Q}} f_{\text{Q}} (t)} \), where \( \bar{n} \) is the average of all \( n_{\text{Q}} \)’s.

3 Results

3.1 Numbers of Repetition Until Gestaltzerfall Occurs

The Numbers of repetition of stimuli until the subject recognized Gestaltzerfall are shown in Table 1. The maximum and minimum numbers of repetitions are the same between both fonts, respectively. However, there was a significant difference of average between the number of repetitions in MS-Gothic font and MS-Mincho font (t 45 =  – 4.259, p < 0.001).

Table 1. Numbers of repetitions until Gestaltzerfall occurred
Full size table

3.2 ERPs Calculated by AM and Distribution of Amplitude by m-DSAM

ERPs obtained by AM are shown in Fig. 3. The horizontal axis is time after a stimulus is displayed, and the vertical axis shows average of amplitude. The curves show ERPs before the subject recognized Gestalzerfall, and the dashed curves represent after the subject recognized Gestaltzerfall. Each number in parentheses shows the number of averaging operations. The dashed curves of “Gothic_after” includes six potentials called P1, N1, P2, N2, and P3. In both curves of “Gothic_before” and “Mincho_before”, P2 and N2 do not appear clearly. Since watching a stimulus and judging the occurrence of Gestaltzerfall are easy tasks, visual evoked potentials P1 and N1 appear and P3 is also observed. P3 is concerned with recognition and judgement.

Fig. 3.
figure 3

ERPs calculated by AM (Position: Cz)

Full size image

The averages of B +(t) and B −(t) for D before and D after are shown in Fig. 4. The solid curves represent the averages of B +(t), and the doted curves are the averages of B −(t). P1, N1, P3 and N3 appear clearly, and the frequencies of P2 and N2 are very small.

Fig. 4.
figure 4

Average of B +(t), B −(t) for “before” and “after” Gestaltzerfall (Position: Cz, Number of averaging operations: the same numbers in Fig. 3)

Full size image

Dynamic changes of ERPs for a stimulus “” (in font MS-Gothic font) are depicted in Figs. 5, 6, and 7. Each figure includes 10 curves. Each curve is obtained by the average of consecutive eleven EEGs. The subject recognized Gestaltzerfall at the 14th repetition. Therefore 9th curve includes one EEG in Gestaltzerfall, 10th curve includes two EEGs in Gestaltzerfall, and so on. By the repetitions, the amplitudes of P3 become smaller and the latencies of P3 become shorter in Fig. 6, while the latencies of P3 become later in Fig. 7 than those of P3 in Fig. 5.

Fig. 5.
figure 5

The first 11-point moving averaged ERPs in the experiment (Position: Cz, letter: )

Full size image
Fig. 6.
figure 6

The second 11-point moving averaged ERPs in the experiment (Position: Cz, letter: )

Full size image
Fig. 7.
figure 7

The last 11-point moving averaged ERPs in the experiment (Position: Cz, letter: )

Full size image

3.3 Parameters of AM and m-DSAM

We calculate the latencies and amplitudes of ERPs obtained by AM, and calculate the parameters (w Q, T Q, sd Q, n Q) (Q = Cz, Pz, C3, and C4) of ERPs obtained by m-DSAM. The averages and the standard deviations (sd) of amplitudes and latencies obtained by AM and by m-DSAM with Mincho letters are shown in Tables 2 and 3, respectively. Table 3 shows the averages and the standard deviations of those values by m-DSAM with Mincho letters. The variable n/m denotes the ratio of numbers, where n is the number of selected data by m-DSAM and m is the number of repetition before (after) Gestaltzerfall.

Table 2. Average and sd of amplitudes and latencies of ERPs in Cz obtained by AM
Full size table
Table 3. Average and sd of parameters in Cz obtained by m-DSAM
Full size table

Since the potentials P1, N1 and P3 appeared clearly in Sect. 3.2, we examine t test for difference of means for paired data, and we obtain Tables 4 and 5. In Table 4, there are significant differences between means of amplitudes of ERPs before Gestaltzerfall and those of ERPs after Gestaltzerfall. In Table 5, there are significant differences regarding to the means of latencies as well as those of amplitudes. By the comparison between Tables 4 and 5, it is suggested that m-DSAM is superior to AM to extract meaningful variables.

Table 4. Results of t test for difference of means of parameters for ERPs by AM (**: p < 0.01)
Full size table
Table 5. Results of t test for difference of means of parameters for ERP by m-DSAM (*: p < 0.05, **: p < 0.01)
Full size table

3.4 Discriminant Analysis Using the Parameters

Using the parameters before and after Gestaltzerfall in Table 4, we execute discriminant analysis to classify the sets of parameters before and after Gestaltzerfall. As our results, these parameters are discriminated with 84.4% accuracy (shown in Table 6). Similarly, we execute the discriminant analysis for Table 5, we obtain 100% accuracy (shown in Table 7). This result shows that m-DSAM is encouraged for the discrimination analysis [7].

Table 6. Classification results using amplitudes and latencies of P1, N1 and P3 in ERPs by AM
Full size table
Table 7. Classification results using parameters of P1, N1 and P3 in ERPs by m-DSAM
Full size table

For each position and each font, we also examine the discriminant analysis. Almost all the discriminant functions discriminate with 100% accuracy.

3.5 ERPs Obtained by Moving Average and the Values of Discriminant Functions

Using the discriminant function for the position Cz and the font “MS-Gothic”, we calculate the values of the discriminant function for the moving averaged ERPs in Figs. 5, 6 and 7, and obtain Fig. 8. The dotted lines are amplitudes and latencies for the moving averaged ERPs, and the upper solid lines is the moving averaged inputs and the lower solid line is the values of discriminant function. The recognition by the subject corresponds to the discrimination by the discriminant function. The result suggests that the process of Gestaltzerfall is expressed well by discriminant function.

Fig. 8.
figure 8

Changes of amplitudes, latencies, input of moving averaged ERPs and values of the discriminant function (position: Cz)

Full size image

4 Discussions

4.1 Validity of Experiments and Analysis

In these experiments, we insert a short interval (10 ms) between each stimulus due to the technical issue. During the short interval, a white screen is displayed. To observe Gestaltzerfall, the subject is required to keep watching an object. Even though a short interval is inserted between each stimulus, Gestaltzerfall is observed. Therefore, ERPs are well recorded in these experiments. We consider that the results of analysis for the ERPs are valid.

4.2 Discussion of the Hypothesis Hypo1

In Sect. 3.3, we examine t Test for the paired parameters, amplitudes, and latencies of Averaged ERPs. And we also examine t Test for the parameters obtained by m-DSAM. There are significant differences between Gestaltzerfall and non-Gestaltzerfall with p < 0.01 level for the amplitudes of N1 and P3 for all positions. Furthermore, there are significant differences between Gestaltzerfall and non-Gestaltzerfall with p < 0.01 or p < 0.05 levels for parameters of latencies and amplitudes obtained by m-DSAM (for all positions). Therefore, it is suggested that our hypothesis “Hypo1: It is possible to measure the occurrence of Gestaltzerfall in the letter recognition by ERPs” is affirmatively confirmed from our experiments and analysis.

4.3 Discussion of the Hypothesis Hypo2

Using the functions for all positions obtained in Sect. 3.3, we calculate the values of discriminant functions. Some of these values are plotted in Fig. 9. The horizontal axis corresponds to the values of discriminant function, and the vertical axis corresponds to moving averaged inputs. The curve of Cz represents the process of Gestaltzerfall precisely. Though other curves include some errors in the indication of Gestaltzerfall, the entire tendency of the curves represents the process of Gestaltzerfall. Similar results are obtained for other stimuli and for all positions. Therefore, our results support affirmatively our hypothesis, “Hypo2: It is possible to grasp the features of Gestaltzerfall and to understand the process getting to Gestaltzerfall in the letter recognition by observing ERPs”.

Fig. 9.
figure 9

The process of Gestaltzerfall using the values of discriminant function and moving averaged input of subjects (positions: Cz, Pz, C3, and C4)

Full size image

5 Summary

Our trials are summarized as follows:

  1. (1)

    Our results support affirmatively the hypotheses Hypo1 and Hypo2.

  2. (2)

    In order to examine these hypotheses, we design experiments. Each time of the experiments, the subject was required to keep watching a Japanese Phonetic letter (“Hiragana”) for one minute. These experiments are executed by “single-case experimental design”.

  3. (3)

    In order to examine Hypo1, we calculate the parameters of ERPs obtained by AM and m-DSAM, and examine t Test. As the result, Hypo1 is affirmatively supported by our experiments and analysis.

  4. (4)

    In order to examine Hypo2, we execute the discriminant analysis for the parameters of ERPs obtained by AM and m-DSAM. As the result, Hype2 is affirmatively supported by our experiments and analysis.

  5. (5)

    In both examinations for the hypotheses, m-DSAM works effectively.

The results discussed in this paper may suggest something important about understanding the characteristics of human brains. We expect that the research direction discussed in this paper would be able to contribute to the design of comfortable, stress-less, and universal BCI systems. These issues are interesting and worthy for the future investigation.