Abstract

1. Introduction

Unraveling causal factors driving various animal behaviors in experimental and in particular in observational studies is challenging, since most behaviors result from an integration of a broad range of social and environmental stimuli, internal states, and past experiences (Chapman et al., 1995; Sapolsky, 2005; Boon et al., 2007; Markham et al., 2015). In laboratory studies, environments and contexts are systematically simplified in order to minimize the number of potential factors influencing behaviors (e.g., Bastian et al., 2001; Pantoni et al., 2020). Such studies are tailored to specific behaviors and well-defined contexts. However, behaviors in such constrained settings often deviate from behaviors in natural environments and thus have to be interpreted with care (Cheney et al., 1995; Rendall et al., 1999; Henninger et al., 2018). To discover behavioral traits of interest in the first place, field studies or laboratory experiments with complex, more naturalistic designs are needed. Recent technological advances in remote recording techniques, tags, and data loggers, as well as advances in data analysis, facilitate the collection and evaluation of comprehensive and viable data in naturalistic settings with freely moving and interacting animals (Dell et al., 2014; Hughey et al., 2018; Mathis et al., 2018; Jolles, 2021). These new big-data approaches open up new opportunities in behavioral research in that they potentially allow to quantitatively study animal behaviors in more complex and naturalistic settings (Gomez-Marin et al., 2014; Egnor and Branson, 2016).

A suitable recording technique can be selected from a large variety of available devices and sensors to match the requirements imposed by the model species, environmental conditions, and the scientific question (Hughey et al., 2018). This allows for studying various aspects of animal behaviors across species (e.g., Nagy et al., 2010; Robinson et al., 2012; Strandburg-Peshkin et al., 2015, 2018). A commonly used technique to study animals in their natural habitats is the utilization of animal mounted bio-loggers, e.g., small devices equipped with different sensors like GPS-trackers or microphones (Nagy et al., 2010; Strandburg-Peshkin et al., 2017; Hughey et al., 2018). However, bio-loggers require frequent animal handling and animals are required to carry devices, both inducing a potential bias (Saraux et al., 2011). Furthermore, bio-loggers might miss relevant information, since not all interacting animals might be equipped with a logger (e.g., Strandburg-Peshkin et al., 2019), signal detection range is limited, or data is recorded discontinuously to extend the overall recording period (Strandburg-Peshkin et al., 2017; Hughey et al., 2018).

Alternatively, behaving animals can be tracked by means of remote sensing devices (Kühl and Burghardt, 2013; Theriault et al., 2014; Henninger et al., 2018, 2020; Hughey et al., 2018; Torney et al., 2018; Raab et al., 2019; Aspillaga et al., 2021). In this approach, recorded signals can originate from small micro-transmitters that get affixed to animals (e.g., acoustic telemetry system for fish: Aspillaga et al., 2021) or from the animals themselves (photography, video recordings: Sherley et al., 2010; Lahiri et al., 2011; Theriault et al., 2014; Nourizonoz et al., 2020; ultrasound vocalizations: Surlykke and Kalko, 2008; Seibert et al., 2013; Hügel et al., 2017; electric signals: Henninger et al., 2018; Raab et al., 2019; Fortune et al., 2020). These methods benefit from minimal interference with the animals themselves. On the other hand, covering large observation areas is costly. Also, tracking animal identities can be quite challenging and requires sophisticated and computationally demanding pre-processing of the data (Lahiri et al., 2011; Kühl and Burghardt, 2013; Hughey et al., 2018; Henninger et al., 2020). Here, specific animal biometrics, certain aspects of an animal's appearance or signaling properties, have been shown to allow for individual identification and tracking (Kühl and Burghardt, 2013). However, in order to enable reliable tracking, selected animal biometrics need to be displayed universally throughout the study population whilst showing sufficient variation between single individuals (i.e., biometric profiles that allow for reliable individual identification). If individuals do not have specific invariant characteristics, like, for example, the stripes of a zebra (Lahiri et al., 2011), then tracking algorithms need to handle temporally changing biometric profiles that often overlap in their characteristics (e.g., spatial position and orientation, Madhav et al., 2018).

Electric fish are particularly well-suited for being tracked in the laboratory and in their natural habitats based on remote sensing (Jun et al., 2013; Henninger et al., 2018; Madhav et al., 2018; Raab et al., 2019; Fortune et al., 2020). These fish are capable of producing an electric field through discharges of an electric organ (EOD, Turner et al., 2007) used for electrolocation (Fotowat et al., 2013) and communication (Albert and Crampton, 2005; Smith, 2013; Benda, 2020). The EODs of many electric fish can be recorded by means of an array of submerged electrodes without the need to catch and tag the fish (Henninger et al., 2018). From these recordings, electric signals of individual fish have to be identified and tracked over time. Dependent on electric fish species, EODs are either emitted in short and discrete pulses (pulse-type electric fish; Hagedorn, 1988; Albert and Crampton, 2005; Smith, 2013) or in a sinosoidal fashion (wave-type electric fish; Moortgat et al., 1998). For pulse-type electric fish, tracking individual EODs is rather challenging, since signal features largely overlap between individual fish, i.e., EOD frequencies are highly variable and context dependent (Hagedorn, 1988). In order to, nevertheless, track electric behaviors of pulse-type fish, additional spatio-tempoal tracking using video recordings and elaborate machine-learning approaches are usually required (Jun et al., 2013; Pedraja et al., 2021). In wave-type electric fish, however, the frequency of EODs is individual specific and remarkably stable over minutes to hours (Moortgat et al., 1998), providing a characteristic biometric cue which facilitates individual signal tracking. Previous tracking approaches were either based on EOD frequency (Henninger et al., 2020) or on spatial electric field properties that can be reconstructed from signal powers across recording electrodes (Madhav et al., 2018). However, both signal features are sensitive to temporal changes. The latter, spatial electric field properties, depends on the fish's spatial position and orientation. The former, EOD frequency, is sensitive to temperature changes (Dunlap et al., 2000) and is actively modulated for electrocommunication (Smith, 2013). Accordingly, both tracking features might fail when fish are close by, either in their EOD frequency or spatially, especially in recordings of electric fish in high densities.

In the following, we describe and evaluate an improved tracking algorithm for wave-type electric fish recorded with electrode arrays. By combining, refining, and extending previous approaches, our algorithm is capable of tracking EODs of individual fish with unprecedented accuracy, i.e., tracking errors occur less often in complex tracking scenarios (e.g., when EOD frequency traces cross each other, Figure 5) which tremendously reduces required post-processing time to manually correct flawed connections. Since both movement behaviors (Madhav et al., 2018; Henninger et al., 2020) and communication (Smith, 2013; Henninger et al., 2018; Fortune et al., 2020) can be analyzed based on EOD recordings, our algorithm is a fundamental advancement for a wide range of behavioral studies on freely moving and interacting electric fish (Raab et al., 2019, 2021). Finally, we demonstrate the performance of our tracking algorithm on recordings of Apteronotus leptorhynchus taken with an array of 64 electrodes in a stream in the Llanos in Colombia.

2. Materials and equipment

2.1. Data acquisition

EODs of freely swimming fish were recorded with arrays of monopolar electrodes at low-noise buffer headstages (1×gain, 10 × 5 × 5 mm³, Figure 1B) arranged in grid-like structures (Figures 1A,​,CC,​,E).E). Electric signals are amplified (100×gain, 100 Hz high-pass filter, 10 kHz low-pass), digitized at 20 kHz with 16 bit resolution, and stored on external data storage devices for later offline analysis. The custom-built recording systems (npi-electronics GmbH, Tamm, Germany) were powered by car batteries (12 V, 80 Ah). Various configurations of the electrode arrays have been successfully used to record populations of electric fish in the wild (Henninger et al., 2018, 2020, unpublished field-trips: Colombia 2016, 2019, Figures 1A,​,CC,​,D),D), as well as in the laboratory (Raab et al., 2019, 2021, Figures 1E,​,F).F). The first 64-channel amplifier system required an external computer with two data acquisition boards (PCI-6259, National Instruments, Austin, Texas, USA) for digitizing and storing the data (Henninger et al., 2018, 2020, Colombia 2016). For this first setup, data acquisition was controlled by a C++ software (https://github.com/bendalab/fishgrid). For the 2019 recordings in Colombia we used a modular 16-channel system based on a Raspberry Pi 3B (Raspberry Pi Foundation, UK) that stores the data digitized by an USB data acquisition board (USB-1608GX, Measurement Computing, Norton, MA, USA) on an 256 GB USB stick controlled by python software (https://whale.am28.uni-tuebingen.de/git/raab/Rasp_grid.git) (Figure 1A).

Open in a separate window

Figure 1

Recording systems, electrode arrangements, and corresponding signals of recorded electric fish. (A) Two of the Raspbery Pi-based 16-channel amplifiers and recorders used for an array with 32 electrodes. (B) Monopolar stainless-steel electrode on headstage used for recordings in the field and laboratory experiments (after Henninger, 2015). (C) Recording setup used to record a population of A. leptorhynchus in the Rio Rubiano, Colombia, in 2016. Sixty-four electrodes were mounted on PVC-tubes and arranged in an 8 × 8 grid covering an area of 3.5 × 3.5 m². (D) Snapshot of the electric signals recorded with the setup shown in (C). The top left panel corresponds to the most upstream electrode mounted on the tube closest to the river bank. (E) Recording setup used to record electric signals of pairs of A. leptorhynchus during competitions in a laboratory experiment (Raab et al., 2021). Fifteen electrodes were uniformly distributed at the bottom of the aquarium and one electrode was placed in the central tube the fish compete for. (F) Snapshot of electric signals recorded during the competition experiment shown in (E). The signal framed in gray is from the central electrode located in the optimal tube. The EOD waveform shows the characteristic shoulder that is generic for EODs of A. leptorhynchus.

2.2. Spectrograms

EODs of individual fish are identified and extracted from the electric recordings based on their EOD frequency and respective harmonic structure (Figure 2C). For each electrode we compute power spectral densities (PSDs) of overlapping data snippets shifted by Δt ≈ 300 ms (Figure 2A). The size of fast Fourier transform (FFT) windows was set to $n_{\mathrm{\text{fft}}}=2^{15}\approx 1.64$ s (e.g., Raab et al., 2021) or $n_{\mathrm{\text{fft}}}=2^{16}\approx 3.28$ s (e.g., Raab et al., 2019; field recordings displayed in Figure 9) to result in frequency resolutions of 0.6 and 0.3 Hz, respectively, needed to resolve EOD frequencies in high fish densities.

Open in a separate window

Figure 2

EOD frequency extraction from recordings with an electrode array. As an example, a 3 min snippet of a recording with the 8 × 8 array from Rio Rubiano, Colombia, taken during the day of April 10^th, 2016 is shown. (A) Spectrograms from three different electrodes. Warmer colors represent increased power in respective frequencies. EOD frequencies of individual A. leptorhynchus remain rather stable, except during electrocommunication (e.g., EOD frequency trace starting at ~ 917 Hz). A non-logarithmic PSD extracted at time 50 s indicated by the dotted line is shown at the side of each panel. (B) The summed up spectrogram over all electrodes contains distinct traces from many different fish. (C) Peaks are detected in the summed up power spectra that are then clustered into frequency groups of a fundamental frequency and at least two of its harmonics, corresponding to a specific fish (Henninger et al., 2020). Fundamental EOD frequencies, their corresponding powers in each electrode and their detection times are stored for subsequent tracking.

3. Methods

In the following we present an algorithm for tracking wave-type electric fish in electrode-array recordings. The algorithm merges and extends two complementary approaches that are based on EOD frequency (Henninger et al., 2018, 2020) or on primarily the spatial distribution of signal powers (Madhav et al., 2018). We then test the performance of the tracking algorithm against manually tracked data. Open-source Python scripts for tracking and post-processing of analyzed data can be obtained from https://github.com/bendalab/wavetracker.

3.1. Algorithm for tracking wave-type electric fish

Both EOD frequency and the spatial distribution of EOD power across electrodes change with time and potentially overlap between fish. EOD frequencies can be actively altered in the context of communication (Smith, 2013; Benda, 2020) and the signal powers across electrodes change with the fish's motion (Madhav et al., 2018). This variability and potential overlap in signal features challenges reliable tracking, especially in recordings with many fish.

Furthermore, the existing algorithms track signals in the order of their temporal detection, i.e., signals detected in consecutive time steps are directly assigned to already tracked EOD frequency traces (Madhav et al., 2018; Henninger et al., 2020). Potentially this leads to tracking errors, because even with the utilization of an electrode array, EODs of freely moving and interacting electric fish are rarely detected continuously, i.e., consecutively in subsequent time steps. Low signal-to-noise ratios, resulting from large distances between fish and recording electrodes or objects like rocks or logs distorting or even blocking electric fields, frequently lead to detection losses. When multiple fish with similar EOD frequencies are recorded simultaneously, EOD frequency traces can potentially cross each other (e.g., in the context of emitted communication signals, Benda, 2020). It is in these occasions in particular, that detection losses frequently result in tracking errors.

In order to improve on these issues, we developed a tracking algorithm which, first, is based on feature vectors that include both EOD frequency and signal power across electrodes (Figure 3) and, second, is less constrained by the temporal sequence of detected signals (Figure 5).

Open in a separate window

Figure 3

Frequency and field errors. (A) Summed spectrogram of a 30 s long part of the recording shown in Figure 2B. For each electric fish signal, potential connection partners are limited by a time difference threshold, Δt_thresh = 10 s, and a frequency difference threshold, Δf_thresh = 2.5 Hz. For a given signal α with EOD frequency f_αi at time step i (dark blue dot), potential connection candidates β at different times j (light blue dots) need to be within these thresholds (box), whereas signals beyond these thresholds (black dots) are not considered. (B) Absolute frequency differences, Δf Equation (2), are mapped (red lines) to frequency errors, ε_f, using a logistic function, Equation (4) (line), favoring small frequency differences. (C) The field error as the second tracking parameter is based on spatial profiles, Equation (5), of signal powers over all electrodes (black dots). The field difference, ΔS, is computed as the Euclidean distance, Equation (6), between the spatial profiles, Equation (5), of potential signal pairs (columns). With decreasing similarity (columns left to right) the field difference increases. Displayed signal pairs (columns) were selected to illustrate the full range of possible field differences and are unrelated to (A). Spatial profiles were interpolated with a gaussian-kernel for illustrative purposes. (D) To obtain normalized field errors, ε_S, in a range similar to the one of the frequency errors, ε_f, each field difference is set into perspective to a representative cumulative distribution [Equation (7), black line] of field differences obtained by collecting all potential field differences of a manually selected 30 s window in the recording. The cumulative distribution of potential field differences is computed only once per recording for a 30 s window where fish are active (night time). This way we incorporate a broad distribution of possible field differences when determining field errors. The examples from (C) are marked by respectively colored dots.

3.1.1. Distance measure

We start out with extracting feature vectors ${\overrightarrow{X}}_{k_i}$, Equation (1), containing an EOD frequency, f_ki, and its powers, L_ki(x), on all electrodes x, for all signals k and each time step i. In a first step the distance between all pairs of feature vectors, ${\overrightarrow{X}}_{{\alpha }_i}$ and ${\overrightarrow{X}}_{{\beta }_j}$, of signals α and β at times i ≠ j are quantified. Only pairs within a time difference of |t_j − t_i| ≤ Δt_thresh = 10 s and a maximum difference

$$\begin{array}{l}\Delta f_{{\alpha }_i,{\beta }_j}=|f_{{\alpha }_i}-f_{{\beta }_j}|\end{array}$$

(2)

≤ Δf_thresh = 2.5 Hz between the two EOD frequencies of the feature vectors are considered (Figure 3A).

The distance between the two signals α_i and β_j

$$\begin{array}{l}{\epsilon }_{{\alpha }_i,{\beta }_j}=\frac{1}{3}{\epsilon }_f+\frac{2}{3}{\epsilon }_S\end{array}$$

(3)

is computed as a weighted sum of the frequency error, ε_f, and the field error, ε_S. Both errors range from 0 to 1 and are explained in the following sections. The field error gets twice the weight of the frequency error, because tracking issues usually arise in spite of low frequency errors. Nevertheless, the frequency error remains a relevant tracking feature, especially when fish are in close proximity resulting in low field errors.

3.1.2. Frequency error

The frequency error is based on the difference in EOD frequencies, Equation (2) and has been used previously to track signals of electric fish (Henninger et al., 2018, 2020). We transform the EOD frequency difference, Equation (2), into the frequency error

$$\begin{array}{l}{\epsilon }_f(\Delta f)=\frac{1}{1+{\mathrm{\text{e}}}^{-\frac{\Delta f-f_0}{df}}}\end{array}$$

(4)

via a logistic function, that maps the EOD frequency difference, Δf, onto the interval from zero to one. The turning point of the logistic function at f₀ = 0.35 Hz and the corresponding inverse slope, df = 0.08 Hz ensure a maximum frequency error already at small EOD frequency differences of about 0.8 Hz (Figure 3B). This transformation mitigates very small frequency differences and equalizes larger frequency differences in the assessment of whether two signals α and β originate from the same or different fish.

3.1.3. Field error

EOD frequency traces of electric fish occasionally cross each other, e.g., when individuals actively alter their EOD frequency in the context of communication (e.g., Zupanc, 2002; Triefenbach and Zakon, 2008; Raab et al., 2021, Figure 3A). In these situations, frequency as a tracking feature fails. This is where the spatial properties of a signal, i.e., signal powers across recording electrodes that reflect the position and orientation of a fish, come into play (Madhav et al., 2018, Figure 3C). The signal powers, L_ki(x), are rescaled to the spatial profile

$$\begin{array}{l}{S}_{k_i}(x)=\frac{L_{k_i}(x)-{\displaystyle \underset{x}{min}}{L}_{k_i}(x)}{{\displaystyle \underset{x}{max}}{L}_{k_i}(x)-{\displaystyle \underset{x}{min}}{L}_{k_i}(x)},\end{array}$$

(5)

ranging between 0 and 1, for the smallest and largest power of that signal, respectively.

The field difference ΔS, i.e., the difference between the spatial profiles of two signals α and β at times i and j, is computed as their Euclidean distance according to

$$\begin{array}{l}\Delta S_{{\alpha }_i,{\beta }_j}=\sqrt{{\displaystyle \sum _{x=1}^n}{(S_{{\alpha }_i}(x)-S_{{\beta }_j}(x))}^2}\end{array}$$

(6)

However, the magnitude of this difference depends on the configuration of the electrode array, especially on the number of recording electrodes. To obtain field errors, ε_S, that are independent of electrode configuration, we map the field differences through a cumulative distribution of field differences extracted from a manually selected and representative 30 s window:

$$\begin{array}{l}{\epsilon }_S(\Delta S_{{\alpha }_i,{\beta }_j})={\displaystyle {\int }_0^{\Delta S_{{\alpha }_i,{\beta }_j}}}p(\Delta S)d\Delta S\end{array}$$

(7)

The distribution of field differences, p(ΔS), is estimated from the field differences between potential signal pair (Δt_i,j ≤ ± 10 s, any frequency difference) within a 30 s data snippet where fish can be assumed to be active, i.e., during night time (Figure 3D). This way we incorporate a broad distribution of possible field differences when determining field errors.

3.1.4. Tracking within a data window

Now that we have a quantification for the distance ε, Equation (3), between to signals we can proceed with the actual tracking algorithm. Based on the distances, the algorithm decides which signal pairs belong together in order to track individual fish throughout a recording. Computing the distances between all pairs of signals of a recording at once, however, is not feasible. Instead we break down the tracking into tracking windows of 30 s at a time (Figure 5). Within these tracking windows, we first compute the distances, ε, between each potential signal pair α_i and β_j and store them in a three-dimensional distance cube, where the first two dimensions refer to signals α_i and β_j and the third dimension to the time steps i where signals α_i have been detected (Figure 4). Accordingly, we have different numbers of signals α_i for each time step i and, consequently, the number of elements in the second dimension, referring to signals β_j from all time steps j > i is also variable. Note that, each signal considered in the distance cube is only referred to as α once, but potentially multiple times as β. For example, a signal that is referred to as β_j = β_i+1 in the first layer of the distance cube (see Figure 4) is referred to as α_i+1 in the next layer of the distance cube.

Open in a separate window

Figure 4

Distance cube containing all distances, ε_{α, β} Equation (3), for possible signal pairs α and β within the current tracking window. Each layer, referring to a time step i, contains the distances between all signals α_i detected at this time and their potential signal partners β_j detected maximally 10 s after signal α_i (ΔI time-steps after i). Distances in gray layers correspond to signal pairs where one signal partner could potentially have a smaller distance to a signal outside the error cube. Only connections based on the distances in the central black layers can be assumed to be valid, since all potential connections of both signal partners are within the error cube. Connections established for the black layers are assigned to signal traces obtained in previous tracking steps in a second step.

For the actual tracking step, signal pairs are connected and assigned to potential fish identities based on the values in the distance cube. The algorithm described in the following (Figure 5) is a kind of clustering algorithm that has a notion of temporal sequence. The resulting clusters are traces of different fish identities (“labels”) tracked over time.

Open in a separate window

Figure 5

Tracking within a data window. Signals detected in a 30 s data window are connected to each other and assigned to fish identities according to their distance ε, Equation (3). Signal pairs with smaller distances are connected first. With increasing distance values, more connections and identities are formed, complemented, or merged, ensuring no temporal overlap. Different stages of this tracking step are displayed in (A–C). (A) Twenty percent of all possible connections of the displayed tracking window are formed. At this tracking stage a multitude of separate signal traces (different colors) are still present. (B) Forty percent of all possible connections of the displayed tracking window are formed. (C) Final output of the tracking step. All possible connections of the displayed tacking window are formed. The remaining three EOD frequency traces (in the displayed time and frequency segment) correspond to three different fish identities. Only signal pairs within the central 10 s of an 30 s tracking window (vertical lines) are assigned to already established fish identities from previous tracking windows. The summed spectrogram of a 30 s long part of the recording shown in Figure 2B is shown in the background.

The signal pairs are traversed in order of ascending distances. If one of α_i or β_j have already been assigned to a fish identity, then this pair is added to this fish identity. If α_i coincides with one fish identity and β_j with another one, then the two fish identities are merged. If neither α_i nor β_j match an existing fish identity, the pair is assigned to a new fish identity. Assignment to or merging of fish identities are only possible in the absence of temporal conflicts, i.e., a fish identity cannot have more than one signal at the same time. In case of temporal conflicts, the signal pair is ignored and the algorithm proceeds with the next one. As a result, we obtain signal traces built upon minimal signal errors within a 30 s tracking window (Figure 5).

Since signals within the first and last 10 s of a tracking window could have lower distances to signals outside the current tracking window, these connections are potentially flawed (gray layers in Figure 4; gray bars in Figure 5). Only connections established within the central 10 s take all other potential signal partners into account. Accordingly, only the section of assembled signal traces corresponding to these central 10 s of the current tracking window is considered for further processing, where the signal traces are appended to already validated, previously detected ones (Figure 6).

Open in a separate window

Figure 6

Assembly of tracking results over data windows. (A) New fish identities established within the current tracking window (gray and black bar on top). Only the central 10 s of these EOD frequency traces (solid traces; black bar) can be assumed to be valid since signals before and after (transparent traces; gray bars) have potential signal partners outside the tracking window. (B) Additional display of EOD frequency traces established in previous iterations of the tracking algorithm. (C) Signal traces are connected according to the smallest possible distance measure between any signal between the last 10 s of the established fish identities (10 s < t < 10 s) and the central 10 s of the new fish identities (10 s < t < 20 s). In the example shown, the distance between the origin signal (black dot) and the target signal (green dot) is the smallest between these two signal traces, accordingly the two signal traces are merged (green and orange lines). An alternative signal (red dot) has a larger distance to the origin signal. (D) Final result of the tracking algorithm that will be used for the next iteration.

3.1.5. Assembly of tracking results over data windows

The assignment of the 10 s long signal traces obtained by the tracking algorithm from 30 s long data windows (Figure 6A) to preceding tracking steps (Figure 6B) proceeds, similar to the algorithm described above, based on the smallest distances between them.

First, the distance between those signals α within the first 10 s of the current tracking window of already established fish identities and new signals β from the central 10 s of the current tracking window are computed. Then, starting with the pair with the smallest distance, the new signal trace containing signal β (for example the green dot in Figure 6C), is connected to the established signal trace (from previous tracking steps) containing signal α (for example, the black dot in Figure 6C). This step is repeated with signal pairs of increasing distance until all possible connections are established (Figure 6D).

The described tracking within a data window and the subsequent assignment to previously established fish identities is repeated with data windows shifted by 10 s until the end of the recording is reached. In each iteration, the distance cube is updated. The first layers corresponding to the first 10 s of the previous tracking window are removed (frontal gray layers in Figure 4) and new layers for the next 10 s beyond the last tracking window are extended to the error cube in preparation for the next iteration of tracking.

3.2. GUI for checking and correcting tracking results

Even though the introduced algorithm is capable of resolving most tracking conflicts correctly when tracking EODs of wave-type electric fish, occasional tracking errors still remain. We developed a GUI that allows to visually inspect and validate tracked EOD frequency traces and to fix flawed connections (Figure 7). Flawed connections can easily be identified by their clear deviation from the spectrogram displayed in the background. Furthermore, signal traces with a detection gap beyond the temporal threshold of Δt_thresh = 10 s of the tracking algorithm can be manually connected based on visual cues from the spectrogram. The resulting validated signal traces are then stored and further analyzed (e.g., Raab et al., 2019, 2021).

Open in a separate window

Figure 7

Graphical user interface for validating and fixing tracking results. The user is presented with the tracked signal traces (EOD frequency traces) displayed on top of a spectrogram summed up across recording electrodes. The user can delete, cut, and connect signal traces or delete signals not originating from electric fish.

4. Results

The complexity of the data set we recorded in Colombia in 2016 led us to the development of the presented tracking algorithm. The high density of fish in this data set (about 25 fish within 3.5 × 3.5 m²) results in many individual EOD frequency traces, where EOD frequencies were often very similar and frequently cross each other, in particular in the context of communication (Figure 9). This severely challenged previous tracking approaches (Madhav et al., 2018; Henninger et al., 2020), thus a better tracking algorithm was required. The improved algorithm resolves many tracking issues resulting from crossing EOD frequency traces and facilitates the evaluation of wave-type electric fish recordings even in abundant populations. In the following we evaluate the performance of the developed algorithm and highlight how it can be used to advance our knowledge about the behavior of freely moving and interacting electric fish by facilitating laboratory studies as well as natural field observations.

4.1. Performance of the tracking algorithm

In order to quantify the performance of the presented tracking algorithm, we evaluate potential tracking conflicts that occur during the analysis of a datasets we recorded with an 8 × 8 electrode array in Colombia during the day of April 10^th, 2016 for 10 h:50 m. First, we tracked the fish with the presented algorithm and then visually inspected, corrected, and validated the tracking results using the GUI (Figure 7). Second, we run the tracking algorithm again and compared the connections made by the algorithm with the manually improved ones. That is, for each signal α_i we inspected all possible connections with a signal β_j (one row in the distance cube) within the central 10 s of the current tracking window. If all the β_j for a given α_i were assigned to the same fish identity in the visually corrected tracking results, we have no potential conflict and these connections were not further considered for quantifying the performance of the algorithm, because these are the simple cases with a single fish within the maximum EOD frequency difference, Δf_thresh, of 2.5 Hz. If, however, the possible connections involved two or more fish identities, a tracking conflict was possible. For each such potential tracking conflict, we extracted the EOD frequency difference Δf, Equation (2), field difference ΔS, Equation (6), frequency error ε_f, Equation (4), field error ε_S, Equation (7), and resulting distance measure ε, Equation (3), between the signal α_i and the best signal partner β_j, the one with the smallest distance ε, associated with the same fish identity as in the visually corrected signal traces (true connection), as well as between the signal α_i and the best signal partner β_j belonging to a different fish identity (false connection). Further fish identities of the β_j with larger distances were ignored.

In order to assess the performance of each signal feature difference (Δf & ΔS) and distance measure (ε_f, ε_S, ε) in separating true from false connections, we computed the fraction of signal differences or errors of true connections being smaller than those of the corresponding false connections. If this fraction would be 100% then the tracking algorithm would always have connected the right signals. In addition we quantified the overlap of the two distributions by the area under the curve (AUC) of a receiver-operating characteristic (ROC). Despite an overlap (low AUC values) in principle 100% correct connections would be possible, but an overlap demonstrates that fixed decision thresholds are not feasible.

We start with evaluating the 464 tracking conflicts from a 5 min snippet being especially challenging to track, because of several crossings of EOD frequency traces (Figure 8). A small frequency range of this 5 min data snippet is displayed in Figure 7. The least reliable tracking feature appears to be the difference in EOD frequency (Δf and ε_f). Frequency differences of true connections were smaller than the ones of false connections in only 94.83% (440/464) of the cases (Figures 8A,​,C).C). Better results can be achieved based on the field error (ΔS and ε_S) as a tracking feature. The field differences of true connections were smaller in 99.57% (462/464) of the cases (Figures 8B,​,D).D). However, this performance can even be improved when using the distance measure, ε, that combines both the frequency error, ε_f, and field error, ε_S. In 99.87% (462/464) of the tracking conflicts, true connections had smaller distances than false connections (Figure 8E). The AUC values for all measures were similar to the fractions of correct connections (Δf and ε_f: AUC = 95.16%, ΔS and ε_S: AUC = 99.77%, ε: AUC = 99.86%), indicating a small but existing overlap between the two distributions.

Open in a separate window

Figure 8

Performance of the tracking algorithm. Conflicts appear if signals could be connected to multiple different fish identities, that have been manually corrected and checked post-hoc (Figure 7). In most but not all cases, correct connections have smaller signal differences or errors (blue) than wrong connections (red). Shown are kernel density estimates (KDE) for the various signal differences, errors, and distances. The overlap of the distributions was quantified by the AUC of an ROC-analysis as indicated in the right column. (A) EOD frequency differences, Δf Equation (2). A logistic function, Equation (4) (black line), translates EOD frequency differences to frequency errors, ε_f. (B) Field differences, ΔS, Equation (6). The cumulative distribution (black line) of field differences of all pairings, not only from conflicts, translates field differences to field errors, ε_S, Equation (7). (C) Frequency error, ε_f, Equation (4). (D) field error, ε_S, Equation (7). (E) Combined distance measure, ε, Equation (3). Note, that frequency and field errors (C,D) are mapped via monotonically increasing functions from signal differences (A,B) and thus result in the same fraction of correct connections and AUC values. However, the distance measure combining both field and frequency error performs best.

The 261344 tracking conflicts of the whole recording yield similar results. However, the higher proportion of “easy” tracking conflicts increased the performance of the various features in general and differences between them were less pronounced. Nevertheless, EOD frequency still performed worse (99.73% correct connections) than field difference (99.81% correct connections). Again, combining both into the distance measure, Equation (3), resulted in the best performance (99.95% correct connections). Correspondingly, the overlap between the two distributions was reduced (Δf and ε_f: AUC = 99.79%, ΔS and ε_S: AUC = 99.85%, ε: AUC = 99.98%).

In order to put these high numbers in perspective, we estimate the time required to post-process signal traces obtained for the whole dataset recorded during the day of April 10^th, 2016, in Colombia (including 261, 344 potential tracking conflicts) when using (i) only frequency difference, (ii) only electric field difference, or (iii) the combined signal error ε_S as tracking parameter. Finding and correcting single tracking errors using our GUI (Figure 7) requires about 15 s each (personal experience). Accordingly, post-processing signal traces of the whole recording would require about 3 h when solely frequency is used as tracking feature, 2 h when the field difference alone is used for tracking, and only about 30 min when our combined signal error ε_S is used. However, note that the dataset used here to illustrate the performance of the algorithm is the most complex ever recorded to our knowledge. With decreasing complexity, i.e., less fish in a recording, the amount of potential tracking conflicts, and thereby the required post-processing time, rapidly decreases.

Furthermore, a major advancement of the presented algorithm is represented by the tracking process itself, i.e., tracking signals in discrete tracking windows according to the similarity of signal pairs (Figure 5). However, this advancement is only validated by human observers, since the recreation of previous tracking approaches is too demanding for the sole purpose of accuracy comparison.

4.2. Applications of the developed algorithm

By means of the developed algorithm we were able, for the first time, to track electric signals of individual fish for multiple consecutive days in a natural, high density population of A. leptorhynchus recorded in a stream in Colombia (Figure 9). This allowed for novel insights into the natural behavior of these fish in the wild, including their communication and movement behaviors. A preliminary analysis of the tracked fish indicates that many fish stay pretty stationary within distinct areas for multiple days (Figure 10). Other fish, especially during the night, can only be tracked for short time periods, suggesting these fish only transit through the area covered by the electrode array (Figure 9A). Furthermore, fish seem to interact with each other by modulating their EOD frequency in various ways and on many different time scales ranging from seconds to many minutes, if not even hours. This includes not only distinct communication signals like rises (Raab et al., 2021, Figure 9B), but also other not yet classified EOD frequency modulations, for example multiple EOD frequency traces entwining each other (Figure 9C).

Open in a separate window

Figure 9

Long-term field recording of A. macrostomus, a member of the A. leptorhynchus species group, in Colombia, 2016. EODs were recorded with a 64 channel electrode array covering 3.5 × 3.5 m³. (A) Eight days of detected and tracked EOD frequencies. Successfully tracked and validated signal traces of different fish are indicated in different colors. Signal traces that could not be clearly validated are indicated in white. Dark gray areas indicate night time, light gray areas day time. (B) Signal traces of three fish where the crossing EOD frequency traces of the upper two fish could reliably be resolved by the tracking algorithm. (C) Too many signal traces with similar frequencies compromise the tracking algorithm (670 − 672 Hz). Frequency peaks in PSDs belonging to multiple fish temporally overlay and prevent successful tracking.

Open in a separate window

Figure 10

Spatial behavior of a single A. macrostomus detected and tracked consecutively for 4 days. Heat-maps and contour lines show the fish's probability of presence across the monitored 3.5×3.5 m² area of the river during the night (top) and day (bottom). The observation area ranged from the river bank (x = 0) to the center of the river (x = 3.5) with similar extend in the flow direction of the river (see Figure 1C). Heat-maps of signal powers over electrodes are interpolated using a gaussian-kernel for illustrative purposes. Orange contour lines include the area in which the fish spends more than 50% of the time, the red lines more than 75% of the time respectively. Even though the fish certainly shows movement behaviors, especially during the night, it remains remarkably stationary in a specific location of the obervation area for four consecutive days.

Such natural observations are invaluable since only in the wild, the whole scope of an animal's behavior can be observed in the context of all relevant stimuli and conditions that shaped these behaviors through evolutionary adaptations. Accordingly, such natural observations yield the unique opportunity to discover novel and unexpected behavioral traits and associated causalities. For example, Fortune et al. (2020) described behavioral and physiological adaptations of Eigenmannia vicentespelea, another gymnotiform wave-type electric fish, in response to living in a constantly dark cave. E. vicentespelea developed increased territoriality and enhanced EOD amplitudes in comparison to Eigenmannia trilineata, not living in caves, to face the challenges of their specific habitat. If not for the corresponding field study, these behavioral and physiological adaptations probably never would have been discovered.

Furthermore, field studies are also essential to validate conclusions drawn form laboratory experiments, which is especially important since behaviors observed in the laboratory often deviate from those observed in the wild (Cheney et al., 1995; Rendall et al., 1999; Henninger et al., 2018). In our case, the preliminary behavioral observations we made in Colombia and described above fit to and support the conclusions of our recent laboratory experiments (Raab et al., 2019, 2021). In these experiments we used the algorithm presented here to track individual electric signals of A. leptorhynchus in different behavioral contexts. This includes the evaluation of individual spatio-temporal movement behaviors in a freely moving and interacting group of 14 A. leptorhynchus (Raab et al., 2019) as well as the communication behavior of pairs of A. leptorhynchus competing over a shelter during staged competitions (Raab et al., 2021). In both laboratory and field observations, fish produce the majority of rises as electrocommunication signals during the night (Raab et al., 2021, Figure 9A), are more stationary during the day compared to the night (Raab et al., 2019, Figure 10), and seem to not remain completely stationary for the whole inactive day-phase but rather show short periods of activity (Raab et al., 2019, Figure 10). The observed stationarity of fish observed in our field recordings also fit to our suggestion of A. leptorhynchus establishing a dominance hierarchy to regulate an individual's access to resources (Raab et al., 2021). Due to the fish's stationarity, repetitive conflicts with the same individuals are presumably inevitable and the establishment of a dominance hierarchy can be assumed to be the most economic way to resolve these conflicts (Sapolsky, 2005).

Finally, the evaluation of natural recordings very accurately illustrate the advantages and limitations of the presented algorithm. While crossing EOD frequency traces can usually be resolved accurately (Figure 9B), reliable tracking usually fails when too many signals traces are of similar EOD frequency and entwine in diffuse EOD frequency alterations (Figure 9C). In these occasions the signals of multiple fish superimpose in the spectrogram analysis (Figure 9C) for longer time periods. As a consequence, the corresponding detected signals comprise signal powers of multiple fish. Accordingly, their clear assignment to one of the involved identities is usually impossible after the EOD frequency traces disentangle.

5. Discussion

Previous approaches on tracking wave-type EODs of individual wave-type electric fish either utilized their EOD frequency (Henninger et al., 2020) or the spatial profile of their electric fields (Madhav et al., 2018) as tracking features. We assessed the performance of both signal features alone as well as a combination of both, based on tracking conflicts occurring while processing a recording of a natural, high density population of A. leptorhynchus in a stream in Colombia. The comparison of spatial field properties clearly performs better than a comparison of EOD frequencies. Certainly, the EOD frequency of A. leptorhynchus can be remarkably stable over minutes to hours (Moortgat et al., 1998). However, EOD frequency changes with various magnitudes on various time scales can regularly be observed, because of its strong temperature dependence (Dunlap et al., 2000), actively produced electrocommunication signals (e.g., Zupanc, 2002; Triefenbach and Zakon, 2008; Smith, 2013; Benda, 2020; Raab et al., 2021), and also as an artifact of the EOD frequency extraction from the PSDs (Figure 2). Accordingly, the suitability of EOD frequency as tracking feature decreases the more fish are recorded and analyzed simultaneously, since EOD frequency differences between fish are potentially smaller and interactions between fish involving active EOD frequency modulations can be assumed to be more frequent. Therefore, spatial field properties reflecting a fish's spatial position and orientation represent a more robust tracking feature, especially when only those signal pairs with small EOD frequency differences are considered for comparison and tracking.

The best tracking performance is achieved by using both EOD frequency differences and field differences. This combined signal distance implements a tracking bias that helps to resolve tracking conflicts in at least two scenarios, where tracking solely based on field differences fail. First, if two fish swim close to each other with similar orientations, then their spatial profiles are similar but they can be still differentiated based on their EOD frequencies. Second, in the event of crossing EOD frequency traces, temporarily only one signal can be extracted by detecting peaks in the PSD (Figure 3A). So neither the EOD frequency difference nor the difference in spatial profiles provide a meaningful hint for tracking in the moment of the intersection. Adding EOD frequency difference to the distance measure then slightly favors connections of signal pairs with more similar EOD frequencies, resulting in a bias for superimposed signals detected at the intersection to be connected to the EOD frequency trace of the fish with a more constant EOD frequency (Figure 5, grey trace in bottom panel). The other signal traces, accordingly, remain to be connected across the intersection afterwards.

More important for the improved performance of the presented tracking algorithm is the algorithm itself, in addition to the combined distance measure. The tracking algorithm establishes connections within an extended tracking window based on smallest distances (Figure 5). This is in contrast to existing tracking algorithms (Henninger et al., 2018, 2020; Madhav et al., 2018), that immediately connect the signals detected in a given time step to known fish identities.

When studying animals and their behaviors by means of evaluating external recordings, we rely on the detection of sensory cues emitted actively or passively by the animals themselves (Dell et al., 2014; Hughey et al., 2018). In cluttered environments or when signals are weak (low signal-to-noise-ratio), reliable signal detection is often impaired and detection losses frequently occur. These detection gaps complicate reliable tracking, especially when signals are tracked according to their temporal occurrence. In recordings of electric fish, detection losses frequently result from fish being too far away from recording electrodes, from the electric fields being blocked by any objects between a fish and recording electrodes, or by intersections of EOD frequencies. The resulting tracking failures can be avoided with the presented algorithm, since it relies less on the temporal sequence of detected signals. Rather, connections are established according to the smallest distances within extended tracking windows.

Despite the high fractions of correct connections (Figure 8), the resulting EOD frequency traces need to be corrected manually. This is in particular necessary in sections with close by EOD frequencies or when EOD frequency traces cross each other due to active modulations. Recordings of only a few fish with well-separated EOD frequencies require much less or even no manual interventions. With the current state of the presented algorithm we push the limits to more complicated signal interactions, but the performance still is not perfect for interesting scenes with a lot of interactions (Figure 9). Deep neural networks that are successfully used to track animal pose (e.g., Mathis et al., 2018), or to annotate acoustic signals from various animals (e.g., Steinfath et al., 2021), might be an interesting option to further improve tracking performance. Such approaches, however, require extensive training data sets. Our tracking algorithm and evaluated data sets might set the basis for developing and training of deep neural networks in the future.

6. Conclusion

The self-generated electric fields of electric fish offer an unique opportunity for studying natural movement and communication behaviors of nocturnal fish in freely interacting populations. The EODs of whole groups can be recorded simultaneously by means of electrode arrays submerged in the water—without the need to catch and tag the fish. The presented algorithm for tracking wave-type electric fish combines previous approaches based on either their individual-specific EOD frequencies (Henninger et al., 2020) or the spatial profiles of electric fields resulting from a fish's location and orientation (Madhav et al., 2018). The algorithm uses a compound signal distance, that incorporates both EOD frequencies and spatial profiles. We developed a new temporal clustering method that assembles fish identities from all signals within a large tracking window according to ascending signal distances. With this approach, our algorithm improves in resolving tracking issues mainly resulting from crossing EOD frequency traces or detection losses, which tremendously reduces required post-processing time and makes this technique more feasible even in long-term observational studies on freely moving electric fish. Since tracked EOD traces allow insights into both movement and communication behaviors, a reliable tracking algorithm is key to many behavioral studies, both in the laboratory and in the field, that have not been possible before. From such big-data behavioral studies we expect many novel insights into the sensory ecology and into social and communication behaviors of these fascinating fishes (Henninger et al., 2018; Raab et al., 2019, 2021; Fortune et al., 2020), that also impact the way we study sensory processing.

Data availability statement

The original contributions presented in the study are included in the article/supplementary materials, further inquiries can be directed to the corresponding author/s.

Ethics statement

Ethical review and approval was not required for the animal study because the illustrated data includes observational recordings of electric fish in the wild (no ethics approval required since recordings were purely observational). Laboratory recordings illustrated in Figure 1E were approved by the Regierungspräsidium Tübingen (permit no. ZP 04/20 G).

Author contributions

The algorithm described and presented in this manuscript was developed by TR but builds upon collective or individual ideas shared between all other authors. MM, RJ, and NC developed the idea to utilize spatial electric field properties of individual electric fish as a feature to track their EODs (Madhav et al., 2018). JH and JB provided the tools required for detecting and extracting electric signals of wave-type electric fish in multi-electrode grid recordings and developed the first algorithmic approaches to track EODs of electric fish by means of their individual specific EOD frequency (Henninger et al., 2020). TR developed the here presented tracking algorithm which combines and refines previous approaches, including the implementation of the combined signal distance, which considers both electric field difference and EOD frequency, to determine signal similarities, as well as the idea to tack signals according to their similarity (combined signal distance) in discrete tracking windows, developed software tools to inspect and post-process tracked EOD traces, illustrated and evaluated the algorithm, and wrote the manuscript. Many of the ideas realized in the presented algorithm originated from collaborations and numerous discussion of TR with other authors. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by Deutsche Forschungsgemeinschaft, Open Access Publishing Fund of University of Tübingen, the Center of Integrative Neuroscience at the University of Tübingen through the mini RTG Sensory Flow Processing across Modalities and Species, and the National Science Foundation under grand no. 1557858 to NC a Kavli NDI Distiguished Postdoctoral Fellowship to MM.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

References