Temporal Scale Selection in Time-Causal Scale Space

When processing sensory data by automatic methods in areas of signal processing such as computer vision or audio processing or in computational modelling of biological perception, the notion of receptive field constitutes an essential concept [2, 14, 15, 30, 31, 90].

For sensory data as obtained from vision or hearing, or their counterparts in artificial perception, the measurement from a single light sensor in a video camera or on the retina, or the instantaneous sound pressure registered by a microphone is hardly meaningful at all, since any such measurement is strongly dependent on external factors such as the illumination of a visual scene regarding vision or the distance between the sound source and the microphone regarding hearing. Instead, the essential information is carried by the relative relations between local measurements at different points and temporal moments regarding vision or local measurements over different frequencies and temporal moments regarding hearing. Following this paradigm, sensory measurements should be performed over local neighbourhoods over space–time regarding vision and over local neighbourhoods in the time–frequency domain regarding hearing, leading to the notions of spatio-temporal and spectro-temporal receptive fields.

Specifically, spatio-temporal receptive fields constitute a main class of primitives for expressing methods for video analysis [35, 39, 45, 47, 94, 98, 108, 118,119,120, 124], whereas spectro-temporal receptive fields constitute a main class of primitives for expressing methods for machine hearing [4, 18, 29, 40, 87, 96, 97, 103, 123].

A general problem when applying the notion of receptive fields in practice, however, is that the types of responses that are obtained in a specific situation can be strongly dependent on the scale levels at which they are computed. A spatio-temporal receptive field is determined by at least a spatial scale parameter and a temporal scale parameter, whereas a spectro-temporal receptive field is determined by at least a spectral and a temporal scale parameter. Beyond ensuring that local sensory measurements at different spatial, temporal and spectral scales are treated in a consistent manner, which by itself provides strong contraints on the shapes of the receptive fields [68, 75, 78, 79], it is necessary for computer vision or machine hearing algorithms to decide what responses within the families of receptive fields over different spatial, temporal and spectral scales they should base their analysis on.

Over the spatial domain, theoretically well-founded methods have been developed for choosing spatial scale levels among receptive field responses over multiple spatial scales [61, 62, 64, 70, 71] leading to, e.g. robust methods for image-based matching and recognition [5, 48, 85, 88, 114, 115, 117] that are able to handle large variations of the size of the objects in the image domain and with numerous applications regarding object recognition, object categorization, multi-view geometry, construction of 3-D models from visual input, human–computer interaction, biometrics and robotics.

Much less research has, however, been performed regarding the topic of choosing local appropriate scales in temporal data. While some methods for temporal scale selection have been developed [46, 60, 121], these methods suffer from either theoretical or practical limitations.

A main subject of this paper is present a theory for how to compare filter responses in terms of temporal derivatives that have been computed at different temporal scales, specifically with a detailed theoretical analysis of the possibilities of having temporal scale estimates as obtained from a temporal scale selection mechanism reflect the temporal duration of the underlying temporal structures that gave rise to the feature responses. Another main subject of this paper is to present a theoretical framework for temporal scale selection that leads to temporal scale invariance and enables the computation of scale-covariant temporal scale estimates. While these topics can for a non-causal temporal domain be addressed by the non-causal Gaussian scale-space concept [21, 32, 41, 43, 57, 58, 66, 112, 122], the development of such a theory has been missing regarding a time-causal temporal domain.

1.1 Temporal Scale Selection

When processing time-dependent signals in video or audio or more generally any temporal signal, special attention has to be put to the facts that:

• the physical phenomena that generate the temporal signals may occur at different speed—faster or slower, and

• the temporal signals may contain qualitatively different types of temporal structures at different temporal scales.

In certain controlled situations where the physical system that generates the temporal signals that is to be processed is sufficiently well known and if the variability of the temporal scales over time in the domain is sufficiently constrained, suitable temporal scales for processing the signals may in some situations be chosen manually and then be verified experimentally. If the sources that generate the temporal signals are sufficiently complex and/or if the temporal structures in the signals vary substantially in temporal duration by the underlying physical processes occurring significantly faster or slower, it is on the other hand natural to (i) include a mechanism for processing the temporal data at multiple temporal scales and (ii) try to detect in a bottom-up manner at what temporal scales the interesting temporal phenomena are likely to occur.

The subject of this article is to develop a theory for temporal scale selection in a time-causal temporal scale space as an extension of a previously developed theory for spatial scale selection in a spatial scale space [61, 62, 64, 70, 71], to generate bottom-up hypotheses about characteristic temporal scales in time-dependent signals, intended to serve as estimates of the temporal duration of local temporal structures in time-dependent signals. Special focus will be on developing mechanisms analogous to scale selection in non-causal Gaussian scale-space, based on local extrema over scales of scale-normalized derivatives, while expressed within the framework of a time-causal and time-recursive temporal scale space in which the future cannot be accessed and the signal processing operations are thereby only allowed to make use of information from the present moment and a compact buffer of what has occurred in the past.

When designing and developing such scale selection mechanisms, it is essential that the computed scale estimates reflect the temporal duration of the corresponding temporal structures that gave rise to the feature responses. To understand the prerequisites for developing such temporal scale selection methods, we will in this paper perform an in-depth theoretical analysis of the scale selection properties that such temporal scale selection mechanisms give rise to for different temporal scale-space concepts and for different ways of defining scale-normalized temporal derivatives.

Specifically, after an examination of the theoretical properties of different types of temporal scale-space concepts, we will focus on a class of recently extended time-causal temporal scale-space concepts obtained by convolution with truncated exponential kernels coupled in cascade [53, 73, 75, 77]. For two natural ways of distributing the discrete temporal scale levels in such a representation, in terms of either a uniform distribution over the scale parameter \(\tau \) corresponding to the variance of the composed scale-space kernel or a logarithmic distribution, we will study the scale selection properties that result from detecting local temporal scale levels from local extrema over scale of scale-normalized temporal derivatives. The motivation for studying a logarithmic distribution of the temporal scale levels, is that it corresponds to a uniform distribution in units of effective scale \(\tau _\mathrm{eff} = A + B \log \tau \) for some constants A and B, which has been shown to constitute the natural metric for measuring the scale levels in a spatial scale space [41, 55].

As we shall see from the detailed theoretical analysis that will follow, this will imply certain differences in scale selection properties of a temporally asymmetric time-causal scale space compared to scale selection in a spatially mirror symmetric Gaussian scale space. These differences in theoretical properties are in turn essential to take into explicit account when formulating algorithms for temporal scale selection, in e.g. video analysis or audio analysis applications.

For the temporal scale-space concept based on a uniform distribution of the temporal scale levels in units of the variance of the composed scale-space kernel, it will be shown that temporal scale selection from local extrema over temporal scales will make it possible to estimate the temporal duration of local temporal structures modelled as local temporal peaks and local temporal ramps. For a dense temporal structure modelled as a temporal sine wave, the lack of true scale invariance for this concept will, however, imply that the temporal scale estimates will not be directly proportional to the wavelength of the temporal sine wave. Instead, the scale estimates are affected by a bias, which is not a desirable property.

For the temporal scale-space concept based on a logarithmic distribution of the temporal scale levels, and taken to the limit to scale-invariant time-causal limit kernel [75] corresponding to an infinite number of temporal scale levels that cluster infinitely close near the temporal scale level zero, it will on the other hand be shown that the temporal scale estimates of a dense temporal sine wave will be truly proportional to the wavelength of the signal. By a general proof, it will be shown this scale-invariant property of temporal scale estimates can also be extended to any sufficiently regular signal, which constitutes a general foundation for expressing scale invariant temporal scale selection mechanisms for time-dependent video and audio and more generally also other classes of time-dependent measurement signals.

As complement to this proposed overall framework for temporal scale selection, we will also present a set of general theoretical results regarding time-causal scale-space representations: (i) showing that previous application of the assumption of a semi-group property for time-causal scale-space concepts leads to undesirable temporal dynamics, which however can be remedied by replacing the assumption of a semi-group structure be a weaker assumption of a cascade property in turn based on a transitivity property, (ii) formulations of scale-normalized temporal derivatives for Koenderink’s time-causal scale-time model [42], and (iii) ways of translating the temporal scale estimates from local extrema over temporal scales in the temporal scale-space representation based on the scale-invariant time-causal limit kernel into quantitative measures of the temporal duration of the corresponding underlying temporal structures and in turn based on a scale-time approximation of the limit kernel.

In these ways, this paper is intended to provide a theoretical foundation for expressing theoretically well-founded temporal scale selection methods for selecting local temporal scales over time-causal temporal domains, such as video and audio with specific focus on real-time image or sound streams. Applications of this scale selection methodology for detecting both sparse and dense spatio-temporal features in video are presented in a companion paper [74].

1.2 Structure of this Article

As a conceptual background to the theoretical developments that will be performed, we will start in Sect. 2 with an overview of different approaches to handling temporal data within the scale-space framework including a comparison of relative advantages and disadvantages of different types of temporal scale-space concepts.

As a theoretical baseline for the later developments of methods for temporal scale selection in a time-causal scale space, we shall then in Sect. 3 give an overall description of basic temporal scale selection properties that will hold if the non-causal Gaussian scale-space concept with its corresponding selection methodology for a spatial image domain is applied to a one-dimensional non-causal temporal domain, e.g. for the purpose of handling the temporal domain when analysing pre-recorded video or audio in an offline setting.

In Sects. 4, 5 we will then continue with a theoretical analysis of the consequences of performing temporal scale selection in the time-causal scale space obtained by convolution with truncated exponential kernels coupled in cascade [53, 73, 75, 77]. By selecting local temporal scales from the scales at which scale-normalized temporal derivatives assume local extrema over temporal scales, we will analyse the resulting temporal scale selection properties for two ways of defining scale-normalized temporal derivatives, by either variance-based normalization as determined by a scale normalization parameter \(\gamma \) or \(L_p\)-normalization for different values of the scale normalization power p.

With the temporal scale levels required to be discrete because of the very nature of this temporal scale-space concept, we will specifically study two ways of distributing the temporal scale levels over scale, using either a uniform distribution relative to the temporal scale parameter \(\tau \) corresponding to the variance of the composed temporal scale-space kernel in Sect. 4 or a logarithmic distribution of the temporal scale levels in Sect. 5.

Because of the analytically simpler form for the time-causal scale-space kernels corresponding to a uniform distribution of the temporal scale levels, some theoretical scale-space properties will turn out to be easier to study in closed form for this temporal scale-space concept. We will specifically show that for a temporal peak modelled as the impulse response to a set of truncated exponential kernels coupled in cascade, the selected temporal scale level will serve as a good approximation of the temporal duration of the peak or be proportional to this measure depending on the value of the scale normalization parameter \(\gamma \) used for scale-normalized temporal derivatives based on variance-based normalization or the scale normalization power p for scale-normalized temporal derivatives based on \(L_p\)-normalization. For a temporal onset ramp, the selected temporal scale level will on the other hand be either a good approximation of the time constant of the onset ramp or proportional to this measure of the temporal duration of the ramp. For a temporal sine wave, the selected temporal scale level will, however, not be directly proportional to the wavelength of the signal, but instead affected by a systematic bias. Furthermore, the corresponding scale-normalized magnitude measures will not be independent of the wavelength of the sine wave but instead show systematic wavelength-dependent deviations. A main reason for this is that this temporal scale-space concept does not guarantee temporal scale invariance if the temporal scale levels are distributed uniformly in terms of the temporal scale parameter \(\tau \) corresponding to the temporal variance of the temporal scale-space kernel.

With a logarithmic distribution of the temporal scale levels, we will on the other hand show that for the temporal scale-space concept defined by convolution with the time-causal limit kernel [75] corresponding to an infinitely dense distribution of the temporal scale levels towards zero temporal scale, the temporal scale estimates will be perfectly proportional to the wavelength of a sine wave for this temporal scale-space concept. It will also be shown that this temporal scale-space concept leads to perfect scale invariance in the sense that (i) local extrema over temporal scales are preserved under temporal scaling factors corresponding to integer powers of the distribution parameter c of the time-causal limit kernel underlying this temporal scale-space concept and are transformed in a scale-covariant way for any temporal input signal and (ii) if the scale normalization parameter \(\gamma = 1\) or equivalently if the scale normalization power \(p = 1\), the magnitude values at the local extrema over scale will be equal under corresponding temporal scaling transformations. For this temporal scale-space concept, we can therefore fulfil basic requirements to achieve temporal scale invariance also over a time-causal and time-recursive temporal domain.

To simplify the theoretical analysis, we will in some cases temporarily extend the definitions of temporal scale-space representations over discrete temporal scale levels to a continuous scale variable, to make it possible to compute local extrema over temporal scales from differentiation with respect to the temporal scale parameter. Section 6 discusses the influence that this approximation has on the overall theoretical analysis.

Section 7 then illustrates how the proposed theory for temporal scale selection can be used for computing local scale estimates from 1-D signals with substantial variabilities in the characteristic temporal duration of the underlying structures in the temporal signal.

In Sect. 8, we analyse how the derived scale selection properties carry over to a set of spatio-temporal feature detectors defined over both multiple spatial scales and multiple temporal scales in a time-causal spatio-temporal scale-space representation for video analysis. Section 9 then outlines how corresponding selection of local temporal and logspectral scales can be expressed for audio analysis operations over a time-causal spectro-temporal domain. Finally, Sect. 10 concludes with a summary and discussion.

To simplify the presentation, we have put some derivations and theoretical analysis in the appendix. “Appendix 1” presents a general theoretical argument of why a requirement about a semi-group property over temporal scales will lead to undesirable temporal dynamics for a time-causal scale space and argue that the essential structure of non-creation of new image structures from any finer to any coarser temporal scale can instead nevertheless be achieved with the less restrictive assumption about a cascade smoothing property over temporal scales, which then allows for better temporal dynamics in terms of shorter temporal delays.

In relation to Koenderink’s scale-time model [42], “Appendix 2” shows how corresponding notions of scale-normalized temporal derivatives based on either variance-based normalization or \(L_p\)-normalization can be defined also for this time-causal temporal scale-space concept.

“Appendix 3” shows how the temporal duration of the time-causal limit kernel proposed in [75] can be estimated by a scale-time approximation of the limit kernel via Koenderink’s scale-time model leading to estimates of how a selected temporal scale level \(\hat{\tau }\) from local extrema over temporal scale can be translated into a estimates of the temporal duration of temporal structures in the temporal scale-space representation obtained by convolution with the time-causal limit kernel. Specifically, explicit expressions are given for such temporal duration estimates based on first- and second-order temporal derivatives.

Temporal scale-space kernels with composed temporal variance \(\tau = 1\) for the main types of temporal scale-space concepts considered in this paper and with their first- and second-order temporal derivatives: (top row) the non-causal Gaussian kernel \(g(t;\; \tau )\), (second row) the composition \(h(t;\; \mu , K=10)\) of \(K = 10\) truncated exponential kernels with equal time constants, (third row) the composition \(h(t;\; K=10, c = \sqrt{2})\) of \(K = 10\) truncated exponential kernels with logarithmic distribution of the temporal scale levels for \(c = \sqrt{2}\), (fourth row) corresponding kernels \(h(t;\; K=10, c = 2)\) for \(c = 2\), (fifth row) Koenderink’s scale-time kernels \(h_{\mathrm{Koe}}(t;\; c = \sqrt{2})\) corresponding to Gaussian convolution over a logarithmically transformed temporal axis with the parameters determined to match the time-causal limit kernel corresponding to truncated exponential kernels with an infinite number of logarithmically distributed temporal scale levels according to (186) for \(c = \sqrt{2}\), (bottom row) corresponding scale-time kernels \(h_{\mathrm{Koe}}(t;\; c = 2)\) for \(c = 2\) (horizontal axis time t)

Full size image

2.1 Temporal Scale-Space Concepts

For processing temporal signals at multiple temporal scales, different types of temporal scale-space concepts have been developed in the computer vision literature (see Fig. 1).

For offline processing of pre-recorded signals, a non-causal Gaussian temporal scale-space concept may in many situations be sufficient. A Gaussian temporal scale-space concept is constructed over the 1-D temporal domain in a similar manner as a Gaussian spatial scale-space concept is constructed over a D-dimensional spatial domain (Florack [21]; Iijima [32]; Koenderink [41]; Koenderink and van Doorn [43]; Lindeberg [57, 58, 66]; ter Haar Romeny [112]; Witkin [122]), with or without the difference that a model for temporal delays may or may not be additionally included [66].

When processing temporal signals in real time, or when modelling sensory processes in biological perception computationally, it is on the other hand necessary to base the temporal analysis on time-causal operations.

The first time-causal temporal scale-space concept was developed by Koenderink [42], who proposed to apply Gaussian smoothing on a logarithmically transformed time axis with the present moment mapped to the unreachable infinity. This temporal scale-space concept does, however, not have any known time-recursive formulation. Formally, it requires an infinite memory of the past and has therefore not been extensively applied in computational applications.

Lindeberg [53, 73, 75] and Lindeberg and Fagerström [77] proposed a time-causal temporal scale-space concept based on truncated exponential kernels or equivalently first-order integrators coupled in cascade, based on theoretical results by Schoenberg [104] (see also Karlin [38] and Schoenberg [105]) implying that such kernels are the only variation-diminishing kernels over a 1-D temporal domain that guarantee non-creation of new local extrema or equivalently zero-crossings with increasing temporal scale. This temporal scale-space concept is additionally time-recursive and can be implemented in terms of computationally highly efficient first-order integrators or recursive filters over time. This theory has been recently extended into a scale-invariant time-causal limit kernel [75], which allows for scale invariance over the temporal scaling transformations that correspond to exact mappings between the temporal scale levels in the temporal scale-space representation based on a discrete set of logarithmically distributed temporal scale levels.

Based on semi-groups that guarantee either self-similarity over temporal scales or non-enhancement of local extrema with increasing temporal scales, Fagerström [19] and Lindeberg [66] have derived time-causal semi-groups that allow for a continuous temporal scale parameter and studied theoretical properties of these kernels.

Concerning temporal processing over discrete time, Fleet and Langley [20] performed temporal filtering for optic flow computations based on recursive filters over time. Lindeberg [53, 73, 75] and Lindeberg and Fagerström [77] showed that first-order recursive filters coupled in cascade constitutes a natural time-causal scale-space concept over discrete time, based on the requirement that the temporal filtering over a 1-D temporal signal must not increase the number of local extrema or equivalently the number of zero-crossings in the signal. In the specific case when all the time constants in this model are equal and tend to zero while simultaneously increasing the number of temporal smoothing steps in such a way that the composed temporal variance is held constant, these kernels can be shown to approach the temporal Poisson kernel [77]. If on the other hand the time constants of the first-order integrators are chosen so that the temporal scale levels become logarithmically distributed, these temporal smoothing kernels approach a discrete approximation of the time-causal limit kernel [75].

Applications of using these linear temporal scale-space concepts for modelling the temporal smoothing step in visual and auditory receptive fields have been presented by Lindeberg [59, 65, 66, 68, 69, 73, 75], ter Haar Romeny et al. [113], Lindeberg and Friberg [78, 79] and Mahmoudi [86]. Nonlinear spatio-temporal scale-space concepts have been proposed by Guichard [24]. Applications of the non-causal Gaussian temporal scale-space concept for computing spatio-temporal features have been presented by Laptev and Lindeberg [45,46,47], Klaser et al. [39], Willems et al. [121], Wang et al. [118], Shao and Mattivi [108] and others, see specifically Poppe [98] for a survey of early approaches to vision-based human action recognition, Jhuang et al. [35] and Niebles et al. [94] for conceptually related non-causal Gabor approaches, Adelson and Bergen [1] and Derpanis and Wildes [16] for closely related spatio-temporal orientation models and Han et al. [26] for a related mid-level temporal representation termed the video primal sketch.

Applications of the temporal scale-space model based on truncated exponential kernels with equal time constants coupled in cascade and corresponding to Laguerre functions (Laguerre polynomials multiplied by a truncated exponential kernel) for computing spatio-temporal features have presented by Rivero-Moreno and Bres [100], Shabani et al. [107] and Berg et al. [116] as well as for handling time scales in video surveillance (Jacob and Pless [33]), for performing edge preserving smoothing in video streams (Paris [95]) and is closely related to Tikhonov regularization as used for image restoration by Surya et al. [111]. A general framework for performing spatio-temporal feature detection based on the temporal scale-space model based on truncated exponential kernels coupled in cascade with specifically the both theoretical and practical advantages of using logarithmic distribution of the intermediated temporal scale levels in terms of temporal scale invariance and better temporal dynamics (shorter temporal delays) has been presented in Lindeberg [75].

2.2 Relative Advantages of Different Temporal Scale Spaces

When developing a temporal scale selection mechanism over a time-causal temporal domain, a first problem concerns what time-causal scale-space concept to base the multi-scale temporal analysis upon. The above reviewed temporal scale-space concepts have different relative advantages from a theoretical and computational viewpoint. In this section, we will perform an in-depth examination of the different temporal scale-space concepts that have been developed in the literature, which will lead us to a class of time-causal scale-space concepts that we argue is particularly suitable with respect to the set of desirable properties we aim at.

The non-causal Gaussian temporal scale space is in many cases the conceptually easiest temporal scale-space concept to handle and to study analytically [66]. The corresponding temporal kernels are scale invariant, have compact closed-form expressions over both the temporal and frequency domains and obey a semi-group property over temporal scales. When applied to pre-recorded signals, temporal delays can if desirable be disregarded, which eliminates any need for temporal delay compensation. This scale-space concept is, however, not time-causal and not time-recursive, which implies fundamental limitations with regard to real-time applications and realistic modelling of biological perception.

Koenderink’s scale-time kernels [42] are truly time-causal, allow for a continuous temporal scale parameter, have good temporal dynamics and have a compact explicit expression over the temporal domain. These kernels are, however, not time-recursive, which implies that they in principle require an infinite memory of the past (or at least extended temporal buffers corresponding to the temporal extent to which the infinite support temporal kernels are truncated at the tail). Thereby, the application of Koenderink’s scale-time model to video analysis implies that substantial temporal buffers are needed when implementing this non-recursive temporal scale-space in practice. Similar problems with substantial need for extended temporal buffers arise when applying the non-causal Gaussian temporal scale-space concept to offline analysis of extended video sequences. The algebraic expressions for the temporal kernels in the scale-time model are furthermore not always straightforward to handle and there is no known simple expression for the Fourier transform of these kernels or no known simple explicit cascade smoothing property over temporal scales with respect to the regular (untransformed) temporal domain. Thereby, certain algebraic calculations with the scale-time kernels may become quite complicated.

The temporal scale-space kernels obtained by coupling truncated exponential kernels or equivalently first-order integrators in cascade are both truly time-causal and truly time-recursive [53, 73, 75, 77]. The temporal scale levels are on the other hand required to be discrete. If the goal is to construct a real-time signal processing system that analyses continuous streams of signal data in real time, one can however argue that a restriction of the theory to a discrete set of temporal scale levels is less of a constraint, since the signal processing system anyway has to be based on a finite amount of sensors and hardware/wetware for sampling and processing the continuous stream of signal data.

In the special case when all the time constants are equal, the corresponding temporal kernels in the temporal scale-space model based on truncated exponential kernels coupled in cascade have compact explicit expressions that are easy to handle both in the temporal domain and in the frequency domain, which simplifies theoretical analysis. These kernels obey a semi-group property over temporal scales, but are not scale invariant and lead to slower temporal dynamics when a larger number of primitive temporal filters are coupled in cascade [73, 75].

In the special case when the temporal scale levels in this scale-space model are logarithmically distributed, these kernels have a manageable explicit expression over the Fourier domain that enables some closed-form theoretical calculations. Deriving an explicit expression over the temporal domain is, however, harder, since the explicit expression then corresponds to a linear combination of truncated exponential filters for all the time constants, with the coefficients determined from a partial fraction expansion of the Fourier transform, which may lead to rather complex closed-form expressions. Thereby certain analytical calculations may become harder to handle. As shown in [75] and “Appendix 3”, some such calculations can on the other hand be well approximated via a scale-time approximation of the time-causal temporal scale-space kernels. When using a logarithmic distribution of the temporal scales, the composed temporal kernels do however have very good temporal dynamics and much better temporal dynamics compared to corresponding kernels obtained by using truncated exponential kernels with equal time constants coupled in cascade. Moreover, these kernels lead to a computationally very efficient numerical implementation. Specifically, these kernels allow for the formulation of a time-causal limit kernel that obeys scale invariance under temporal scaling transformations, which cannot be achieved if using a uniform distribution of the temporal scale levels [73, 75].

The temporal scale-space representations obtained from the self-similar time-causal semi-groups have a continuous scale parameter and obey temporal scale invariance [19, 66]. These kernels do, however, have less desirable temporal dynamics (see “Appendix 1” for a general theoretical argument about undesirable consequences of imposing a temporal semi-group property on temporal kernels with temporal delays) and/or lead to pseudodifferential equations that are harder to handle both theoretically and in terms of computational implementation. For these reasons, we shall not consider those time-causal semi-groups further in this treatment.

2.3 Previous Work on Methods for Scale Selection

A general framework for performing scale selection for local differential operations was proposed in Lindeberg [56, 57] based on the detection of local extrema over scale of scale-normalized derivative expressions and then refined in Lindeberg [61, 62]—see Lindeberg [64, 71] for tutorial overviews.

This scale selection approach has been applied to a large number of feature detection tasks over spatial image domains including detection of scale-invariant interest points (Lindeberg [62, 70]; Mikolajczyk and Schmid [88]; Tuytelaars and Mikolajczyk [115]), performing feature tracking (Bretzner and Lindeberg [9]), computing shape from texture and disparity gradients (Gårding and Lindeberg [23]; Lindeberg and Gårding [80]), detecting 2-D and 3-D ridges (Frangi et al. [22]; Krissian et al. [44]; Lindeberg [61]; Sato et al. [102]), computing receptive field responses for object recognition (Chomat et al. [12], Hall et al. [25]), performing hand tracking and hand gesture recognition (Bretzner et al. [8]) and computing time-to-collision (Negre et al. [93]).

Specifically, very successful applications have been achieved in the area of image-based matching and recognition (Bay et al. [5]; Lindeberg [67, 72]; Lowe [85]). The combination of local scale selection from local extrema of scale-normalized derivatives over scales (Lindeberg [57, 62]) with affine shape adaptation (Lindeberg and Gårding [81]) has made it possible to perform multi-view image matching over large variations in viewing distances and viewing directions (Lazebnik et al. [49]; Mikolajczyk and Schmid [88]; Mikolajczyk et al. [89]; Rothganger et al. [101]; Tuytelaars and van Gool [114]). The combination of interest point detection from scale-space extrema of scale-normalized differential invariants (Lindeberg [57, 62]) with local image descriptors (Bay et al. [5]; Lowe [85]) has made it possible to design robust methods for performing object recognition of natural objects in natural environments with numerous applications to object recognition (Bay et al. [5]; Lowe [85]), object category classification (Bosch et al. [7]; Mutch and Lowe [92]), multi-view geometry (Hartley and Zisserman [27]), panorama stitching (Brown and Lowe [11]), automated construction of 3-D object and scene models from visual input (Agarwal et al. [3]; Brown and Lowe [10]), synthesis of novel views from previous views of the same object (Liu et al. [82]), visual search in image databases (Datta et al. [13]; Lew et al. [50]), human–computer interaction based on visual input (Jaimes and Sebe [34]; Porta [99]), biometrics (Bicego et al. [6]; Li [51]) and robotics (Se et al. [106]; Siciliano and Khatib [109]).

Alternative approaches for performing scale selection over spatial image domains have also been proposed in terms of (i) detecting peaks of weighted entropy measures (Kadir and Brady [36]) or Lyaponov functionals (Sporring et al. [110]) over scales, (ii) minimizing normalized error measures over scale (Lindeberg [63]), (iii) determining minimum reliable scales for edge detection based on a noise suppression model (Elder and Zucker [17]), (iv) determining at what scale levels to stop in nonlinear diffusion-based image restoration methods based on similarity measurements relative to the original image data (Mrazek and Navara [91]), (v) by comparing reliability measures from statistical classifiers for texture analysis at multiple scales (Kang et al. [37]), (vi) by computing image segmentations from the scales at which a supervised classifier delivers class labels with the highest reliability measure (Li et al. [52]; Loog et al. [84]), (vii) selecting scales for edge detection by estimating the saliency of elongated edge segments (Liu et al. [83]) or (viii) considering subspaces generated by local image descriptors computed over multiple scales (Hassner et al. [28]).

More generally, spatial scale selection can be seen as a specific instance of computing invariant receptive field responses under natural image transformations, to (i) handle objects in the world of different physical size and to account for scaling transformations caused by the perspective mapping, and with extensions to (ii) affine image deformations to account for variations in the viewing direction and (iii) Galilean transformations to account for relative motions between objects in the world and the observer as well as to (iv) illumination variations [69].

Early theoretical work on temporal scale selection in a time-causal scale space was presented in Lindeberg [60] with primary focus on the temporal Poisson scale-space, which possesses a temporal semi-group structure over a discrete time-causal temporal domain while leading to long temporal delays (see “Appendix 1” for a general theoretical argument). Temporal scale selection in non-causal Gaussian spatio-temporal scale space has been used by Laptev and Lindeberg [46] and Willems et al. [121] for computing spatio-temporal interest points, however, with certain theoretical limitations that are explained in a companion paper [74].^{Footnote 1} The purpose of this article is to present a much further developed and more general theory for temporal scale selection in time-causal scale spaces over continuous temporal domains and to analyse the theoretical scale selection properties for different types of model signals.

In this section, we will present an overview of theoretical properties that will hold if the Gaussian temporal scale-space concept is applied to a non-causal temporal domain, if additionally the scale selection mechanism that has been developed for a non-causal spatial domain is directly transferred to a non-causal temporal domain. The set of temporal scale-space properties that we will arrive at will then be used as a theoretical baseline for developing temporal scale-space properties over a time-causal temporal domain.

3.1 Non-causal Gaussian Temporal Scale-Space

Over a one-dimensional temporal domain, axiomatic derivations of a temporal scale-space representation based on the assumptions of (i) linearity, (ii) temporal shift invariance, (iii) semi-group property over temporal scale, (iv) sufficient regularity properties over time and temporal scale and (v) non-enhancement of local extrema imply that the temporal scale-space representation

should be generated by convolution with possibly time-delayed temporal kernels of the form [66]

where \(\tau \) is a temporal scale parameter corresponding to the variance of the Gaussian kernel and \(\delta \) is a temporal delay. Differentiating the kernel with respect to time gives

see the top row in Fig. 1 for graphs. When analysing pre-recorded temporal signals, it can be preferable to set the temporal delay to zero, leading to temporal scale-space kernels having a similar form as spatial Gaussian kernels:

3.2 Temporal Scale Selection from Scale-Normalized Derivatives

As a conceptual background to the treatments that we shall later develop regarding temporal scale selection in time-causal temporal scale spaces, we will in this section describe the theoretical structure that arises by transferring the theory for scale selection in a Gaussian scale space over a spatial domain to the non-causal Gaussian temporal scale space:

Given the temporal scale-space representation \(L(t;\; \tau )\) of a temporal signal f(t) obtained by convolution with the Gaussian kernel \(g(t;\; \tau )\) according to (1), temporal scale selection can be performed by detecting local extrema over temporal scales of differential expressions expressed in terms of scale-normalized temporal derivatives at any scale \(\tau \) according to [61, 62, 64, 71]

where \(\zeta = t/\tau ^{\gamma /2}\) is the scale-normalized temporal variable, n is the order of temporal differentiation and \(\gamma \) is a free parameter. It can be shown [62, Section 9.1] that this notion of \(\gamma \)-normalized derivatives corresponds to normalizing the nth-order Gaussian derivatives \(g_{\zeta ^n}(t;\; \tau )\) over a one-dimensional domain to constant \(L_p\)-norms over scale \(\tau \)

with

where the perfectly scale-invariant case \(\gamma = 1\) corresponds to \(L_1\)-normalization for all orders n of temporal differentiation.

Temporal scale invariance A general and very useful scale-invariant property that results from this construction of the notion of scale-normalized temporal derivatives can be stated as follows: consider two signals f and \(f'\) that are related by a temporal scaling transformation

and assume that there is a local extremum over scales at \((t_0;\; \tau _0)\) in a differential expression \(\mathcal{D}_{\gamma {\text {-norm}}} L\) defined as a homogeneous polynomial of Gaussian derivatives computed from the scale-space representation L of the original signal f. Then, there will be a corresponding local extremum over scales at \((t_0';\; \tau _0') = (S \, t_0;\; S^2 \tau _0)\) in the corresponding differential expression \(\mathcal{D}_{\gamma {\text {-norm}}} L'\) computed from the scale-space representation \(L'\) of the rescaled signal \(f'\) [62, Section 4.1].

This scaling result holds for all homogeneous polynomial differential expression and implies that local extrema over scales of \(\gamma \)-normalized derivatives are preserved under scaling transformations. Specifically, this scale-invariant property implies that if a local scale temporal level in dimension of time \(\sigma = \tau \) is selected to be proportional to the temporal scale estimate \(\hat{\sigma } = \sqrt{\hat{\tau }}\) such that \(\sigma = C \, \hat{\sigma }\), then if the temporal signal f is transformed by a temporal scale factor S, the temporal scale estimate and therefore also the selected temporal scale level will be transformed by a similar temporal factor \(\hat{\sigma }' = S \, \hat{\sigma }\), implying that the selected temporal scale levels will automatically adapt to variations in the characteristic temporal scale of the signal. Thereby, such local extrema over temporal scale provide a theoretically well-founded way to automatically adapt the scale levels to local scale variations.

Specifically, scale-normalized scale-space derivatives of order n at corresponding temporal moments will be related according to

which means that \(\gamma = 1\) implies perfect scale invariance in the sense that the \(\gamma \)-normalized derivatives at corresponding points will be equal. If \(\gamma \ne 1\), the difference in magnitude can on the other hand be easily compensated for using the scale values of the corresponding scale-adaptive image features (see below).

3.3 Temporal Peak

For a temporal peak modelled as a Gaussian function with variance \(\tau _0\)

it can be shown that scale selection from local extrema over scale of second-order scale-normalized temporal derivatives

implies that the scale estimate at the position \(t = 0\) of the peak will be given by Lindeberg [61, Equation (56)] [70, Equation (212)]

If we require the scale estimate to reflect the temporal duration of the peak such that

then this implies

which in the specific case of \(q = 1\) corresponds to [61, Section 5.6.1]

and in turn corresponding to \(L_p\)-normalization for \(p = 2/3\) according to (8).

If we additionally renormalize the original Gaussian peak to having maximum value equal to one

then if using the same value of \(\gamma \) for computing the magnitude response as for selecting the temporal scale, the maximum magnitude value over scales will be given by

and will not be independent of the temporal scale \(\tau _0\) of the original peak unless \(\gamma = 1\). If on the other hand using \(\gamma = 3/4\) as motivated by requirements of scale calibration (14) for \(q = 1\), the scale dependency will for a Gaussian peak be of the form

To get a scale-invariant magnitude measure for comparing the responses of second-order temporal derivative responses at different temporal scales for the purpose of scale calibration, we should therefore consider a scale-invariant magnitude measure for peak detection of the form

which for a Gaussian temporal peak will assume the value

Specifically, this form of post-normalization corresponds to computing the scale-normalized derivatives for \(\gamma = 1\) at the selected scale (14) of the temporal peak, which according to (8) corresponds to \(L_1\)-normalization of the second-order temporal derivative kernels.

3.4 Temporal Onset Ramp

If we model a temporal onset ramp with temporal duration \(\tau _0\) as the primitive function of the Gaussian kernel with variance \(\tau _0\)

it can be shown that scale selection from local extrema over scale of first-order scale-normalized temporal derivatives

implies that the scale estimate at the central position \(t = 0\) will be given by Lindeberg [61, Equation (23)]

If we require this scale estimate to reflect the temporal duration of the ramp such that

then this implies

which in the specific case of \(q = 1\) corresponds to [61, Section 4.5.1]

and in turn corresponding to \(L_p\)-normalization for \(p = 2/3\) according to (8).

If using the same value of \(\gamma \) for computing the magnitude response as for selecting the temporal scale, the maximum magnitude value over scales will be given by

which is not independent of the temporal scale \(\tau _0\) of the original onset ramp unless \(\gamma = 1\). If using \(\gamma = 1\) for temporal scale selection, the selected temporal scale according to (24) would, however, become infinite. If on the other hand using \(\gamma = 1/2\) as motivated by requirements of scale calibration (25) for \(q = 1\), the scale dependency will for a Gaussian onset ramp be of the form

To get a scale-invariant magnitude measure for comparing the responses of first-order temporal derivative responses at different temporal scales, we should therefore consider a scale-invariant magnitude measure for ramp detection of the form

which for a Gaussian onset ramp will assume the value

Specifically, this form of post-normalization corresponds to computing the scale-normalized derivatives for \(\gamma = 1\) at the selected scale (25) of the onset ramp and thus also to \(L_p\)-normalization of the first-order temporal derivative kernels for \(p = 1\).

3.5 Temporal Sine Wave

For a signal defined as a temporal sine wave

it can be shown that there will be a peak over temporal scales in the magnitude of the nth-order temporal derivative \(L_{\zeta ^n} = \tau ^{n\gamma /2} L_{t^n}\) at temporal scale [62, Section 3]

If we define a temporal scale parameter \(\sigma \) of dimension \([\text{ time }]\) according to \(\sigma = \sqrt{\tau }\), then this implies that the scale estimate is proportional to the wavelength \(\lambda _0 = 2\pi /\omega _0\) of the sine wave according to [62, Equation (9)]

and does in this respect reflect a characteristic time constant over which the temporal phenomena occur. Specifically, the maximum magnitude measure over scale [62, Equation (10)]

is for \(\gamma = 1\) independent of the angular frequency \(\omega _0\) of the sine wave and thereby scale invariant.

In the following, we shall investigate how these scale selection properties can be transferred to two types of time-causal temporal scale-space concepts.

In this section, we will present a theoretical analysis of the scale selection properties that are obtained in the time-causal scale-space based on truncated exponential kernels coupled in cascade, for the specific case of a uniform distribution of the temporal scale levels in units of the composed variance of the composed temporal scale-space kernels, and corresponding to the time constants of all the primitive truncated exponential kernels being equal.

We will study three types of idealized model signals for which closed-form theoretical analysis is possible: (i) a temporal peak modelled as a set of \(K_0\) truncated exponential kernels with equal time constants coupled in cascade, (ii) a temporal onset ramp modelled as the primitive function of the temporal peak model and (iii) a temporal sine wave. Specifically, we will analyse how the selected scale levels \(\hat{K}\) obtained from local extrema of temporal derivatives over scale relate to the temporal duration of a temporal peak or a temporal onset ramp alternatively how the selected scale levels \(\hat{K}\) depends on the wavelength of a sine wave.

We will also study how good approximation the scale-normalized magnitude measure at the maximum over temporal scales is compared to the corresponding fully scale-invariant magnitude measures that are obtained from the non-causal temporal scale concept as listed in Sect. 3.

4.1 Time-Causal Scale Space Based on Truncated Exponential Kernels with Equal Time Constants Coupled in Cascade

Given the requirements that the temporal smoothing operation in a temporal scale-space representation should obey (i) linearity, (ii) temporal shift invariance, (iii) temporal causality and (iv) guarantee non-creation of new local extrema or equivalently new zero-crossings with increasing temporal scale for any one-dimensional temporal signal, it can be shown [53, 73, 75, 77] that the temporal scale-space kernels should be constructed as a cascade of truncated exponential kernels of the form

If we additionally require the time constants of all such primitive kernels that are coupled in cascade to be equal, then this leads to a composed temporal scale-space kernel of the form

corresponding to Laguerre functions (Laguerre polynomials multiplied by a truncated exponential kernel) and also equal to the probability density function of the Gamma distribution having a Laplace transform of the form

Differentiating the temporal scale-space kernel with respect to time t gives

see the second row in Fig. 1 for graphs. The \(L_1\)-norms of these kernels are given by

The temporal scale level at level K corresponds to temporal variance \(\tau = K \mu ^2\) and temporal standard deviation \(\sigma = \sqrt{\tau } = \mu \sqrt{K}\).

4.2 Temporal Peak

Consider an input signal defined as a time-causal temporal peak corresponding to filtering a delta function with \(K_0\) first-order integrators with time constants \(\mu \) coupled in cascade:

With regard to the application area of vision, this signal can be seen as an idealized model of an object with temporal duration \(\tau _0 = K_0 \, \mu ^2\) that first appears and then disappears from the field of view, and modelled on a form to be algebraically compatible with the algebra of the temporal receptive fields. With respect to the application area of hearing, this signal can be seen as an idealized model of a beat sound over some frequency range of the spectrogram, also modelled on a form to be compatible with the algebra of the temporal receptive fields.

Define the temporal scale-space representation by convolving this signal with the temporal scale-space kernel (43) corresponding to K first-order integrators having the same time constants \(\mu \)

where we have applied the semi-group property that follows immediately from the corresponding Laplace transforms

By differentiating the temporal scale-space representation (44) with respect to time t, we obtain

implying that the maximum point is assumed at

and the inflection points at

This form of the expression for the time of the temporal maximum implies that the temporal delay of the underlying peak \(t_\mathrm{max,0} = \mu (K_0 - 1)\) and the temporal delay of the temporal scale-space kernel \(t_{\mathrm{max},U} = \mu (K - 1)\) are not fully additive, but instead composed according to

If we define the temporal duration d of the peak as the distance between the inflection points, if furthermore follows that this temporal duration is related to the temporal duration \(d_0 = 2 \mu \sqrt{K_0-1}\) of the original peak and the temporal duration \(d_U = 2 \mu \sqrt{K-1}\) of the temporal scale-space kernel according to

Notably these expressions are not scale invariant, but instead strongly dependent on a preferred temporal scale as defined by the time constant \(\mu \) of the primitive first-order integrators that define the uniform distribution of the temporal scales.

Scale-normalized temporal derivatives When using temporal scale normalization by variance-based normalization, the first- and second-order scale-normalized derivatives are given by

where \(\sigma = \sqrt{\tau }\), \(\tau = K \mu ^2\) and with \(L_t(t;\; \mu , K)\) and \(L_{tt}(t;\; \mu , K)\) according to (46) and (47).

When using temporal scale normalization by \(L_p\)-normalization, the first- and second-order scale-normalized derivatives are on the other hand given by Lindeberg [75, Equation (75)]

with the scale normalization factors \(\alpha _{n,p}(\mu , K)\) determined such that the \(L_p\)-norm of the scale-normalized temporal derivative computation kernel

equals the \(L_o\)-norm of some other reference kernel, where we here take the \(L_p\)-norm of the corresponding Gaussian derivative kernels [75, Equation (76)]

for \(\tau = \mu ^2 K\), thus implying

where \(G_{1,p}\) and \(G_{2,p}\) denote the \(L_p\)-norms (7) of corresponding Gaussian derivative kernels for the value of \(\gamma \) at which they become constant over scales by \(L_p\)-normalization, and the \(L_p\)-norms \(\Vert U_t(\cdot ;\; \mu , K) \Vert _p \) and \(\Vert U_{tt}(\cdot ;\; \mu , K) \Vert _p\) of the temporal scale-space kernels \(U_t\) and \(U_{tt}\) for the specific case of \(p = 1\) are given by (41) and (42).

Temporal scale selection Let us assume that we want to register that a new object has appeared by a scale-space extremum of the scale-normalized second-order derivative response.

To determine the temporal moment at which the temporal event occurs, we should formally determine the time where \(\partial _{\tau }(L_{\zeta \zeta }(t;\; \mu , K)) = 0\), which by our model (54) would correspond to solving a third-order algebraic equation. To simplify the problem, let us instead approximate the temporal position of the peak in the second-order derivative by the temporal position of the peak \(t_\mathrm{max}\) according to (48) in the signal and study the evolution properties over scale K of

In the case of variance-based normalization for a general value of \(\gamma \), we have

and in the case of \(L_p\)-normalization for \(p = 1\)

To determine the scale \(\hat{K}\) at which the local maximum is assumed, let us temporarily extend this definition to continuous values of K and differentiate the corresponding expressions with respect to K. Solving the equation

numerically for different values of \(K_0\) then gives the dependency on the scale estimate \(\hat{K}\) as function of \(K_0\) shown in Table 1 for variance-based normalization with either \(\gamma = 3/4\) or \(\gamma = 1\) and \(L_p\)-normalization for \(p = 1\).

As can be seen from the results in Table 1, when using variance-based scale normalization for \(\gamma = 3/4\), the scale estimate \(\hat{K}\) closely follows the scale \(K_0\) of the temporal peak and does therefore imply a good approximate transfer of the scale selection property (14) to this temporal scale-space concept. If one would instead use variance-based normalization for \(\gamma = 1\) or \(L_p\)-normalization for \(p = 1\), then that would, however, lead to substantial overestimates of the temporal duration of the peak.

Furthermore, if we additionally normalize the input signal to having unit contrast, then the corresponding time-causal correspondence to the post-normalized magnitude measure (20)

is for scale estimates proportional to the temporal duration of the underlying temporal peak \(\hat{K} \sim K_0\) very close to constant under variations of the temporal duration of the underlying temporal peak as determined by the parameter \(K_0\), thus implying a good approximate transfer of the scale selection property (21).

4.3 Temporal Onset Ramp

Consider an input signal defined as a time-causal onset ramp corresponding to the primitive function of \(K_0\) first-order integrators with time constants \(\mu \) coupled in cascade:

With respect to the application area of vision, this signal can be seen as an idealized model of a new object with temporal diffuseness \(\tau _0 = K_0 \, \mu ^2\) that appears in the field of view and modelled on a form to be algebraically compatible with the algebra of the temporal receptive fields. With respect to the application area of hearing, this signal can be seen as an idealized model of the onset of a new sound in some frequency band of the spectrogram, also modelled on a form to be compatible with the algebra of the temporal receptive fields.

Define the temporal scale-space representation of the signal by convolution with the temporal scale-space kernel (43) corresponding to K first-order integrators having the same time constants \(\mu \)

Then, the first-order temporal derivative is given by

which assumes its temporal maximum at \(t_\mathrm{ramp} = \mu (K_0 + K -1)\).

Temporal scale selection Let us assume that we are going to detect a new appearing object from a local maximum in the first-order derivative over both time and temporal scales. When using variance-based normalization for a general value of \(\gamma \), the scale-normalized response at the temporal maximum in the first-order derivative is given by

When using \(L_p\)-normalization for a general value of p, the corresponding scale-normalized response is

where the \(L_p\)-norm of the first-order scale-space derivative kernel can be expressed in terms of exponential functions, the Gamma function and hypergeometric functions, but is too complex to be written out here. Extending the definition of these expressions to continuous values of K and solving the equation

numerically for different values of \(K_0\) then gives the dependency on the scale estimate \(\hat{K}\) as function of \(K_0\) shown in Table 2 for variance-based normalization with \(\gamma = 1/2\) or \(L_p\)-normalization for \(p = 2/3\).

As can be seen from the numerical results, for both variance-based normalization and \(L_p\)-normalization with corresponding values of \(\gamma \) and p, the numerical scale estimates in terms of \(\hat{K}\) closely follow the diffuseness scale of the temporal ramp as parameterized by \(K_0\). Thus, for both of these scale normalization models, the numerical results indicate an approximate transfer of the scale selection property (14) to this temporal scale-space model. Additionally, the maximum magnitude values according to (69) can according to Stirling’s formula \({\varGamma }(n+1) \approx (n/e)^n \sqrt{2 \pi n}\) be approximated by

and are very stable under variations of the diffuseness scale \(K_0\) of the ramp, and thus implying a good transfer of the scale selection property (31) to this temporal scale-space concept.

4.4 Temporal Sine Wave

Consider a signal defined as a sine wave

This signal can be seen as a simplified model of a dense temporal texture with characteristic scale defined as the wavelength \(\lambda _0 =2\pi /\omega \) of the signal. In the application area of vision, this can be seen as an idealized model of watching some oscillating visual phenomena or watching a dense texture that moves relative the gaze direction. In the area of hearing, this could be seen as an idealized model of temporally varying frequencies around some fixed frequency in the spectrogram corresponding to vibrato.

Define the temporal scale-space representation of the signal by convolution with the temporal scale-space kernel (43) corresponding to K first-order integrators with equal time constants \(\mu \) coupled in cascade

where \(|\hat{U}(\omega _0;\; \mu , K)|\) and \(\arg \hat{U}(\omega ;\; \mu , K)\) denote the magnitude and the argument of the Fourier transform \(\hat{h}_\mathrm{composed}(\omega ;\; \mu , K)\) of the temporal scale-space kernel \(U(\cdot ;\; \mu , K)\) according to

By differentiating (74) with respect to time t, it follows that the magnitude of the nth-order temporal derivative is given by

Temporal scale selection Using variance-based temporal scale normalization, the magnitude of the corresponding scale-normalized temporal derivative is given by

Extending this expression to continuous values of K and differentiating with respect to K implies that the maximum over scale is assumed at scale

with the following series expansion for small values of \(\omega _0\) corresponding to temporal structures of longer temporal duration

Expressing the corresponding scale estimate \(\hat{\sigma }\) in terms of dimension length and parameterized in terms of the wavelength \(\lambda _0 = 2 \pi /\omega _0\) of the sine wave

we can see that the dominant term \(\frac{\sqrt{\gamma n} \lambda _0 }{2 \pi }\) is proportional to the temporal duration of the underlying structures in the signal and in agreement with the corresponding scale selection property (34) of the scale-invariant non-causal Gaussian temporal scale-space concept, whereas the overall expression is not scale invariant.

If the wavelength \(\lambda _0\) is much longer than the time constant \(\mu \) of the primitive first-order integrators, then the scale selection properties in this temporal scale-space model will constitute a better approximation of the corresponding scale selection properties in the scale-invariant non-causal Gaussian temporal scale-space model.

(Left column) Temporal scale estimates \(\hat{\sigma }\) (marked in blue) obtained from a sine wave using the maximum value over scale of the amplitude of either (top row) first-order scale-normalized temporal derivatives or (bottom row) second-order scale-normalized temporal derivatives and as function of the wavelength \(\lambda _0\) for variance-based temporal scale normalization with \(\gamma = 1/2\) for first-order derivatives and \(\gamma = 3/4\) for second-order derivatives. Note that for this temporal scale-space concept based on truncated exponential kernels with equal time constants, the temporal scale estimates \(\hat{\sigma }\) do not constitute a good approximation to the temporal scale estimates being proportional to the wavelength \(\lambda _0\) of the underlying sine wave. Instead, the temporal scale estimates are affected by a systematic scale-dependent bias. (Right column) The maximum scale-normalized magnitude for (top row) first-order scale-normalized temporal derivatives or (bottom row) second-order scale-normalized temporal derivatives as function of the wavelength \(\lambda _0\) using variance-based temporal scale normalization with \(\gamma = 1\) (marked in blue). The brown curves show corresponding scale estimates and magnitude values obtained from the scale-invariant but not time-causal Gaussian temporal scale space. Thus, when the temporal scale-space concept based on truncated exponential kernels with equal time constants coupled in cascade is applied to a sine wave, the maximum magnitude measures over temporal scales do not at all constitute a good approximation of temporal scale invariance (horizontal axis wavelength \(\lambda _0\) in units of \(\mu \))

Full size image

The maximum value over scale is

with the following series expansion for large \(\lambda _0=2\pi /\omega _0\) and \(\gamma = 1\):

Again we can note that the first term agrees with the corresponding scale selection property (35) for the scale-invariant non-causal Gaussian temporal scale space, whereas the higher-order terms are not scale invariant.

Figure 2 shows graphs of the scale estimate \(\hat{\sigma }\) according to (82) for \(n = 2\) and \(\gamma = 3/4\) and the maximum response over scale \(L_{\zeta ^n,\mathrm{ampl,max}}\) for \(n = 2\) and \(\gamma = 1\) as function of the wavelength \(\lambda _0\) of the sine wave (marked in blue). For comparison, we also show the corresponding scale estimates (34) and magnitude values (35) that would be obtained using temporal scale selection in the scale-invariant non-causal Gaussian temporal scale space (marked in brown).

As can be seen from the graphs, both the temporal scale estimate \(\hat{\sigma }(\lambda _0)\) and the maximum magnitude \(L_{\zeta ,\mathrm{ampl,max}}(\lambda _0)\) obtained from a set of first-order integrators with equal time constants coupled in cascade approach the corresponding results obtained from the non-causal Gaussian scale space for larger values of \(\lambda _0\) in relation to the time constant \(\mu \) of the first-order integrators. The scale estimate obtained from a set of first-order integrators with equal time constants is, however, for lower values of \(\lambda _0\) generally significantly higher than the scale estimates obtained from a non-causal Gaussian temporal scale space. The scale-normalized magnitude values, which should be constant over scale for a scale-invariant temporal scale space when \(\gamma = 1\) according to the scale selection property (35), are for lower values of \(\lambda _0\) much higher than the scale-invariant limit value when performing scale selection in the temporal scale-space concept obtained by coupling a set of first-order integrators with equal time constants in cascade. The scale selection properties (34) and (35) are consequently not transferred to this temporal scale-space concept for the sine wave model, which demonstrates the need for using a scale-invariant temporal scale-space concept when formulating mechanisms for temporal scale selection. The scale selection properties of such a scale-invariant time-causal temporal concepts will be analysed in Sect. 5, and showing that it is possible to obtain temporal scale estimates for a dense sine wave that are truly proportional to the wavelength of the signal, i.e. a characteristic estimate of the temporal duration of the temporal structures in the signal.

Concerning this theoretical analysis, it should be noted that we have here for the expressions (82), (83) and (84) disregarded the rounding of the continuous value \(\hat{K}\) in (81) to the nearest integer upwards or downwards where it assumes its maximum value over temporal scales. Thereby, the graphs in Fig. 2 may appear somewhat different if such quantization effects because of discrete temporal scale levels are also included. The lack of true temporal scale invariance will, however, still prevail.

Concerning the motivation to the theoretical analysis in this section, while the purpose of this analysis has been to investigate how the temporal scale estimates depend on the frequency or the wavelength of the signal, it should be emphasized that the primary purpose has not been to develop a method for only estimating the frequency or the wavelength of a sine wave. Instead, the primary purpose has been to carry out a closed-form theoretical analysis of the properties of temporal scale selection when applied to a model signal for which such closed-form theoretical analysis can be carried out. Compared to using a Fourier transform for estimating the local frequency content in a signal, it should be noted that the computation of a Fourier transform requires a complementary parameter—a window scale over which the Fourier transform is to be computed. The frequency estimate will then be an average of the frequency content over the entire interval as defined by the window scale parameter. Using the proposed temporal scale selection methodology it is on the other hand possible to estimate the temporal scale without using any complementary window scale parameter. Additionally, the temporal scale estimate will be instantaneous and not an average over multiple cycles of a periodic signal, see also the later experimental results that will be presented in Sect. 7 in particular Fig. 7.

In this section, we will analyse the scale selection properties for the time-causal scale-space concept based on convolution with the scale-invariant time-causal limit kernel.

The analysis starts with a detailed study of a sine wave, for which closed-form theoretical analysis is possible and showing that the selected temporal scale level \(\hat{\sigma }\) measured in units of dimension time \([\text{ time }]\) according to \(\hat{\sigma } = \sqrt{\hat{\tau }}\) will be proportional to the wavelength of the signal, in accordance with true scale invariance. We also show that despite the discrete nature of the temporal scale levels in this temporal scale-space concept, local extrema over scale will nevertheless be preserved under scaling transformations of the form \(\lambda _1 = c^j \lambda _0\), with c denoting the distribution parameter of the time-causal limit kernel.

Then, we present a general result about temporal scale invariance that holds for temporal derivatives of any order and for any input signal, showing that under a temporal scaling transformation of the form \(t' = c^j t\), local extrema over scales are preserved under such temporal scaling transformations with the temporal scale estimates transforming according to \(\tau ' = c^j \tau \). We also show that if the scale normalization power \(\gamma = 1\) corresponding to \(p = 1\), the scale-normalized magnitude responses will be preserved in accordance with true temporal scale invariance.

5.1 Time-Causal Temporal Scale Space Based on the Scale-Invariant Time-Causal Limit Kernel

Given the temporal scale-space model based on truncated exponential kernels (36) coupled in cascade

having a composed Fourier transform of the form

and as arises from the assumptions of (i) linearity, (ii) temporal shift invariance, (iii) temporal causality and (iv) non-creation of new local extrema or equivalently zero-crossings with increasing scale, it is more natural to distribute the temporal scale levels logarithmically over temporal scales

so that the distribution in terms of effective temporal scale \(\tau _\mathrm{eff} = \log \tau \) [55] becomes uniform. This implies that time constants of the individual first-order integrators should for some \(c > 1\) be given by [73, 75]

Specifically, if one lets the number of temporal scale levels tend to infinity with the density of temporal scale levels becoming infinitely dense towards \(\tau \rightarrow 0\), it can be shown that this leads to a scale-invariant time-causal limit kernel having a Fourier transform of the form [75, Section 5]

Scale-space signatures showing the variation over scales of the amplitude of scale-normalized temporal derivatives in the scale-space representation of sine waves with angular frequencies \(\omega _0 = 1/4\) (green curves), \(\omega _0 = 1\) (blue curves) and \(\omega _0 = 4\) (brown curves) under convolution with the time-causal limit kernel approximated by the slowest \(K = 32\) temporal smoothing steps and for \(\gamma = 1\) corresponding to \(p = 1\). (Left column) First-order temporal derivatives \(n = 1\). (Right column) Second-order temporal derivatives \(n = 2\). (Top row) Distribution parameter \(c = \sqrt{2}\). (Bottom row) Distribution parameter \(c = 2\). Note how these graphs reflect temporal scale invariance in the sense that (i) a variation in the angular frequency of the underlying signal corresponds to a mere shift of the scale-space signature to either finer or coarser temporal scales and (ii) the maximum magnitude response is independent of the frequency of the underlying signal (horizontal axis Temporal scale \(\tau \) on a logarithmic axis)

Full size image

Corresponding results as in Fig. 3 above but with (left column) \(\gamma = 1/2\) for first-order derivatives and (right column) \(\gamma = 3/4\) for second-order derivatives. Note that the use of scale normalization powers \(\gamma < 1\) implies that the maxima over temporal scales are moved to finer temporal scales and that the (uncompensated) maximum magnitude responses are no longer scale invariant

Full size image

Scale estimates \(\hat{\sigma } = \sqrt{\hat{\tau }}\) as function of the wavelength \(\lambda _0\) for sine waves of different angular frequencies \(\omega _0 = 2 \pi /\lambda _0\) and different orders n of temporal differentiation and different values of \(\gamma \): (blue curves) \(n = 1\) and \(\gamma = 1/2\), (brown curves) \(n = 2\) and \(\gamma = 3/4\), (green curves) \(n = 1\) and \(\gamma = 1\), (red curves) \(n = 2\) and \(\gamma = 1\). Note how the scale estimates expressed in dimensions of time \(\sigma = \sqrt{\tau }\) are directly proportional to the wavelength \(\lambda _0\) of the sine wave and in agreement with temporal scale invariance (horizontal axis wavelength \(\lambda _0\)) (for \(c = \sqrt{2}\) the time-causal limit kernel has been approximated by the slowest \(K = 32\) temporal smoothing stages and for \(c = 2\) by the slowest \(K = 12\) temporal smoothing stages)

Full size image

5.2 Temporal Sine Wave

Consider an input signal defined as a sine wave

and taken as an idealized model of a oscillating signal with temporal structures having characteristic temporal duration \(\lambda _0 = 2 \pi /\omega _0\).

For the time-causal temporal scale-space defined by convolution with the time-causal semi-group \({\varPsi }(t;\; \tau , c)\), the temporal scale-space representation \(L(t;\; \tau , c)\) is given by

where the magnitude \(|\hat{{\varPsi }}(\omega ;\; \tau , c) |\) and the argument \(\arg \hat{{\varPsi }}(\omega ;\; \tau , c) \) of the Fourier transform \(\hat{{\varPsi }}(\omega ;\; \tau , c)\) of the time-causal limit kernel are given by

Thus, the magnitude on the nth-order temporal derivative is given by

and for the nth-order scale-normalized derivate based on variance-based scale normalization the amplitude as function of scale is

Figure 3 shows graphs of the variation of this entity as function of temporal scale for different angular frequencies \(\omega _0\), orders of temporal differentiation n and the distribution parameter c. As can be seen from the graphs, the maxima over scales are assumed at coarser scales with increasing wavelength of the sine wave. The maxima over temporal scale are also assumed at coarser scales for second-order derivatives than for first-order derivatives.

Specifically, when \(\gamma = 1\) the magnitude values at the local extrema over scale are constant over scale, which implies a transfer of the scale selection property (10) to this temporal scale-space model. Notably, this situation is in clear contrast to the situation for the temporal scale-space generated by first-order integrators with equal time constants coupled in cascade. In Fig. 2 it was shown that because of the lack of true temporal scale invariance of that temoral scale-space concept, the maximum magnitude values are not constant over scales for \(\gamma = 1\) as they should be according to the scale-invariant scale selection property (10).

When choosing lower values of \(\gamma \) as motivated from the determination of the parameter \(\gamma \) for scale selection in a Gaussian scale space to make the scale estimate reflect the width of a Gaussian peak for second-order derivatives or reflect the width of a diffuse ramp for first-order derivatives, which leads to \(\gamma = 1/2\) for first-order derivatives (27) and \(\gamma = 3/4\) for second-order derivatives (16), the local extrema over scale are moved to finer scales (see Fig. 4). Then, however, the maximum magnitude values are no longer the same for sine waves of different frequencies, implying that a complementary magnitude normalization step is necessary (see Sect. 5.3 for additional details).

Local extrema over temporal scale Taking the logarithm of the expression (96) gives

If we treat \(\tau \) as a continuous variable and differentiate with respect to \(\tau \) we obtain

Figure 5 shows graphs of how the scale estimate \(\hat{\sigma }\) obtained by setting the derivative with respect to temporal scale to zero increases linearly with the wavelength \(\lambda _0\) of the signal, with different slopes of the linear curve depending on the order of temporal differentiation and the value of the scale normalization parameter \(\gamma \). This overall linear scaling behaviour can directly be proved by rewriting the expression \(\partial _{\tau } \left( \log L_{\zeta ^n,\mathrm{ampl}} \right) = 0\) in (98) into

Since this expression is a direct function of the dimensionless entity \(\tau \omega _0^2\), it follows that the temporal scale estimates will be of the form

for some function \(\varphi (n\gamma , c)\), and thus obeying temporal scale invariance in the sense that the scale estimate in dimension length is proportional to the wavelength of the signal

Notice how this situation is in contrast to the results of scale selection in the temporal scale-space concept obtained by coupling first-order integrators with equal time constants in cascade, where the scale estimate for a sine wave is not directly proportional to the wavelength of the temporal signal, but also affected by a wavelength-dependent temporal scale bias (see Eq. (82) and Fig. 2 in Sect. 4.4).

Table 3 shows numerical values of the differences between the results for different orders of differentiation n and different values of \(\gamma \). These numerical entities become particularly illuminating by forming the ratio \(\hat{\sigma }/\sqrt{n\gamma }\) as shown in Table 4. For a non-causal Gaussian scale space, this ratio should be equal to one for all combinations of n and \(\gamma \) [see Eq. (33)]. For this non-causal temporal scale space, we can, however, note that the deviation from one increases both with larger values of the distribution parameter^{Footnote 2} c and with increasing order of temporal differentiation n, which both lead to larger degrees of temporal asymmetry due to the non-causal temporal dimension.

In the essential proportionality of the scale estimate \(\hat{\sigma }\) to the wavelength \(\lambda _0\) of the signal according to (101), the main component of the scale selection property (34) is thereby transferred to this temporal scale-space concept, although the proportionality constant has to be modified depending on the value of the temporal scale distribution parameter c, the order of temporal differentiation n and the scale normalization parameter \(\gamma \).

Preservation of local extrema over temporal scale under temporal scaling transformations Let us assume that the continuous magnitude function (97) assumes a maximum over temporal scales for some pair \((\omega _0, \tau _0)\) and that the derivative with respect to temporal scale is thereby zero

Let us next assume that we feed in a different sine wave with wavelength \(\lambda _1 = c^j \lambda _0\) for some integer j (for the same value of c as used in the definition of the time-causal limit kernel) and corresponding to \(\omega _1 = c^{-j} \omega _0\) with its matching scale \(\tau _1 = c^{2j} \tau _0\). Then, it holds that

This result implies that the sign of the derivate with respect to temporal scale is preserved between matching angular frequencies and scales \((\omega _0, \tau _0)\) and \((\omega _1, \tau _1)\). Specifically, local extrema over temporal scales are preserved under uniform scaling transformations of the temporal domain \(t' = c^j t\), implying scale covariance of the temporal scales that are selected from local extrema over scales of scale-normalized temporal derivatives.

Note also that although the analysis in Eq. (103) is performed based on a temporary extension of \(\tau \) into a continuous variable, the scale covariance still holds when the continuous function is sampled into a discrete set of temporal scale levels.^{Footnote 3}

5.3 General Scale Invariance Property Under Temporal Scaling Transformations

In Lindeberg [75, Appendix 3] it is shown that for two temporal signals f and \(f'\) that are related by a temporal scaling transform \(f'(t') = f(t)\) for \(t' = c^{j'-j} t\) with the corresponding transformation between corresponding temporal scale levels \(\tau ' = c^{2(j'-j)} \tau \), the scale-normalized temporal derivatives defined by either \(L_p\)-normalization or variance-based normalization in the scale-space representation defined by convolution with the time-causal limit kernel \({\varPsi }(t;\; \tau , c)\) are for any temporal input signal f related according to

This result specifically implies that the scale-space signatures, which are the graphs that show the variation in the strength of scale-normalized derivatives over scale, will be rescaled copies of each other for signals that are related by a uniform scaling transformation of the temporal domain.

Specifically, local temporal scale estimates \(\hat{\tau }\) and \(\hat{\tau }'\) as determined from local extrema over temporal scales in the two temporal domains will be assumed at corresponding temporal scale levels and will thus be transformed in a scale-covariant way for any temporal scaling transformation of the form \(t' = c^{j'-j} t\). In units of the temporal variance, it holds that

and in units of the temporal standard deviation

If \(\gamma = 1\) corresponding to \(p = 1\), the magnitude values at corresponding temporal scale levels will be equal. If \(\gamma \ne 1\) corresponding to \(p \ne 1\), the magnitude values will be related according to (104). Thereby, this expression provides a way to normalize maximum strength measures between local extrema over scales assumed at different temporal scales as obtained e.g. in the scale-space signatures shown in Fig. 4.

Note that by this construction we have been able to transfer the temporal scale invariance property (9) and (10) that holds for a non-causal Gaussian temporal scale-space concept to also hold for a time-causal temporal scale-space concept, which is a novel type of theoretical construction. This property is, however, restricted to the temporal scale-space concept based on convolution with the time-causal limit kernel and does, for example, not hold for the time-causal temporal scale-space concept based on convolution with a cascade of truncated exponential kernels having equal time constants and corresponding to a uniform distribution of the temporal scale levels in units of the composed temporal variance.

In the theoretical analysis of scale selection properties of (i) a temporal peak in Sect. 4.2, (ii) a temporal onset ramp in Sect. 4.3 and (ii) a temporal sine wave in Sect. 4.4 and Sect. 5.2, we did first compute closed-form expressions for how the scale-normalized temporal magnitude measures depend upon the temporal scale levels \(\tau _K = K \mu ^2\) according to (62), (69) and (79) for the time-causal scale-space concept based on truncated exponential kernels with equal time constant coupled in cascade or how the scale-normalized magnitude measure depends on the temporal scale level \(\tau _K = c^{2K} \tau _0\) according to (96) for the time-causal temporal scale-space concept based on the scale-invariant limit kernel:

Then, to compute the temporal scale levels at which the scale-normalized derivative responses assumed their maximum values over temporal scales, we temporarily extended these magnitude measures from being defined over discrete integer temporal scale levels K to a continuum over K, to be able to differentiate the closed-form expressions with respect to the temporal scale level.

Graphs of the three types of model signals for which closed-form expressions can be computed for how the scale-normalized derivative-based magnitude responses vary as function of the scale parameter: (top row) Time-causal temporal peak model according to (43) with temporal duration determined by \(K_0 = 5\) for which the second-order scale-normalized temporal derivative response at the peak varies according to (107) here with \(\gamma = 3/4\), (second row) Time-causal onset ramp model according to (66) with temporal duration determined by \(K_0 = 5\) for which the first-order scale-normalized temporal derivative response at the ramp varies according to (108) here with \(\gamma = 1/2\), (third row) Temporal sine wave according to (73) with angular frequency \(\omega _0 = 1\) for which the amplitude of the second-order temporal derivative response varies according to (109) here with \(\gamma = 1\) and (bottom row) Temporal sine wave according to (91) for which the amplitude of the second-order temporal response measure varies according to (110) here with \(\gamma = 1\) . The scale-space signatures in the first three rows have been computed using the time-causal temporal scale-space concept based on truncated exponential kernels with equal time constants \(\mu = 1\) coupled in cascade, whereas the scale-space signature in the bottom row has been computed using the time-causal temporal scale-space concept obtained by convolution with the scale-invariant limit kernel with distribution parameter \(c = 2\). Note specifically that the shape of the scale-space signature in the third row is different from the shape in the fourth row because of the uniform distribution of temporal scale levels K in the third row and the logarithmic distribution in the fourth row (horizontal axis for the figures in the left column: time t) (Horizontal axis for the figures in the right column: temporal scale level K)

Full size image

A general question that could be raised in this context therefore concerns how good approximation the results from the continuous approximation of local extrema over scales are with respect to a setting where the temporal scale levels are required to be discrete. A common property of the four types of scale-space signatures according to equations (107)–(109) and shown in Fig. 6 is that they are unimodal, i.e., they assume a single maximum over temporal scales and do first increase and then decrease. Thereby, when the continuous variable \(K_{\mathrm{c}}\) with its associated maximum over temporal scales \(\hat{K}_{\mathrm{c}}\) obtained from a continuous analysis is in a second stage restricted to be discrete, it follows that the discrete maximum over discrete temporal scales \(\hat{K}_{\mathrm{d}}\) is guaranteed to be assumed at either the nearest lower or the nearest higher integer. Thus, we obtain the discrete temporal scale estimate by rounding the continuous scale estimate \(\hat{K}_{\mathrm{c}}\) to either the nearest lower or the nearest higher integer.

Whether the value should be rounded upwards or downwards depends on how close the continuous estimate \(\hat{K}_{\mathrm{c}}\) is to the nearest downwards versus upwards integers and on the local degree of asymmetry of the scale-space signature around the maximum over temporal scales.

When implementing and executing a temporal scale selection algorithm in practice, the situation can on the other hand be reverse. Given a set of discrete temporal scale levels, we may detect a local maximum over temporal scales at some discrete temporal scale level \(\hat{K}_{\mathrm{d}}\). If we would like to use this temporal scale estimate for estimating the temporal duration of the underlying temporal structure that gave rise to the response, e.g. according to the methodology outlined in “Appendix 3”, we may on the other hand would like to compute a better continuous estimate \(\hat{K}_{\mathrm{c}}\) of the temporal scale level than as restricted by the discrete temporal scale levels.

A straightforward way of computing a more accurate temporal scale estimate in such a situation is by interpolating a parabola over the measurements over the temporal scale levels in an analogous way as subresolution spatial scale estimates can be obtained over a spatial scale-space representation [62, 76, 85]. Let \((x_0, y_0)\) denote the scale level and the magnitude measure at the discrete maximum and let \((x_0-1, y_{-1})\) and \((x_0+1, y_{1})\) denote the corresponding scale level and magnitude measure at the nearest lower and upper temporal scales, respectively. Assuming the following form of the interpolating function

the interpolation coefficients become

with the corresponding subresolution estimate of the maximum over scales

Note that the correction offset \(\Delta x = -b/a\) is restricted to the interval \(\Delta x \in [-1/2, 1/2]\) implying that the location \(x_0\) of the discrete maximum is guaranteed to be on the sampling grid point \(x_i\) nearest to the subresolution estimate \(\hat{x}\).

To illustrate the derived scale selection properties, we will in this section show the result of applying temporal scale selection to different types of purely temporal signals.

Temporal scale selection by scale-space extrema detection applied to a synthetic sine wave signal \(f(t) = \sin (\exp ((b-t)/a))\) for \(a = 200\) and \(b = 1000\) with temporally varying frequency so that the wavelength increases with time t. (Top left) Temporal scale-space maxima of the scale-normalized second-order temporal derivative \(-L_{\zeta \zeta }\) detected using the time-causal temporal scale-space concept corresponding to convolution with the time-causal limit kernel for \(c = \sqrt{2}\) and with each scale-space maximum marked at the point \((\hat{t}, \hat{\sigma })\) with \(\hat{\sigma } = \sqrt{\hat{\tau }}\) at which the scale-space maximum is assumed. (Top right) Temporal scale-space maxima detected using the non-causal Gaussian temporal scale-space concept using five temporal scale levels per scale octave. For both temporal scale-space concepts, the scale-normalized temporal derivatives have been defined using \(l_p\)-normalization for \(p = 2/3\) and corresponding to \(\gamma = 3/4\) for second-order temporal derivatives (horizontal axis time t) (Vertical axis in top row Temporal scale estimates in units of \(\sigma = \sqrt{\tau }\)) (vertical axis in bottom row Signal strength f(t))

Full size image

The bottom rows in Fig. 7 shows a one-dimensional model signal having a temporally varying frequency of the form

defined such that the local wavelength increases with time t. Figure 8 shows the temporal scale-space representation of the scale-normalized second-order temporal \(-L_{\zeta \zeta }\) as function of scale for the maximally scale-invariant choice of \(\gamma = 1\) corresponding to \(p = 1\). Note how structures in the signal of longer temporal duration give rise responses at coarser temporal scales in agreement with the derived theoretical properties of the scale-covariant scale-space concepts over a time-causal vs. a non-causal temporal domain.

In the top rows in Fig. 7 we show the results of applying local temporal scale selection by detecting local maxima over both time and scale of the scale-normalized second-order temporal derivative \(-L_{\zeta \zeta }\) for the scale-calibrated choice of \(\gamma = 3/4\) corresponding to \(p = 2/3\). Each detected scale-space extremum has been marked by a star at the point in scale-space \((\hat{t}, \hat{\sigma }) = (\hat{t}, \sqrt{\hat{\tau }})\) at which the local maximum over temporal scale was assumed. Such temporal scale selection has been performed using two types of temporal scale-space concepts: (i) based on the time-causal scale-space representation corresponding to convolution with the scale-invariant limit kernel with distribution parameter \(c = \sqrt{2}\) approximated by a finite number of the at least \(K = 8\) slowest primitive smoothing steps at the finest level of scale or (ii) based on the non-causal Gaussian kernel using 5 temporal scale levels per scale octave.

The discrete implementation of the time-causal temporal scale-space representation corresponding to convolution with the scale-invariant limit kernel has been based on recursive filters over time according to the methodology described in Lindeberg [75, section 6.2] whereas the discrete implementation of the non-causal Gaussian temporal scale-space concept has been based on the discrete analogue of the Gaussian kernel described in Lindeberg [53, 54]. Discrete implementation of scale-normalized temporal derivatives has in turn been based on discrete \(l_p\)-normalization according to the methodology outlined in [75, section 7].

As can be seen from the results in the top rows in Fig. 7, the temporal scale estimates \(\hat{\sigma }\) increase proportional to the local wavelength \(\lambda \) according to the derived scale-invariant properties of the temporal scale-space concepts based on convolution with the time-causal limit kernel or the non-causal Gaussian kernel. The scale estimates obtained using the time-causal versus the non-causal temporal scale-space concepts are, however, not equal in units of the standard deviation \(\hat{\sigma }\) of the temporal scale-space kernel—see also the theoretical analysis in Sect. 5 with specifically the numerical comparisons in Table 4. Thus, different scale calibration factors are needed to transform the temporal scale estimates in units of the temporal standard deviation to units of the temporal duration of the signal for the time-causal versus the non-causal Gaussian temporal scale-space concepts—compare with the theoretical analysis in “Appendix 3”.

Temporal scale-space representation of the scale-normalized second-order temporal derivative \(-L_{\zeta \zeta }\) computed from a synthetic sine wave signal \(f(t) = \sin (\exp ((b-t)/a))\) for \(a = 200\) and \(b = 1000\) with temporally varying frequency so that the wavelength increases with time t. (Left) Using the time-causal temporal scale-space concept corresponding to convolution with the time-causal limit kernel for \(c = \sqrt{2}\). (Right) Using the non-causal Gaussian temporal scale-space concept. For both temporal scale-space concepts, the scale-normalized temporal derivatives have been defined using \(l_p\)-normalization for \(p = 1\) and corresponding to \(\gamma = 1\). Note how the notion of scale-normalized temporal derivatives implies that stronger responses are obtained at temporal scale levels proportional to the duration of the underlying structures in the temporal signal (horizontal axis time t) (Vertical axis effective temporal scale \(\log \tau \))

Full size image

Another qualitative difference that can be noted with regard to temporal scale selection in a time-causal temporal scale-space representation versus a non-causal Gaussian temporal scale-space representation is that any measurement performed in a time-causal temporal scale-space concept is associated with an inherent temporal delay \(\delta \), whereas the temporal delay can be defined to be zero for a non-causal Gaussian temporal scale-space. Thus, the local extrema over scales will be assumed later with increasing temporal scale levels as induced by temporal structures of having longer temporal duration. Varying the parameter q in the scale calibration criteria (14) and (25) to values of \(q < 1\) provides a straightforward way of enforcing responses to be obtained at finer temporal scales and thereby implying shorter temporal delays, at the potential cost of a larger likelihood of false positive responses by not detecting the underlying temporal structures at the same temporal scales as they occur.

Yet another side effect of the longer temporal delays at coarser temporal scales is that multiple local responses over scales may be obtained with respect to the same underlying temporal structure, with first responses obtained at finer temporal scales followed by later responses at coarser temporal scales. Because of the temporal shift caused by different temporal delays between adjacent temporal scale levels, scale-space extrema detection over a local \(3 \times 3\) neighbourhood may not detect a single extremum over temporal scales as for the non-causal Gaussian temporal scale-space concept. Therefore, explicit handling of different temporal delays at different temporal scale levels is needed when performing temporal scale selection in a time-causal temporal scale-space representation. The presented theory of temporal scale selection properties is intended to be generally applicable with respect to different such strategies for handling the temporal delays in specific algorithms.

The result of post-filtering the scale-space extrema shown in Fig. 7 (left) to result in a single extremum over temporal scales for each underlying temporal structure

Full size image

7.1 Post-Filtering of Responses at Adjacent Temporal Scales

In view of the behaviour of image structures over temporal scales illustrated in Fig. 8, one way of suppressing multiple responses to the same underlying structure at different temporal scales is by performing an additional search around each scale-space extremum as follows: if a point \((\hat{t}, \hat{\tau })\) at temporal scale level with temporal scale index k is a scale-space maximum, perform an additional search at the nearest finer temporal scale level \(k-1\) to previous temporal moments \(t-j\) as long as the scale-normalized values monotonically increase. When the monotone increase stops and a local temporal maximum has been found, then compare if the temporal maximum value at the nearest finer temporal scale is greater than the temporal maximum value at the current scale. If so, suppress the scale-space maximum at the current scale. In a corresponding manner, perform a search at the next coarser temporal scale \(k+1\) to the following temporal moments \(t+j\) as long as the scale-normalized values monotonically increase. When the monotone increase stops and a local temporal maximum has been found, then compare if the temporal maximum value at the next coarser temporal scale is greater than the temporal maximum value at the current scale. If so, suppress the scale-space maximum at the current scale. By this type of straightforward scale-space tracking over adjacent temporal scales, a single response will be obtained to the same underlying structure as illustrated in Fig. 9. Additionally, a more accurate temporal scale estimate can be computed by performing the parabolic interpolation according to (111) and (115) over the nearest backward and forward temporal maxima at the adjacent finer and coarser temporal scales as opposed to an interpolation over temporal scales at the same temporal moment as at which the temporal scale-space maximum was assumed.

Illustration of the temporal dynamics over multiple temporal scales that arises when a temporal scale-space representation responds to a temporal peak with temporal standard deviation \(\sigma _0 = 32\) for (left column) the time-causal representation corresponding to convolution with the time-causal limit kernel and (right column) the non-causal Gaussian temporal scale space. (Top row) Graphs of the input kernels for the cases of a time-causal peak and a non-causal peak, respectively. (Bottom row) Scale-space maps of the scale-normalized second-order temporal derivative \(-L_{\zeta \zeta }\). Note that if we slice the time-causal scale-space map vertically at a temporal moment before the full development of the temporal scale-space maximum at the temporal scale corresponding to the temporal scale of the peak, we will get earlier temporal responses at finer temporal scales, whereas if we slice the time-causal scale-space map at a temporal moment after the full development of the temporal scale-space maximum, we will get later temporal responses at coarser temporal scales. When handling multiple temporal scale levels in a time-causal real-time situation, it is therefore natural to include explicit mechanisms for tracking and handling how the temporal structures evolve over temporal scales (horizontal axis time t) (vertical axis in bottom row effective temporal scale \(\log \tau \))

Full size image

The result of performing temporal delay compensation of the scale-space extrema shown in Fig. 9 by correcting the temporal moment \(\hat{t}\) of the scale-space extremum by an estimate of the temporal delay computed from the position of the temporal maximum of the temporal scale-space kernel according to [75, Section 4]. Note that although the computations by necessity have to be associated with a temporal delay, we can nevertheless compute a good estimate of when the underlying event occurred that gave rise to the feature response

Full size image

Note that in a real-time situation, the necessary information needed to perform a search to the past can be stored by a process that records the value of temporal maxima at each temporal scale. In a corresponding manner, later deletion or subresolution interpolation of the scale estimate of a scale-space maximum can only be performed when time has passed to the location of the next temporal maximum at the nearest coarser temporal scale. If a preliminary feature response has been obtained at any temporal scale while a potential response at the nearest coarser temporal scale is not available yet because of its longer temporal scale, a real-time system operating over multiple scales should preferably be designed with the ability to correct or adjust preliminary measurements when more information at coarser scales becomes available—see also Fig. 10 for an illustration of the underlying temporal dynamics that arises when processing signals at multiple scales using a time-causal temporal scale-space representation.

While these descriptions have been given regarding scale-space maxima and local maxima, the procedure for handling scale-space minima and local minima is analogous with the polarity of the signal reversed.

7.2 Temporal Delay Compensation

In Fig. 11 we have additionally adjusted the temporal location of every temporal scale-space maximum by an estimate of the temporal delay \(\delta \) computed from the location of the temporal maximum of the underlying temporal scale-space kernel according to [75, Section 4]. Note that although any temporal event detected at a coarser temporal scale will by necessity be associated with nonzero temporal delay, we can nevertheless retrospectively compute a good estimate of when the underlying event occurred that gave rise to the registered feature response.

Temporal scale selection by scale-space extrema detection applied to a measured temperature signal (showing hourly temperature measurements at Tullinge outside Stockholm during the years 1997–2002, open data courtesy of the Swedish Meteorological and Hydrological Institute, SMHI, in Sweden). (Top left) Temporal scale-space maxima of the scale-normalized second-order temporal derivative \(-L_{\zeta \zeta }\) detected using the time-causal temporal scale-space concept corresponding to convolution with the time-causal limit kernel for \(c = \sqrt{2}\) and with each scale-space maximum marked at the delay-compensated point \((\hat{t}-\delta , \hat{\sigma })\) with \(\hat{\sigma } = \sqrt{\hat{\tau }}\) at which the scale-space maximum is assumed with the temporal delay estimate \(\delta \) determined from the temporal location of the peak of the corresponding temporal scale-space kernel at the given scale. (Top right) Temporal scale-space maxima detected using the non-causal Gaussian temporal scale-space concept using five temporal scale levels per scale octave. For both temporal scale-space concepts, the scale-normalized temporal derivatives have been defined using discrete \(l_p\)-normalization for \(p = 2/3\) corresponding to \(\gamma = 3/4\) for second-order temporal derivatives. Note how the temporal scale selection method is able to extract the coarse scale temporal phenomena although the signal contains very strong variations at finer temporal scales (horizontal axis time t in units of days) (Vertical axis t in upper figure temporal scale estimates in units of days)

Full size image

Temporal scale-space representation of the scale-normalized quasi quadrature measure \(\mathcal{Q}_t L = (L_{\zeta }^2 + C L_{\zeta \zeta }^2)/t^{{\varGamma }}\) for \(C = 1/\sqrt{(1 - {\varGamma }))(2 - {\varGamma })}\), \(\gamma = 1\) and \({\varGamma }= 0\) computed from the temperature signal in Fig. 12. (Left) Using the time-causal temporal scale-space concept corresponding to convolution with the time-causal limit kernel for \(c = \sqrt{2}\). (Right) Using the non-causal Gaussian temporal scale-space concept. For both temporal scale-space concepts, the scale-normalized temporal derivatives have been defined using discrete \(l_p\)-normalization. Note how this operator responds to different types of temporal structures at different temporal scales with a particularly strong response due to temporal variations caused by the annual temperature cycle (horizontal axis time t) (Vertical axis effective temporal scale \(\log \tau \))

Full size image

7.3 Temporal Scale Selection for a Real Measurement Signal

While a main purpose of this article is to develop a theory of temporal scale selection to be used in conjunction with a spatio-temporal scale-space concept for video analysis or a spectro-temporal scale-space concept for audio analysis, we argue that this theory is applicable to much larger classes of time-dependent measurement signals. For the purpose of isolating the effect to a purely one-dimensional measurement signal, we do in Fig. 12 show the result of applying corresponding temporal scale selection to a real measurement signal showing hourly measurements of the temperature at a weather station. Note how the temporal scale selection method based on scale-space extrema is able to extract the coarse scale temperature peaks although the signal contains substantial high-amplitude variations at finer scales.

For the purpose of having the local feature responses being less dependent on the local phase of the signal than for either first- or second-order temporal derivatives \(L_{\zeta }\) or \(L_{\zeta \zeta }\), we do in Fig. 13 show the result of computing a quasi quadrature measure

derived in [74] to for

constitute an improved version of an earlier proposed quasi quadrature measure in Lindeberg [60].

Graphs that for each temporal scale level \(\sigma _k\) show the sum of the quasi quadrature responses in Fig. 13 over time t. With a complementary stationarity assumption, the peaks over temporal scales reflect the temporal scale levels \(\sigma _k\) at which there is the largest scale-normalized variability in terms of first- and second-order temporal derivative responses. For the time-causal model to the left, the local maxima over temporal scale are assumed at temporal scale levels \(\hat{\sigma }_1 = 0.207\) days and \(\hat{\sigma }_2 = 71.5\) days, respectively. For the non-causal model to the right, the local maxima over temporal scales are assumed at temporal scale levels \(\hat{\sigma }_1 = 0.236\) days and \(\hat{\sigma }_2 = 85.2\) days. The ratios between these temporal scale levels are 332 for the time-causal model and 345 for the non-causal model in good qualitative agreement with the finer scale peaks corresponding to the daily cycle and the coarser scale peaks corresponding to the yearly cycle of 365 days (horizontal axis temporal scale index k corresponding to a uniform sampling in terms of effective temporal scale \(\log \tau \))

Full size image

Note how this operator responds to different types of temporal structures at different temporal scales with a particularly strong response due to the temporal variations caused by the daily and annual temperature cycles. This effect becomes more immediately noticeable if we complement the above temporal scale selection method with a complementary assumption about stationarity of the signal and sum up the scale-normalized feature responses of the quasi quadrature measure over time for every temporal scale level. In the graphs shown in Fig. 14 we do then obtain two major peaks over temporal scales, with the finer scale peak corresponding to the daily temperature cycle and the coarser scale peak to the annual temperature cycle.

Note that the purpose of this experiment is not primarily to develop an algorithm for detecting periodic variations in a signal but to illustrate that the temporal scale selection mechanism produces intuitively reasonable temporal scale estimates for a real-world 1-D signal with known properties.

7.4 Numerical Accuracy of the Temporal Scale Estimates

To investigate how well the temporal scale estimates generated by the resulting temporal scale selection mechanism reflect the temporal scale in the underlying temporal signal, we generated temporal model signals defined as either (i) a temporal peak modelled as time-causal limit kernel with temporal scale \(\tau _0\) and distribution parameter c for different values of \(\tau _0\) and c and (ii) a temporal onset ramp modelled as the primitive function of the temporal peak model. Then, we detected scale-space extrema of the second-order scale-normalized temporal derivative \(-L_{\zeta \zeta }\) for peak detection or scale-space extrema of the first-order scale-normalized temporal derivative \(L_{\zeta }\) for onset detection, using the time-causal temporal scale-space concept corresponding to convolution with the time-causal limit kernel for the same value of c and resulting in temporal scale estimates \(\hat{\sigma } = \sqrt{\hat{\tau }}\).

Tables 5 and 6 show results from this experiment for the time-causal temporal peak and the temporal onset ramp models, respectively. Note how well the theoretical temporal scale selection results derived for truncated exponential kernels with a uniform distribution of the temporal scale levels in Sects. 4.2 and 4.3 do also generalize to truncated exponential kernels with a logarithmic distribution of the temporal scale levels. Table 7 shows corresponding results for the non-causal Gaussian scale-space concept applied to non-causal temporal peak and temporal onset ramp models, respectively, and again with a very good agreement between the local scale estimates \(\hat{\sigma }\) in relation to the inherent temporal scale \(\sigma _0\) in the signal.

For this experiment, we can specifically note that variance-based temporal scale normalization does on average lead to slightly more accurate temporal scale estimates compared to discrete \(l_p\)-normalization, which can be contrasted to previous results regarding spatial scale selection in hybrid pyramids by Lindeberg and Bretzner [76], where \(l_p\)-normalization lead to much more accurate scale selection results compared to variance-based normalization.

In this section, we will develop a basic proof of concept of applying the proposed theory and methodology for selecting local temporal scales in video data based on a small set of specific spatio-temporal feature detectors formulated in terms of spatio-temporal differential invariants. A more detailed treatment of this topic with examples for more general families of differential expressions for spatio-temporal scale selection with associated spatio-temporal image models for scale calibration is presented in a companion paper [74].

8.1 Spatio-Temporal Receptive Field Model

For applying the proposed framework for temporal scale selection to spatio-temporal video data, we follow the approach with idealized models of spatio-temporal receptive fields of the form

as previously derived, proposed and studied in Lindeberg [66, 68, 75] where

• \(x = (x_1, x_2)^T\) denotes the image coordinates,

• t denotes time,

• s denotes the spatial scale,

• \(\tau \) denotes the temporal scale,

• \(v = (v_1, v_2)^T\) denotes a local image velocity,

• \({\varSigma }\) denotes a spatial covariance matrix determining the spatial shape of an affine Gaussian kernel \(g(x;\; s, {\varSigma }) = \frac{1}{2 \pi s \sqrt{\det {\varSigma }}} \hbox {e}^{-x^T {\varSigma }^{-1} x/2s}\),

• \(g(x_1 - v_1 t, x_2 - v_2 t;\; s, {\varSigma })\) denotes a spatial affine Gaussian kernel that moves with image velocity \(v = (v_1, v_2)\) in space–time and

• \(h(t;\; \tau )\) is a temporal smoothing kernel over time.

and we specifically here choose as temporal smoothing kernel over time either (i) the time-causal temporal scale-space kernel corresponding to a set of first-order integrators with equal time constants coupled in cascade (37)

with \(\tau = K \mu ^2\) or (ii) the time-causal limit kernel

defined via its Fourier transform of the form (90).

For simplicity, we shall in this treatment restrict ourselves to space–time separable receptive fields obtained by setting the image velocity to zero \(v = (v_1, v_2) = (0, 0)\) and to receptive fields that are rotationally symmetric over the spatial domain as obtained by setting the spatial covariance matrix to a unit matrix \({\varSigma }= I\). The resulting spatio-temporal receptive fields that we then obtain correspond to complementing our time-causal temporal scale-space concepts studied in Sects. 4, 5 with a rotationally symmetric spatial Gaussian scale-space concept over the spatial domain.

Specifically, the natural way of expressing spatio-temporal scale selection mechanisms within this space–time separable spatio-temporal scale-space concept

is by studying scale-normalized partial derivates of the form [75, Section 8.5, Equation (108)]

where the factor \(s^{(m_1 + m_2) \gamma _s/2}\) transforms the regular partial spatial derivatives to corresponding scale-normalized spatial derivatives with \(\gamma _s\) denoting the spatial scale normalization parameter [62] and the factor \(\alpha _n(\tau )\) is the scale normalization factor for scale-normalized temporal derivatives according to either variance-based normalization [75, Section 7.2, Equation (74)]

with \(\gamma _{\tau }\) denoting the temporal scale normalization parameter for scale-normalized temporal derivatives according to \(L_p\)-normalization [75, Section 7.2, Equation (76)]

and \(G_{n,\gamma _{\tau }}\) denotes the \(L_p\)-norm of the non-causal temporal Gaussian derivative kernel for the \(\gamma _{\tau }\)-value for which this \(L_{\!p}\)-norm becomes constant over temporal scales.

8.2 Differential Entities for Spatio-Temporal Scale Selection

Inspired by the way neurons in the lateral geniculate nucleus (LGN) respond to visual input [14, 15], which for many LGN cells can be modelled by idealized operations of the form [68, Equation (108)]

let us consider the following differential entities [75, Section 8.4, Equations (95)–(96)]

which correspond to first- and second-order temporal derivatives of the spatial Laplacian operator and study the corresponding scale-normalized spatio-temporal derivative expressions for \(\gamma _s = 1\):

Notably, we do not focus on extending the previously established use of the spatial Laplacian operator for spatial scale selection to a spatio-temporal Laplacian operator for spatio-temporal scale selection, since the most straightforward way of defining such an operator \(\nabla _{(x, y, t)}^2 L = L_{xx} + L_{yy} + \kappa ^2 L_{tt}\) for some \(\kappa \) is not covariant under independent rescaling of the spatial and temporal coordinates as occurs if observing the same scene with cameras having independently different spatial and temporal sampling rates. The differential entities \(\partial _{t,\mathrm{norm}} (\nabla _{(x,y),\mathrm{norm}}^2 L)\) and \(\partial _{tt,\mathrm{norm}} (\nabla _{(x,y),\mathrm{norm}}^2 L)\) are on the other hand truly covariant under independent rescalings of the spatial and temporal dimensions and therefore better candidates to be used as primitives in spatio-temporal scale selection algorithms.

8.3 Spatio-Temporal Scale Selection Properties

8.3.1 Response to a Localized Gaussian Blink

Consider a local idealized spatio-temporal image pattern defined as the combination of a rotationally symmetric Gaussian blob \(g(x, y;\; s_0)\) over the spatial domain and a time-causal temporal peak \(U(t;\; \mu , K_0)\) of the form (43) over the temporal domain

If we define the spatio-temporal scale-space representation of this spatio-temporal image pattern of the form (122) with the temporal scale-space kernel chosen as the composed kernel \(U(t;\; \mu , K)\) corresponding to a cascade of first-order integrators with equal time constants coupled in cascade (120), then it follows from the semi-group property of the spatial Gaussian kernel and the semi-group property (45) of the time-causal temporal kernel \(U(t;\; \mu , K)\) that the spatio-temporal scale-space representation will be of the form

The second-order temporal derivative of the spatial Laplacian of the Gaussian Specifically, the scale-normalized differential entity \(\partial _{tt,\mathrm{norm}} (\nabla _{(x,y),\mathrm{norm}}^2 L)\) constituting an idealized model of a “lagged” LGN cell [75, Figure 3(right)] will by a combination of the Laplacian response of Gaussian

the second-order temporal derivative of \(L_{tt}(t;\; \mu , K)\) of a time-causal peak in Eq. (47) and the temporal scale normalization operation

in Eq. (54) assume the form

Specifically, based on previously established scale selection properties of the spatial Laplacian of the Gaussian [62, 70] and the second-order scale-normalized temporal derivatives of the time-causal scale space (Sect. 4.2), it follows that this spatio-temporal differential entity will for spatial and temporal scale normalization powers \(\gamma _s = 1\) and \(\gamma _{\tau } = 3/4\), respectively, assume its local extremum over both spatial and temporal scales at spatial scale

and at a temporal scale that is a good approximation of the temporal scale of the temporal peak \(\hat{K} \approx K_0\) corresponding to (see Table 1)

Thus, simultaneous spatio-temporal scale selection using the differential entity \(\partial _{tt,\mathrm{norm}} (\nabla _{(x,y),\mathrm{norm}}^2 L)\) applied to the model signal (131) will estimate both the spatial extent and the temporal duration of the Gaussian blink.

The determinant of the spatio-temporal Hessian For general values of the spatial and temporal scale normalization parameters \(\gamma _s\) and \(\gamma _{\tau }\), the determinant of the spatio-temporal Hessian is given by

In the specific case when the spatio-temporal scale-space representation of a time-causal Gaussian blink is of the form (132), if we restrict the analysis to the spatial origin \((x, y) = (0, 0)\) where \(g_x = g_y = 0\) and to the temporal maximum point \(t_\mathrm{max}\) where \(h_t = 0\), implying that \(L_{xy} = L_{xt} = L_{yt} = 0\), it follows that the determinant of spatio-temporal Hessian at the spatio-temporal maximum reduces to the form

we can observe that the spatial and temporal scale selection properties of the determinant of the spatio-temporal Hessian will be different from the scale selection properties of the second-order temporal derivative of the spatial Laplacian. In a companion paper [74], it is shown that for a corresponding non-causal Gaussian temporal scale-space concept, spatial and temporal scale normalization parameters equal to \(\gamma _s = 5/4\) and \(\gamma _{\tau } = 5/4\) lead to scale estimates \(\hat{s}\) and \(\hat{\tau }\) corresponding to the spatial and temporal extents \(\hat{s} = s_0\) and \(\hat{\tau } = \tau _0\) of a Gaussian blink.

By performing a corresponding study of the temporal scale selection properties of the purely temporal component of this expression

for the specific case of time-causal temporal scale-space representation based on a uniform distribution of the intermediate temporal scale levels as previously done for the temporal scale selection properties of the second-order temporal derivative of a temporal peak in Table 1, we obtain the results shown in Table 8. The column labelled \(\hat{K}\) shows that the maximum over temporal scales is obtained at a temporal scale level near the temporal scale of the underlying temporal peak

whereas the column labelled \(\theta _{\mathrm{postnorm}}\) shows that if we normalize the input signal \(f(t) = U(t;\; \mu , K_0)\) to having unit contrast, then the corresponding post-normalized differential entity

is approximately constant for temporal peaks with different temporal duration as determined by the parameter \(K_0\). In these respects, this time-causal scale selection method implies a good approximate transfer of the scale selection property of treating similar temporal structures of different temporal duration in a uniform manner.

8.3.2 Response to a Localized Gaussian Onset Blob

Consider a local idealized spatio-temporal image pattern defined as the combination of a rotationally symmetric Gaussian blob \(g(x, y;\; s_0)\) over the spatial domain and a time-causal onset ramp \(\int _{u = 0}^t U(u;\; \mu , K_0) \, du\) of the form (66) over the temporal domain

Again defining the temporal scale-space representation of this spatio-temporal image pattern of the form (122) with the temporal scale-space kernel chosen as the composed kernel \(U(t;\; \mu , K)\) corresponding to a cascade of first-order integrators with equal time constants coupled in cascade (120), it follows from the semi-group property of the spatial Gaussian kernel and the semi-group property (45) of the time-causal temporal kernel \(U(t;\; \mu , K)\) that the spatio-temporal scale-space representation will be of the form

The first-order temporal derivative of the spatial Laplacian of the Gaussian For the scale-normalized differential entity \(\partial _{t,\mathrm{norm}} (\nabla _{(x,y),\mathrm{norm}}^2 L)\) constituting an idealized model of a “non-lagged” LGN cell [75, Figure 3(left)] will by a combination of the Laplacian response of Gaussian \(\nabla ^2 g(x, y;\; s) = (x^2 + y^2 -2 s)/s^2 \, g(x, y;\; s)\), the first-order temporal derivative \(L_t(t;\; \mu , K)\) of a time-causal onset ramp in Eq. (68) and the temporal scale normalization operation \(L_{\zeta }(t;\; \mu , K) = (\mu \sqrt{K})^{\gamma _{\tau }} \, L_{t}(t;\; \mu , K)\) in Eq. (53) assume the form

Specifically, based on previously established scale selection properties of the spatial Laplacian of the Gaussian [62, 70] and the first-order scale-normalized temporal derivatives of the time-causal scale space (Sect. 4.3), it follows that this spatio-temporal differential entity will for spatial and temporal scale normalization powers \(\gamma _s = 1\) and \(\gamma _{\tau } = 1/2\), respectively, assume its local extremum over both spatial and temporal scales at spatial scale

and at a temporal scale that is a good approximation of the temporal scale of the temporal onset ramp \(\hat{K} \approx K_0\) corresponding to (see Table 2)

Thus, simultaneous spatio-temporal scale selection using the differential entity \(\partial _{t,\mathrm{norm}} (\nabla _{(x,y),\mathrm{norm}}^2 L)\) applied to the model signal (143) will estimate both the spatial extent and the temporal duration of a Gaussian onset blob.

8.4 General Scale Selection Property for Temporal Modelling and Time-Causal Scale Space Based on the Scale-Invariant Time-Causal Limit Kernel

If we instead model the Gaussian blink (131) and the onset Gaussian blob (143) by the scale-invariant time-causal limit kernel \({\varPsi }(t;\; \tau , c)\) over the temporal domain

then by the general transformation property of the time-causal limit kernel under temporal scaling transformations by a temporal scaling factor \(S =c^j\) that is an integer power of the distribution parameter c of the time-causal limit kernel [75, Equation (44)]

it holds that the corresponding spatio-temporal scale-space representations of two temporally scaled video sequences \(f'(x', y', t) = f(x, y, t)\) for \((x', y', t') = (x, y, S t)\) are related according to Lindeberg [75, Equation (46)]

for \((s', \tau ') = (s, S^2 \tau )\) if \(S = c^j\). The corresponding scale-normalized temporal derivatives are in turn related according to (104)

If the scale-normalized temporal derivative \(L_{\zeta ^n}(x, y, t;\, s,\tau , c)\) computed from the original video sequence f assumes a local extremum over temporal scales at \((x, y, t;\; s, \tau ) = (x_0, y_0, t_0;\; s_0, \tau _0\)), then by the general scale invariance property of temporal scale selection in the temporal scale-space concept based on the time-causal limit kernel, which is described in Sect. 5.3, it follows that the scale-normalized temporal derivative \(L_{\zeta '^n}(x', y', t';\, s',\tau ', c)\) computed from the temporally scaled video sequence \(f'\) will assume a local extremum over temporal scales at

This scale-invariant property can also be extended to spatio-temporal derivatives \(L_{\xi ^{m_1}\eta ^{m_2}\zeta ^n}\) and spatio-temporal differential invariants \(\mathcal{D}_{\mathrm{norm}}\) defined in terms of homogenous polynomials as well as homogeneous rational expressions of such scale-normalized spatio-temporal derivatives. In this way, by performing both the temporal modelling of the underlying temporal signal in terms of the time-causal limit kernel and using a temporal scale-space concept based on the time-causal limit kernel, we can support fully scale-covariant temporal scale estimates for temporal scale selection in video data defined over a time-causal spatio-temporal domain.

The only component that remains is to determine how the original temporal scale estimate \(\hat{\tau }\) depends on the distribution parameter c and the temporal scale normalization parameter \(\gamma _{\tau }\) for some value of \(\tau _0\).

For audio signals, corresponding temporal scale selection methods can be applied to a time-causal spectro-temporal domain, with the 2-D spatial domain of video data over the spatial dimensions (x, y) conceptually replaced by a 1-D logspectral domain over the logspectral dimension \(\nu \) in the spectrogram computed at any temporal moment using a time-causal receptive field model as proposed in Lindeberg and Friberg [78, 79] or with the time-causal kernels in that model replaced by the time-causal limit kernel [75].

The analogous operations to the first- and second-order temporal derivatives of the spatial dimension would then be the first- and second-order temporal derivatives of the second-order derivative in the logspectral dimension

where s denotes the logspectral scale and \(\tau \) the temporal scale.

By calibrating the logspectral scale normalization parameter \(\gamma _s\) such that the selected temporal scale should reflect the logspectral width of a spectral band, it follows that we should use \(\gamma _s = 3/4\). By calibrating the temporal scale normalization parameter \(\gamma _{\tau }\) such that the selected temporal scale of \(\partial _{t,\mathrm{norm}}(L_{\nu ,\mathrm{norm}})\) should reflect the temporal duration of an onset, if follows that we should use \(\gamma _{\tau } = 1/2\) for this operator. By instead calibrating the temporal scale normalization parameter \(\gamma _{\tau }\) such that the selected temporal scale of \(\partial _{tt,\mathrm{norm}}(L_{\nu \nu ,\mathrm{norm}})\) should reflect the temporal duration of a beat, if follows that we should use \(\gamma _{\tau } = 3/4\) for that operator.

Note that these operations can be expressed both over frequency-time separable spectro-temporal receptive fields and over glissando-adapted spectro-temporal receptive fields if we for glissando-adapted receptive fields also replace the temporal derivative operator \(\partial _t\) by the corresponding glissando-adapted temporal derivative operator \(\partial _{\bar{t}} = \partial _t + v \, \partial _{\nu }\), where v denotes the glissando parameter.

In this treatment, we have proposed a new theoretical framework for temporal scale selection in a time-causal scale-space representation. Starting from a general survey of previously proposed temporal scale-space concepts and a detailed analysis of their relative advantages and disadvantages, we have focused our efforts on the time-causal scale-space concept based on first-order integrators coupled in cascade and analysed the extent to which scale-space properties that hold for the scale-invariant non-causal Gaussian temporal scale-space concept can be transferred to this time-causal scale-space concept. Specifically, we have analysed this time-causal scale-space concept for two specific ways of distributing the intermediate temporal scale levels using either (i) a uniform distribution over the temporal scales as parameterized by the variance of the temporal scale-space kernel and corresponding to convolution with temporal kernels that are Laguerre functions and in turn corresponding to temporal derivatives of the Gamma distribution or (ii) a logarithmic distribution taken to the limit of a recently proposed time-causal limit kernel with an infinitely dense distribution of temporal scale levels towards zero temporal scale.

For peak and ramp detection, we have shown that for the time-causal temporal scale-space concept based on first-order integrators with equal time constant coupled in cascade, we can reasonably well estimate the temporal scale of a localized temporal peak or a localized onset ramp with corresponding good approximation of constancy of appropriately post-normalized scale-normalized magnitude measures of the corresponding feature detectors under variations in the temporal duration of the underlying temporal peak or the underlying temporal ramp. For a non-localized sine wave signal, the lack of temporal scale invariance is, however, substantial both with regard to a systematic offset in temporal scale estimates and a lack of corresponding constancy of the magnitude measures over variations of the wavelength of the underlying sine wave.

For the time-causal temporal scale-space concept based on convolution with the time-causal limit kernel with an underlying logarithmic distribution of the temporal scale levels and taken to the limit of the time-causal limit kernel with an infinitely dense distribution of temporal scale levels near temporal scale zero, we have on the other hand shown that it is possible to achieve perfect temporal invariance in the respects that (i) the temporal scale estimates in dimension \([\text{ time }]\) are proportional to the wavelength of the underlying sine wave and (ii) the magnitude measures remain constant under variations of the wavelength of the sine wave.

Additionally, we have shown a general scale invariance result that holds for any temporal signal and which states that for temporal scaling transformation with a temporal scaling factor given as an integer power of the distribution parameter c of the time-causal limit kernel that is used for generating the temporal scale space, it holds that:

1. (i)

   local extrema over temporal scales of scale-normalized derivatives are preserved under this group of temporal scaling transformations with scaling factors of the form \(t' = c^j t\) for integer j,

2. (ii)

   the corresponding scale estimates are transformed in a scale-covariant way corresponding to \(\hat{\tau }' = c^{2j} \hat{\tau }\) in units of the variance of the temporal scale-space kernel,

3. (iii)

   if the scale normalization parameter \(\gamma \) of variance-based scale-normalized derivatives is chosen as \(\gamma = 1\) and corresponding to \(p = 1\) for \(L_p\)-normalization, then the magnitude values of the scale-normalized temporal derivatives are preserved under scaling transformations with any temporal scaling factor \(c^j\) that is an integer power of the distribution parameter c of the time-causal limit kernel and

4. (iv)

   for other values of the scale normalization parameters \(\gamma \) or p, the corresponding scale-normalized derivatives are transformed according to a scale-covariant power law (104), which is straightforward to compensate for by post-normalization.

In these respects, the proposed framework for temporal scale selection in the scale-space concept based on the time-causal limit kernel provides the necessary mechanisms to achieve temporal scale invariance while simultaneously being expressed over a time-causal and time-recursive temporal domain. From a theoretical perspective, this is a conceptually novel type of construction that has not previously been achieved based on any other type of time-causal temporal scale-space concept.

As experimental confirmation of the derived theoretical results regarding temporal scale selection properties, we have presented experimental results of applying two types of more specific temporal scale selection algorithms to one-dimensional temporal signals, based on either (i) sparse scale-space extrema detection by detecting local extrema of feature responses over both time and temporal scales or (ii) dense feature maps over temporal scales here specifically manifested in terms of a temporal quasi quadrature entity that constitutes an energy measure of the local strength of scale-normalized first- and second-order temporal derivates.

We have also described practical details to handle in time-causal scale selection algorithms in relation to the inherent temporal delays of time-causal image measurements and proposed specific mechanisms to handle the differences in temporal delays between time-causal scale-space representations at different temporal scales.

Experimental results presented for synthetic and real one-dimensional temporal signals show that it is possible to compute local estimates of temporal scale levels that in units of the standard deviation of the underlying temporal scale-space kernel are proportional to the temporal duration of the underlying structures in the temporal signal that gave rise to the filter responses.

Beyond these two specific ways of expressing temporal scale selection mechanisms, we argue that the theoretical results presented in the paper should also more generally open up for extensions to other ways of comparing time-dependent filter responses at multiple temporal scales.

Experimental results obtained by applying this temporal scale selection theory to video analysis will be reported in a companion paper [74].