Sådhanå (2019) 44:161 https://doi.org/10.1007/s12046-019-1138-5 Ó Indian Academy of Sciences Sadhana(0123456789().,-volV)FT3 ](0123456789().,-volV) Towards smartphone-based touchless fingerprint recognition PARMESHWAR BIRAJADAR1,* , MEET HARIA1, PRANAV KULKARNI1, SHUBHAM GUPTA1, PRASAD JOSHI2, BRIJESH SINGH2 and VIKRAM GADRE1 1 Indian Institute of Technology Bombay, Mumbai 400076, India Department of Cyber Maharashtra, Mumbai 400021, India e-mail: birajadar20@gmail.com 2 MS received 28 June 2018; revised 7 April 2019; accepted 8 April 2019; published online 13 June 2019 Abstract. The widely used conventional touch-based fingerprint identification system has drawbacks like the elastic deformation due to nonuniform pressure, fingerprints collection time and hygiene. To overcome these drawbacks, recently the touchless fingerprint technology is gaining popularity and various touchless fingerprint acquisition solutions have been proposed. Nowadays due to the wide use of the smartphone in various biometric applications, smartphone-based touchless fingerprint systems using an embedded camera have been proposed in the literature. These touchless fingerprint images are very different from conventional ink-based and live-scan fingerprints. Due to varying contrast, illumination and magnification, the existing touch-based fingerprint matchers do not perform well while extracting reliable minutiae features. A touchless fingerprint recognition system using a smartphone is proposed in this paper, which incorporates a novel monogenic-wavelet-based algorithm for enhancement of touchless fingerprints using phase congruency features. For the comparative performance analysis of our system, we created a new touchless fingerprint database using the developed android app and this is publicly made available along with its corresponding live-scan images for further research. The experimental results in both verification and identification mode on this database are obtained using three widely used touch-based fingerprint matchers. The results show a significant improvement in Rank-1 accuracy and equal error rate (EER) achieved using the proposed system and the results are comparable to that of the touch-based system. Keywords. Biometrics; touchless fingerprint recognition; monogenic wavelet; phase congruency; fingerprint enhancement; android app. 1. Introduction A tremendous development has taken place in the area of automated touch-based fingerprint identification in the past [1] and recently it has applications ranging from the simple biometric attendance system to the large-scale national identification programme like Aadhaar [2] launched by the Indian Government. The advancements in fingerprint acquisition have changed from ink-based techniques to a touchless acquisition, as shown in figure 1. Although the touchless acquisition technology is in the initial stage of development, it is drawing more attention of researchers and sensor industries. In [3], Labati et al have presented a comprehensive analysis and the state of the art of touchless fingerprint recognition technologies. More recently, NIST (National Institute of Standards and Technology, USA) also initiated a research program CRADA (Cooperative Research and Development Agreement) [4] to promote research in touchless fingerprint recognition. The main objective of this program is to produce open testing methods, metrics and artefacts for contactless fingerprint acquisition and recognition. The primary reason for opting the touchless technology is the non-uniform contact area and elastic distortion that exist in touch-based systems. This increases the FNMR (false nonmatching rate), which is a serious problem in applications like deduplication [2]. Different techniques have been explored by the researchers to overcome the distortion in touch-based systems [5] at the acquisition stage, prior to matching and during the matching stage. To overcome the problem of non-linear distortion, the touchless fingerprint acquisition is an ultimate solution and it simultaneously offers added benefits like hygiene and minimum fingerprint collection time. This research work attempts to address the following research questions: 1. Can we use the existing touch-based fingerprint matchers for touchless fingerprints to extract the reliable minutiae features for matching? *For correspondence 1 161 Page 2 of 15 Sådhanå (2019) 44:161 touchless fingerprint databases. In section 4, we describe the details of the monogenic-wavelet-based phase congruency feature estimation of the proposed enhancement algorithm. Section 5 introduces our proposed touchless fingerprint recognition architecture and details of the developed android app. In section 6, experimental results are summarized using three different fingerprint matchers on the created database and finally, section 7 concludes the paper. 2. Related work Figure 1. Advancements in fingerprint acquisition techniques. 2. Are the in-built enhancement algorithms of the existing touch-based fingerprint matchers capable of sufficiently improving the quality of touchless fingerprint images? 3. Can we use a smartphone-based touchless fingerprint recognition system in large-scale identification application instead of using it for small-scale applications like locking and unlocking of smartphones for security? In our work [6] reported at an international conference, we have proposed monogenic-wavelet-based phase congruency features for touchless fingerprint enhancement, which are effectively used for extraction of minutiae features. In this work, we have proposed monogenic-waveletbased phase congruency features for touchless fingerprint enhancement, which are effectively used for extraction of minutiae features. To enable complete system design and implementation, an approach to android-based touchless fingerprint recognition is proposed and the architecture of our proposed system is shown in figure 18. The key contributions of this paper can be summarized as follows: 1. A new touchless fingerprint benchmark database captured by smartphone camera along with touch-based equivalents acquired by the live-scan fingerprint scanner. 2. A novel monogenic-wavelet-based touchless fingerprint enhancement algorithm using phase congruency features. 3. A detailed comparative performance analysis between touchless and touch-based systems in both verification and identification modes, using three standard fingerprint matchers. 4. An implementation of a prototype smartphone-based touchless fingerprint recognition system. 5. An android app development with a user-friendly interface for touchless fingerprint enrollment and authentication. The rest of the paper is organized as follows. Section 2 describes the work related to touchless fingerprint recognition. Section 3 explains about the touch-based and In the literature, various 2D and 3D touchless fingerprint acquisition approaches [7] are proposed. The 3D fingerprints [8] can be generated using techniques such as shape from shading (SFS) or photometric stereo, which are bulky and costly. The accuracy of 3D fingerprint is highly dependent on its precise reconstruction as well as its subsequent feature extraction, and hence the performance of 3D fingerprint matching is lower than that of live-scan fingerprint-based matching. The simplest approach to acquire a touchless fingerprint is to capture a 2D fingerphoto using a less costly camera. In the literature, various attempts [9–12] are made to acquire a fingerphoto using a smartphone camera and various segmentation approaches to obtain a touchless fingerprint from a fingerphoto. In [9], the authors have proposed an approach based on Scattering Wavelet Network to extract texture-based features [13] for matching fingerphotos captured from mobile phone cameras. They also performed segmentation and enhancement on fingerphoto images to improve matching accuracy. They also created a database consisting of 128 classes captured in uncontrolled environment and with different backgrounds. The authors performed the comparative analysis of live scan to fingerphoto matching using different machine learning algorithms. In [10], the authors have captured fingerphotos from 41 subjects using smartphone cameras in an uncontrolled environment. They have performed feature extraction based on segmentation and minutia on fingerphoto images. The authors have mainly focused on different fingerphoto capturing techniques and quality assurance to obtain a good quality ROI from fingerphoto images. In [11], the authors have used non-conventional scaleinvariant texture features (SURF) for fingerphoto matching and evaluated results on a database of 50 clients captured in an uncontrolled environment. The fingerprint image is extracted from fingerphoto image using segmentation based on morphological operations such as erosion and dilation. The authors have used adaptive histogram equalization technique for fingerphoto enhancement. The authors in [12] collected 190 fingerphoto images using a mobile phone camera from 2 subjects with 5 samples under 19 scenarios. They have mainly focused on Sådhanå (2019) 44:161 quality assessment and fingerprint quality metric mapping using Fast Fourier Transform (FFT)-based features. On the other side, mobile biometric technology has become more popular, convenient to use and is replacing costlier biometric scanners. Various biometric traits such as fingerprints, face and iris are increasingly being adopted by law enforcement agencies for identification and verification of suspects/criminals. Mobile apps for different biometrics (iris, palmprint and finger knuckle) [14–16] have become a popular choice and can be used to capture and enroll a biometric template and send it to a remote server via 3G/4G networks at a very fast speed. Companies like Samsung [17], TBS [18], TrueID [19] and Diamond Fortress [20] are also incorporating touchless fingerprint recognition feature in their products. In our proposed approach, we have captured 800 touchless fingerprint images collected from 200 subjects (4 samples per subject). The fingerprint images are captured by placing the finger within the bounding box [20] of the camera interface provided in the app, which eliminates need of segmentation. A novel monogenic-wavelet-based algorithm for enhancement of touchless fingerprint images using phase congruency features is proposed. Most of the approaches described earlier use texture-based features for fingerprint matching. In this work, we have used minutiae features for purpose of fingerprint identification. The acquired database also consists of 800 touch-based fingerprint images of the corresponding touchless fingerprint images. We have performed comparative analysis between touchless and touch-based recognition using three different widely used standard touch-based matchers. A systematic performance analysis is evaluated using both verification and identification experiments. 3. Touchless and touch-based fingerprint database We have developed a smartphone-captured large touchless fingerprint database and its equivalent touch-based fingerprint database. IIT Bombay, Touchless and Touch-Based Fingerprint Database is a fingerprint database prepared by Indian Institute of Technology Bombay, Mumbai, India. The snapshots of android app for fingerprint acquisition are shown in figure 2. The images of the database are captured under uncontrolled illumination conditions both in indoor and outdoor environments. The dataset consists of 800 touchless fingerprint images of 200 subjects, 4 samples per subject having image size 170  260. It also consists of 800 touch-based fingerprint images of the same 200 subjects having image size 260  330. The touchless fingerprints are captured using a Lenovo Vibe k5 plus smartphone with our developed android app. The images are captured using embedded flash for proper illumination. Touch-based fingerprints are captured using an eNBioScan-C1 (HFDU08) scanner. The database will be made publicly available at https://www.ee.iitb.ac.in/ Page 3 of 15 161 Figure 2. Fingerprint acquisition. Left: touchless acquisition and right: touch-based acquisition. Figure 3. Database sample images. Top row: touchless fingerprints and bottom row: corresponding touch-based images. *dsplab/Biometrics/Touchless_Database.html to promote further research in this area. The sample images from the database are shown in figure 3. Currently, smartphone-captured touchless fingerphoto databases are available [9, 11], which are captured in an unconstrained environment and requires segmentation for further processing. The utilization of bounding box [20] for implementing our touchless fingerprint database eliminates the need for segmentation and minimizes the effect of scaling, rotation and translation on the fingerprint images. The aim of preparing and sharing such a database is to help researchers in their endeavours in comparing the performance of touchless and touch-based fingerprint biometric systems. 4. Touchless fingerprint enhancement using monogenic wavelet 4.1 Motivation The fingerprint images acquired by a smartphone camera are very different from touch-based fingerprints due to 161 Page 4 of 15 Sådhanå (2019) 44:161 Figure 4. Top row: (a)–(d) contrast changed synthetic cosine grating images and bottom row: (e)–(h) corresponding minutia-marked images using Verifinger SDK. varying illumination, contrast and magnification. Varying illumination across the fingerprint image reduces the accuracy of minutiae extraction. To demonstrate this effect, we conducted an experiment for minutiae extraction on synthetic images using Verifinger SDK. For assessing the minutiae extraction results, we need the ground truth location of minutiae points in advance. Since, in the touchless fingerprint image, it is difficult to mark all minutiae points manually, we created synthetic continuous phase cosine grating images and placed minutiae points by adding spiral phase [21] at specific locations. As shown in figure 4, the minutiae point extraction in a touchless fingerprint image is sensitive to variable illumination and contrast across the image. Since the fingerprint is a structured ridge–valley oriented pattern, it is important to extract ridges efficiently from touchless fingerprint images to extract reliable minutiae features. The magnitude of the signal has the information about the energy content whereas its phase gives overall structural information [22]. The analytic signal [23] is widely used in 1D signal processing applications. For 2D images, the 1D local phase and its orientation are important factors in describing the local structure of the images. In 2001, M. Felsberg and G. Sommer proposed the monogenic signal as an extension to the 1D analytic signal [3] using Riesz transform [24] for multiple dimensions. Kovesi [25] proposed the concept of phase congruency (described in section 3.3), where the phase can be effectively utilized for illumination-invariant edge detection. The validity of phase congruency depends on the local phase extraction over a band of frequencies, and hence the use of wavelets [26] is appropriate in phase congruency estimation. The main idea behind wavelet analysis is that one can use a bank of filters to examine the signal locally in space and frequency, simultaneously. The local phase of a signal can be extracted using its analytic signal, which is obtained by means of quadrature filters. In sections 4.3 and 4.4, we summarize the concept of phase congruency and its estimation using monogenic wavelets. 4.2 Monogenic wavelets After recognizing the role of the analytic signal in describing the local structure (phase) in the 1D scenario, a natural way to move forward is to extend this concept to multiple dimensions. In the case of images, the instantaneous phase is insufficient to describe the local structure. The local orientation becomes another important factor in describing the local structure. Hence, the 1D analytic signal needs to be extended for multi-dimensional signals. There have been several attempts made in the literature [27]. Almost all of them give an extension to the Hilbert transform and then propose the construction of the 2D analytic signal from the original signal and generalization of the Hilbert transform. The Riesz-transform-based monogenic signal (2D analytic signal) [28, 29] is a generalization of an analytic signal in higher dimensions. The Riesz transform [24] is a vector-valued extension of the Hilbert transform in multiple dimensions. The Riesz transform, f RðxÞ ¼ ½fR1 ðxÞ fR2 ðxފT of f ðxÞ, in 2D can be defined as in Eq. (1) in a generalized form: Sådhanå (2019) 44:161 Page 5 of 15 161 Figure 5. Riesz transform kernels in spatial and frequency domains. fRj ðxÞ ¼ f ðxÞ  Rj ðxÞ ¼ lim cn !1 Z xj jyj [  jx yj yj nþ1 f ðyÞdy ð1Þ where j ¼ 1; 2; . . .; n, cn ¼ cððnþ1Þ=2Þ and cð:Þ is the Gamma pðnþ1Þ=2 function. If RðuÞ and RðxÞ are the frequency response and the impulse response of the Riesz transform, their expressions are given by Eqs. (2) and (3), respectively: RðuÞ ¼ RðxÞ ¼ x iu ½u vŠT ¼ i pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ½R1 ðuÞ R2 ðuފT juj u2 þ v 2 3 2pjxj ¼ ½x1 x2 ŠT 2pðx21 þ 3 x22 Þ2 ð2Þ ¼ ½R1 ðxÞ R2 ðxފT ð3Þ where R1 ðxÞ ¼ R1 ðx1 ; x2 Þ and R2 ðxÞ ¼ R2 ðx1 ; x2 Þ are the Riesz kernels in the image domain. The frequency domain representation of monogenic signal is given by Eq. (4): fM ðuÞ ¼ f ðuÞ þ i fR1 ðuÞ þ j fR2 ðuÞ ð4Þ where wavelets [30]. Let wðx1 ; x2 Þ be a real 2D wavelet; then the monogenic wavelet triplet is given by Eq. (7): 3 2 wðx1 ; x2 Þ 7 6 wM ðx1 ; x2 Þ ¼ 4 wR1 ðx1 ; x2 Þ ¼ R1 ðx1 ; x2 Þ  wðx1 ; x2 Þ 5: wR2 ðx1 ; x2 Þ ¼ R2 ðx1 ; x2 Þ  wðx1 ; x2 ÞÞ ð7Þ The first observation one can make here is that the Riesz transform yields two signals corresponding to one in the form of a vector. Combination of these two signals with the original signal will result in a quaternionic signal with three components one of which is considered as real and the other two, imaginary. Also, Hilbert transform can be considered as a special case of the Riesz transform with v ¼ 0. The vector representation of the monogenic wavelet triplet in the spherical co-ordinate system is as shown in figure 6. The main ability of the monogenic wavelets is the simultaneous extraction of local phase and orientation. We have used the Gabor wavelet-based monogenic wavelet analysis to extract phase congruency features of touchless fingerprint images and its estimation is described in section 4.4. 4.3 Phase congruency fRi ðuÞ ¼ f ðuÞ Ri ðuÞ; i  f1; 2g; iu R1 ðuÞ ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; 2 u þ v2 iv R2 ðuÞ ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi : u2 þ v 2 ð5Þ ð6aÞ ð6bÞ The Riesz transform kernels in spatial and frequency domains are as shown in figure 5. The concept of monogenic signal (2D analytic signal) can also be extended for Phase congruency [25] is a phase-based image processing model, widely used in feature extraction, primarily in edge detection applications. Other gradient-based edge detection methods, like the Sobel and Canny [31] operators, attempt to highlight edges on both the sides of fingerprint ridges in a touchless fingerprint image as shown in figure 9. These gradient-based operators are sensitive to illumination variations and do not localize edges accurately. However, while using these edge detector operators, it becomes difficult to separate these ridges and valleys. On the other hand, the 161 Page 6 of 15 Sådhanå (2019) 44:161 energy can be seen geometrically in figure 7. The Fourier ~. components of the signal can be plotted as a vector OB The projection of the sum of all Fourier components on real axis gives an original signal I(x) and projection on imaginary axis give Hilbert transform of I(x), that is H(x). The energy at any given point, represented by point ~ represents total A, is given by Eq. (11). The length of OA ~ can energy E(x). The projection of any component on OA be written as An cosð/n ðxÞ /ðxÞÞ as shown by green ~ is equal to the colour in figure 7. The length of OA summation of projection length of Fourier components and is given by Eq. (10): Figure 6. A vector representation of the monogenic wavelet. illumination-invariant phase congruency highlights a single phase response on ridges of touchless fingerprint image and hence spurious minutia detection can be avoided. For a 1D signal I(x) where X IðxÞ ¼ An cosð/n ðxÞÞ ð8Þ n the phase congruency is defined as PCðxÞ ¼ max /ðxÞ2½0;2pŠ P n An cosð/n ðxÞ P n An /ðxÞÞ ð9Þ where An and /n ðxÞ represent the amplitude and local phase of the nth Fourier component, respectively, at position x. The value of /ðxÞ that maximizes Eq. (9) at a considered point x is the weighted mean local phase angle of all the Fourier components. Using Taylor expansion for small x, cosðxÞ ¼ 1 x2 . Hence, finding the phase congruency maximum point is equivalent to finding where the weighted variance of local phase angles of Fourier components, relative to the weighted mean local phase (/ðxÞ), is minimum. The relationship between phase congruency and local ~j¼ jOA X An cosð/n ðxÞ /ðxÞÞ; ð10Þ EðxÞ ¼ X An cosð/n ðxÞ /ðxÞÞ: ð11Þ n n We can define phase congruency at point A as the ratio of ~ (the energy at x) to the overall path length the length of OA (Fourier components amplitude) required to reach the point A, as given by Eq. (12): ~j jOA PCðxÞ ¼ P : n An ð12Þ Phase congruency at any point is equal to the local energy [32] of the signal divided by the sum of the Fourier components amplitudes. Venkatesh and Owen [33] show that the local energy peaks in an image correspond to the points where phase congruency peaks occur. The local energy function is determined using quadrature filters as given in Eq. (13): EðxÞ ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi I 2 ðxÞ þ H 2 ðxÞ ð13Þ where H(x) is the Hilbert transform of I(x). An example of illustrating the local energy peaks at signal edges is shown in figure 8. Figure 7. A geometric illustration of phase congruency for 1D signal [25]. Sådhanå (2019) 44:161 Page 7 of 15 161 Figure 8. Example showing local energy peaks at signal edges. Phase congruency at any spatial location ðxÞ is defined as (14) the ratio of the local energy of the signal to the sum of the amplitudes of Fourier components: EðxÞ PCðxÞ ¼ P : n An ð14Þ The PC value ranges from 0 to 1. A higher value of phase congruency denotes a more significant and discernible feature. 4.4 Phase congruency estimation using monogenic wavelet framework interested in calculating local frequency in a signal, in particular the phase information. To preserve the phase congruency in a signal, we need to use only linear and zero phase filters, that is the filter must be either symmetric or antisymmetric. Hence, we need to design the bank of filters in such a way that the transfer function of each filter overlaps sufficiently with its neighbours so that the sum of the transfer function forms uniform band coverage in the spectrum. We can construct the decomposed signal up to a scale factor over a spectrum of frequencies. For maximum uniform coverage, the upper cutoff frequency of one transfer function should coincide with lower cutoff frequency of the neighbouring transfer function, as shown in figures 11 and 12. It is important to obtain spatially localized frequency information in images (figures 9 and 10) while calculating the phase congruency; the use of wavelets is ideal in such a situation. The main idea behind wavelets analysis is that one can use a bank of filters to examine the signal. The filters are created by rescaling the mother wavelet and are designed in such a way that each filter picks a particular spectrum of frequency from the analysed signal. The scales in the spatial domain vary in such a way that they give rise to a logarithmic scale in the frequency domain. We are Figure 9. Comparison between Canny and phase-congruencybased ridge detection: (a) original touchless fingerprint image, (b) ridge detection using Canny operator and (c) ridge detection using monogenic-wavelet-based phase congruency detection. Figure 10. Minutiae extraction using Verifinger SDK. Top row: raw touchless image with corresponding extracted minutiae points and bottom row: enhanced image with corresponding extracted minutiae points. 161 Page 8 of 15 Sådhanå (2019) 44:161 The upper cutoff and lower cutoff frequency are the frequencies at which transfer function falls to half its maximum value. Quadrature filters are used to obtain local phase by means of the 2D analytic signal (monogenic signal). The monogenic signal of a 2D real image f ðx1 ; x2 Þ is defined by Eq. (15): 3 2 fw ðx1 ; x2 Þ ¼ wðx1 ; x2 Þ  f ðx1 ; x2 Þ 7 6 fM ðx1 ; x2 Þ ¼ 4 fw1 ðx1 ; x2 Þ ¼ w1 ðx1 ; x2 Þ  f ðx1 ; x2 Þ 5 ð15Þ fw2 ðx1 ; x2 Þ ¼ w2 ðx1 ; x2 Þ  f ðx1 ; x2 ÞÞ Figure 11. Example of 1D frequency response profile of logGabor filters in linear and logarithmic scales at different scales. where wðx1 ; x2 Þ is isotropic wavelet, and w1 ðx1 ; x2 Þ and w2 ðx1 ; x2 Þ are the Riesz wavelet filters. A single phase along with its orientation is obtained using the monogenic framework. The local monogenic phase / and its orientation h are defined by Eqs. (16) and (17), respectively: Figure 12. Top row: (a)–(d) contrast-changed synthetic cosine grating images and bottom row: (e)–(h) corresponding minutia-marked images using Verifinger SDK. Sådhanå (2019) 44:161 /ðx1 ; x2 Þ ¼ tan 1 Page 9 of 15 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ! fR1 ðx1 ; x2 Þ2 þ fR2 ðx1 ; x2 Þ2 f ðx1 ; x2 Þ  f ðx ; x Þ  R2 1 2 : hðx1 ; x2 Þ ¼ tan 1 fR1 ðx1 ; x2 Þ ; ð16Þ ð17Þ The h ranges from 0 to p and / ranges from 2p to p2. We have used a real 2D log-Gabor isotropic wavelet for enhancement of touchless fingerprint. The isotropic wavelet exclusively handles the directionality using the monogenic nature of the signal as well as localizes the scale due to its multiscale nature. The radial frequency response profile of the 2D log-Gabor wavelet is given by Eq. (18). The real isotropic wavelet and its Riesz components together form the monogenic wavelet and the corresponding frequency responses are shown in figure 13. 0qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi112 ðu1 Þ2 þ ðu2 Þ2 @log@ AA x0 wðu1 ; u2 Þ ¼ exp   2 f 2 log x0 0 ð18Þ where u1 and u2 denote the frequency variables, x0 is the centre frequency of wavelet and f is the bandwidth scaling factor. 161 The advantages of using a log-Gabor function are the following: 1. No DC component, 2. a wide spectrum range can be obtained with less number of scales due to the absence of restriction for setting maximum bandwidth, 3. it has a Gaussian shaped frequency response along logarithmic frequency scale. Generally, the spacing between the ridges ranges from 3 to 24 pixels in fingerprint images of 500 dpi resolution, 2p resulting in a band of frequencies ranging from 2p 24 to 3 . In order to estimate phase congruency, we have used a multiscale (N) approach. For monogenic wavelet analysis at multiple scales, we have used N ¼ 4 scale decomposition and the first scale x0 is set as 2p 3 . Figure 14 shows the example of monogenic wavelet decomposition. At each scale, the Fourier component amplitude is given by Eq (19): An ðx1 ; x2 ; sÞ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi f ðx1 ; x2 ; sÞ2 þ fR1 ðx1 ; x2 ; sÞ2 þ fR2 ðx1 ; x2 ; sÞ2 : ð19Þ For N scales, the energy of the image is given by Eq. (20): Figure 13. Frequency responses of monogenic wavelet triplet: (a) isotropic wavelet wðu1 ; u2 Þ and (b) Riesz wavelet wR1 ðu1 ; u2 Þ and (c) Riesz wavelet wR2 ðu1 ; u2 Þ: Figure 14. Example of monogenic wavelet decomposition: (a) original image f ðx1 ; x2 Þ, (b) isotropic wavelet-filtered image fw ðx1 ; x2 Þ; (c) Riesz component-1 wR1 ðu1 ; u2 Þ, (d) Riesz component-2 wR2 ðu1 ; u2 Þ; (e) local amplitude Aðx1 ; x2 Þ and (f) local phase /ðx1 ; x2 Þ. 161 Page 10 of 15 Sådhanå (2019) 44:161 Eðx1 ; x2 Þ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ½fsum ðx1 ; x2 ފ2 þ ½fR1 sum ðx1 ; x2 ފ2 þ ½fR2 sum ðx1 ; x2 ފ2 ð20Þ where fsum ðx1 ; x2 Þ, fR1 sum ðx1 ; x2 Þ and fR2 sum ðx1 ; x2 Þ are the sums of the amplitudes of Fourier components of original signal and its Reisz components at different scales (N) given as follows: fsum ðx1 ; x2 Þ ¼ N X f ðx1 ; x2 ; sÞ; ð21Þ fR1 sum ðx1 ; x2 Þ ¼ N X fR1 ðx1 ; x2 ; sÞ; ð22Þ fR2 sum ðx1 ; x2 Þ ¼ N X fR2 ðx1 ; x2 ; sÞ: ð23Þ s¼1 s¼1 s¼1 The phase congruency defined at location ðx1 ; x2 Þ is determined by Eq. (24): PCðx1 ; x2 Þ ¼ PN Eðx1 ; x2 Þ s¼1 An ðx1 ; x2 ; sÞ : ð24Þ The local energy construction using the Fourier components in the spherical monogenic framework is shown in figure 15. Figure 16 shows the touchless fingerprint sample images and their corresponding enhanced images using the proposed enhancement algorithm. It can be clearly observed that the phase congruency has a single and high response on ridge structure of touchless fingerprint images. The enhanced touchless fingerprint images using our Figure 16. Top row: raw touchless fingerprints and bottom row: corresponding enhanced images using proposed algorithm. algorithm have a higher contrast between ridges and valleys as compared with the raw touchless fingerprint images. For illustration purpose, minutiae extraction experiment is conducted on raw and enhanced fingerprint image using a commercial fingerprint extractor and matcher (Verifinger SDK). For illustration purpose, minutiae extraction experiment is conducted on raw and enhanced synthetic cosine grating and fingerprint images using a commercial fingerprint extractor and matcher (Verifinger SDK). It can be clearly seen in figures 10 and 12 that a reliable and greater number of minutia points are extracted from the enhanced synthetic cosine grating and touchless fingerprint images as compared with that of the raw synthetic and touchless fingerprint images, respectively. 5. Touchless fingerprint recognition system The main purpose of implementing our own smartphonebased touchless fingerprint prototype recognition system is to do a systematic comparative performance analysis between touchless and touch-based fingerprint recognition and to bring the performance of touchless recognition to a level that is comparable to that of touch-based system using a novel monogenic-wavelet-based approach. The implementation details of the architecture of touchless fingerprint recognition system and development of android app are described in sections 5.1 and 5.2, respectively. Figure 15. Illustration of the local energy construction from its Fourier components in the spherical monogenic framework. The vectors shown in purple, blue and black denote the local Fourier components. The energy is represented by the vector shown in red. 5.1 Architecture of touchless fingerprint recognition system The touchless fingerprint recognition system shown in figure 18 is developed using Android platform 5.1.1 Sådhanå (2019) 44:161 (Lollipop). The application is developed on Android Studio version 2.3.1 [34] with compiled and target SDK version 24. The fingerprint image is of size 170260, which is captured with the help of a bounding box [20]. The touch-based fingerprints are captured using the eNBioScan-C1 (HFDU08) scanner from the developed application, which has a size of 260330. The fingerprint image along with the demographic information of the subjects is stored in a JSON object, which is converted to a string using GSON (an open source Java Library) and sent over-the-air in a UTF-8 encoded format, which is posted to the server via HTTP URL Connection. The server IP-address and port address are set in the socket and the data are sent from the mobile phone through the socket to the server. On the server side, these data are accepted by the Eclipse IDE [35], which runs a java socket program to accept the data. The server decodes the received string, extracts the fingerprint image and demographic information, enhances the touchless fingerprint image using our proposed monogenic-wavelet-based algorithm and extracts the minutiae features, which are then stored as enrollment templates in the SQL database. During identification, the template extraction and matching are performed using Verifinger SDK 7.1 [36]. The reason for selecting Verifinger as a matcher in this implementation is described in section 6. Both fingerprint verification and identification can be performed on the server. XAMPP [37] is an open source, cross-platform package, which is used to deal with server-side communication and to deal with the database. PhpMyAdmin package is available within XAMPP to work with MySQL with the use of web browser. MySQL queries are used to query the database. PHP scripts are used on the server side to communicate with the app and the database. MySQL queries are held in PHP scripts. Page 11 of 15 161 However, there is a lot of research yet required to bring their performance to an acceptable level. We have also developed the touchless fingerprint android application, which would enable a user to capture, enroll, match and store touchless as well as touch-based fingerprints directly on to a remote server. Figure 18 shows a user-friendly interface to capture the touchless and touch-based fingerprint images. The purpose of developing this app is to use the existing touch-based fingerprint matchers for touchless fingerprints and for comparing the performance of touchless fingerprints to that of touch-based ones over a large database. We have collected touchless and touch-based fingerprint database from 200 subjects. The finger must be placed 3–5 inches away from the rear camera of smartphone and within the provided bounding box. The camera autofocus and flash-LED are required for proper capturing of touchless finger images. The minimum required camera resolution is 8 MP for successful fingerprint capture. The touch-based fingerprint images can be acquired using the eNBioScan-C1 (HFDU08) scanner connected to the smartphone through OTG cable. The app has been tested on Google Nexus 5, Lenovo Vibe K5 plus and Redmi Note 3. The minimum API level supported by the app is 21 (Lollipop 5.0 and above versions of android operating system). The app is tested on smartphone having Octa-core Qualcomm Snapdragon 616 processor with a speed of 1.5 GHz and 3 GB RAM. A video demonstration of the developed android app is available at https://www.ee.iitb.ac.in/ *dsplab/Biometrics/Video.html for verification and identification. The app can have a number of applications in the fields of law enforcement like verification and identification of criminals/suspects in the field, information on missing children/adults and fugitive attendance. 6. Experimental results 5.2 Android application Of late, some of the commercial touchless fingerprint android applications (figure 17) are invading the biometric industry for user verification and identification [19, 20]. We analysed the performance of our touchless fingerprint recognition system in both verification and identification mode on the collected touchless and touch-based database Figure 17. Snapshots of android app illustrating user-friendly interface: (a) user information enrollment interface and (b) verification and identification. 161 Page 12 of 15 Sådhanå (2019) 44:161 Figure 18. Touchless fingerprint recognition system architecture. of 200 subjects. In order to verify the effectiveness of our enhancement algorithm, the experiments are conducted on raw touchless fingerprints as well as on enhanced touchless fingerprints. Finally, the overall performance is compared to that of the touch-based (live-scan) system. The experiments are performed using the two widely used open source and one Commercial Off the Shelf (COTS) matching systems, namely: Source-AFIS [38], NBIS-NIST [39] and Verifinger-SDK [36]. 1. Source-AFIS: SourceAFIS is an open source minutiaebased Automated Fingerprint Identification System library implemented in Java and .NET and developed by Robert Vanzan. It performs touch-based fingerprint preprocessing, minutia extraction and matching in verification and identification modes. 2. NBIS-NIST: It is an open source minutiae-based matching algorithm developed by NIST in the Linux environment. The MINDTCT and BOZORTH3 packages of NBIS are used for minutia detection and template matching, respectively. The minutiae information is available in the format \x; y; h; c [ , where x and y refer to the minutia location, h denotes the minutia orientation and c provides the confidence value in percentage of minutia detection. 3. Verifinger SDK: VeriFinger is a well-known minutiabased commercial software development kit (SDK) developed by Neurotechnology and is used by researchers and biometric solution providers. In verification mode, the matching accuracy for touchless and the touch-based system is ascertained from the equal error rate (EER) and the receiver operating characteristic (ROC). Since the database consists of m ¼ 200 classes and n ¼ 4 samples per class, the total number of genuine ðm  n  ðn 1ÞÞ=2Þ and imposter ððn2  m  ðm 1Þ=2ÞÞ comparisons are 1200 and 318400, respectively. In identification mode, the performance of the proposed system is measured from the Rank-1 accuracy and the cumulative match characteristics (CMC). The probe set contains 200 images (first image of each subject) and gallery set contains 600 images (remaining three images of each subject). Each probe image is compared against all the gallery images and totally 12000 matching scores are determined. The resulting scores are sorted and ranked. The ROC and CMC curves for three different matchers are shown in figures 19 and 20, respectively. In table 1, the EER for the verification experiments and in table 2, the Rank-1 (R1) recognition accuracy for the identification experiments are reported. It can be ascertained from these curves and the performance metrics (EER and R1) illustrated in tables 1 and 2 that the raw touchless fingerprint performance is very poor compared with that of the touchbased system. This can be observed consistently in case of all fingerprint matchers. It clearly indicates that inbuilt enhancement algorithm of these touch-based matchers is not suitable to extract the reliable minutia features from raw touchless fingerprints. However, the experiments conducted on the enhanced touchless fingerprint images using the proposed enhancement algorithm show significant improvement in EER and R1. The COTS Verifinger matcher outperforms the other two open source matchers in Sådhanå (2019) 44:161 Page 13 of 15 161 Figure 19. ROC curves showing verification performance with three different matchers. Figure 20. CMC curves showing identification performance with three different matchers. Table 2. Performance comparison of touchless and touch-based recognition using three fingerprint matchers in terms of Rank-1 (R1) recognition accuracy. Rank-1 accuracy (R1) (%) Matching algorithm Source-AFIS NBIS-NIST Verifinger-SDK Touch based Touchless enhanced Touchless raw 92.5 94.5 100 36.5 71 100 2 49.5 89 Table 1. Performance comparison of touchless and touch-based recognition using three fingerprint matchers in terms of equal error rate (EER). Equal error rate (EER) Matching algorithm Source-AFIS NBIS-NIST Verifinger-SDK Touch based Touchless enhanced Touchless raw 7.37 5.65 0.6 33.99 14.74 1.18 47.85 35.9 10.67 touchless as well as in touch-based fingerprint recognition. An overlap of ROC and CMC curves for touch-based and enhanced touchless fingerprints using Verifinger SDK can be clearly observed in figures 19(c) and 20(c), respectively. Local phase plays a major role in describing the ridge structure of the fingerprint image. Gabor wavelets allow access to the local phase, but they are distributed over several scales and orientations. The monogenic wavelets capture the local phase and local orientation orthogonally with respect to the magnitude and hence are suitable for extracting the ridge structures of touchless fingerprints. In our previous work [40], we have proved the ability of capturing local phase with monogenic wavelets compared with Gabor wavelet and Fourier phase by conducting phase-based reconstruction experiments. The main reason of the performance improvement is the effective enhancement of ridge structures of touchless fingerprint images using the proposed enhancement algorithm. The illumination-invariant phase congruency features are extracted using multiscale monogenic wavelets as described in section 4.4. As shown in the block diagram (figure 18), touchless fingerprint images are acquired using a smartphone camera with created bounding box, which enables proper focus and segmentation. 161 Page 14 of 15 7. Conclusions and future work In this work, we have successfully implemented the touchless fingerprint recognition system based on smartphones including the necessary android app, server and feature extraction/matching modules. We have compared the performance of touchless and touch-based fingerprint recognition system on a newly created touchless fingerprint database. The database will be made available to researchers and this will help promote further research in this field. We proposed a novel monogenic-wavelet-based touchless fingerprint enhancement algorithm using phase congruency features to improve the matching accuracy and this is incorporated in our system. The log-Gabor filters effectively extract the illumination-invariant phase congruency features. Experimental results conducted using the three existing matchers show a significant improvement in Rank-1 accuracy (R1) and EER using the proposed enhancement algorithm on touchless fingerprint images. Hence, the existing matchers designed for touch-based fingerprints can also be efficiently used for touchless fingerprint matching by adding appropriate enhancement preprocessing steps like the one proposed in this work. Further improvement in the matching accuracy can be achieved by designing the robust isotropic analytic wavelets to estimate phase congruency features for touchless fingerprint enhancement. The concept of colour monogenic wavelets [41] can be explored for phase congruency estimation for colour touchless fingerprint images. The proposed touchless fingerprint recognition system can be improved by incorporating features like touchless fingerprint quality assessment, fingerprint liveness detection, template security and cross-sensor matching. Acknowledgements This work was supported by the NCETIS (National Center of Excellence in Technology for Internal Security) and MHRD-TEQIP-KITE, a TEQIP initiative of the Ministry of Human Resource Development at IIT Bombay. The authors would also like to thank the students of IIT Bombay, for helping them create the touchless fingerprint database. They also wish to acknowledge the active participation and support of Shri Balsing Rajput and Shri Deepak Dhole of Department of Cyber Maharashtra, Mumbai. References [1] Maltoni D, Maio D and Jain A 2009 Handbook of fingerprint recognition, 2nd ed. New York: Springer-Verlag New York Inc [2] Aadhaar [online] Unique Identification Authority of India. https://uidai.gov.in/ [3] Labati R, Piuri V and Scotti F 2015 Touchless fingerprint biometrics, 1st ed. Boca Raton, FL, USA: CRC Press Sådhanå (2019) 44:161 [4] CRADA [online] NIST research program. https://www.nist. gov/programs-projects/contactless-fingerprint-capture [5] Si X et al 2015 Detection and rectification of distorted fingerprints. IEEE Trans. Pattern Anal. Mach. Intell. 37(3): 555–568 [6] Birajadar P et al 2016 Touchless fingerphoto feature extraction, analysis and matching using monogenic wavelets. In: Proceedings of the IEEE International Conference on Signal and Information Processing (IConSIP), Nanded, October, pp. 1–7 [7] Labati R et al 2014 Touchless fingerprint biometrics: a survey on 2D and 3D technologies. J. Internet Technol. 15(3): 325–332 [8] Kumar A and Kwong C 2015 Towards contactless, low-cost and accurate 3D fingerprint identification. IEEE Trans. Pattern Anal. Mach. Intell. 37(3): 681–696 [9] Sankaran A et al 2015 On smartphone camera based fingerphoto authentication. In: Proceedings of the 7th IEEE International Conference on Biometrics Theory, Applications and Systems (BTAS), September, pp. 1–7 [10] Stein C, Nickel C and Busch C 2012 Fingerphoto recognition with smartphone cameras. In: Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG), September, pp. 1–12 [11] Tiwari K and Gupta P 2015 A touchless fingerphoto recognition system for mobile hand-held devices. In: Proceedings of the International Conference on Biomertrics (ICB), May, pp. 151–156 [12] Yang B et al 2012 Collecting fingerprints for recognition using mobile phone cameras. Proceedings of SPIE 8304: Multimedia on Mobile Devices [13] Bruna J and Mallat S 2013 Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell. 35(8): 1872–1886 [14] Cheng K and Kumar A 2012 Contactless finger knuckle identification using smartphones. In: Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG), pp. 1–6 [15] Putra G et al 2014 Android based palmprint recognition system. TELKOMNIKA 12(1): 263–272 [16] Raja K et al 2015 Smartphone based visible iris recognition using deep sparse filtering. Pattern Recognit. Lett. 57(1): 33–42 [17] Son B et al 2015 Method of recognizing contactless fingerprint and electronic device for performing the same. US Patent 0146943, May [18] TBS [online]. Available: http://www.tbs-biometrics.com/en/ [19] Trueid [online]. Available: http://www.trueid.co.za/touch less-fingerprints/ [20] Diamondfortress [online]. Available: http://www.diamond fortress.com/ [21] Feng J and Jain A 2011 Fingerprint reconstruction: from minutiae to phase. IEEE Trans. Pattern Anal. Mach. Intell. 33(2): 209–223 [22] Oppenheim A and Lim J 1981 The importance of phase in signals. Proc. IEEE 69(5): 529–541 [23] Gabor D 1946 Theory of communication. Part 1: the analysis of information. J. Inst. Electr. Eng. 93(26): 429–441 [24] Stein E and Weiss G 1971 Introduction to Fourier analysis on Euclidean spaces. Princeton, NJ: Princeton University Press Sådhanå (2019) 44:161 [25] Kovesi P 2003 Phase congruency detects corners and edges. In: Proceedings of the Australian Pattern Recognition Society Conference, DICTA Australia, pp. 309–318 [26] Gadre V and Abhayankar A 2017 Multiresolution and multirate signal processing, 1st ed. McGraw Hill Education, Noida, Uttar Pradesh, India [27] Bulow T and Sommer G 2001 Hypercomplex signals—a novel extension of the analytic signal to the multidimensional case. IEEE Trans. Signal Process. 49(11): 2844–2852 [28] Felsberg M and Sommer G 2001 The monogenic signal. IEEE Trans. Signal Process. 49(12): 3136–3144 [29] Unser M et al 2009 Multiresolution monogenic signal analysis using the Riesz–Laplace wavelet transform. IEEE Trans. Image Process. 18(11): 2402–2418 [30] Olhede S and Metikas G 2009 The monogenic wavelet transform. IEEE Trans. Signal Process. 57(9): 3426–3441 [31] Canny J 1986 Computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6): 679–698 [32] Aw Y et al 1998 An analysis of local energy and phase congruency models in visual feature detection. J. Australian Math. Soc. Ser. B 40(1): 97–122 Page 15 of 15 161 [33] Venkatesh S and Owens R 1989 An energy feature detection scheme. In: Proceedings of the International Conference on Image Processing, Singapore, pp. 553–557 [34] Android SDK [online]. Available: https://developer.android. com/index.html [35] Eclipse JAVA IDE [online]. Available: https://www.eclipse. org/downloads/ [36] Neurotechnology Inc. [online] Verifinger SDK. Available: http://www.neurotechnology.com/verifinger.html [37] XAMPP Apache Distribution [online]. Available: http:// www.neurotechnology.com/verifinger.html [38] Vanzan R SourceAFIS [online]. Available: https://sourceafis. machinezoo.com [39] NBIS Software-NIST [online]. Available: https://www.nist.gov/ services-resources/software/nist-biometric-image-software-nbis [40] Birajadar P et al 2016 A novel iris recognition technique using monogenic wavelet phase encoding. In: Proceedings of the IEEE International Conference on Signal and Information Processing (IConSIP), Nanded, October, pp. 1–6 [41] Soulard R et al 2013 Vector extension of monogenic wavelets for geometric representation of color images. IEEE Trans. Image Process. 22(3): 1070–1083