Hidden in Plain Sight: Exploring Privacy Risks of Mobile Augmented Reality Applications

There has been extensive work recently in exploring visual privacy implications of mobile and wearable camera-enabled systems. Work by Koelle et al. [38] proposes a gesture-controlled privacy framework; bystanders can provide a gesture to indicate whether the capturing device has permission to include their face in a photograph. Shu et al. [66] provide a more robust contextually sensitive framework to allow bystanders to control how they are captured by cameras, but only for individual images, rather than real-time video feeds, as mobile AR systems require. However, neither of these solutions protect bystanders who are unaware or unable to provide a gesture, nor do they protect the end-user from capturing privacy-sensitive content in the environment.

It is also possible for applications to capture sensitive information from the user’s environment that the user would prefer to keep private, such as written or printed material, faces of unaware bystanders, and images taken in a bathroom or bedroom. Systems have been proposed to protect environmental information using markers such as near-infrared labels [44] as well as QR codes and other physical markers [28, 61, 62] to declare to the capturing device what can and cannot be recorded. Schiboni et al. [63] address the visual privacy problem in a dietary monitoring system by manually modifying the camera viewing angle and field-of-view to limit the system’s ability to capture bystanders. Ultimately, the main drawback of both gesture-based and marker-based access control systems is that they only work if the application developer is proactive in protecting user and bystander privacy and chooses to implement these solutions; these systems do nothing to protect against a malicious developer.

When focusing on malicious AR applications specifically, there has been some work on using the device’s OS to limit the access that developers have to sensitive visual information. Two works by Jana et al. [34, 35] create an intermediate layer between the camera and AR application code, but introduce a significant amount of overhead and limit the flexibility of honest app logic. More recent work by Lebeck et al. [40] proposes the ARYA framework for AR output security, but their system targets visible output issues such as content occlusion, distraction, and physiological reactions such as motion sickness. Their system cannot defend against a malicious developer whose attacks take place completely behind the scenes.

One recent work by Srivastava et al. [69] seeks to categorize camera-based applications by the operations performed on visual data and how well that aligns with user expectations of application behavior. The key difference between this work (known as “CamForensics”) and ours is that we specifically focus on malicious developers of mobile AR applications, leveraging commercially available vision libraries to conduct additional hidden operations behind the scenes of an otherwise honest application. While the CamForensics system can help identify what high-level operations are being performed, it is unable to determine the context of such an operation, e.g., whether it is honest or malicious. This is especially important when the same high-level operation (such as image labelling) is being used for both honest and malicious purposes within the same application.

One of the principal methods of access control for mobile and wearable systems is for application developers to request permission from users to access sensitive features such as the camera and microphone at runtime. It is well-understood, however, that these types of permissions structures are deeply flawed; work by Felt et al. [26] found that over a third of publicly available Android applications request more permissions than they need, with many issues stemming from developer confusion regarding the nature of each permissions group and what functions they represent. End-users are similarly unaware of the scope of requested permissions [27]. Regardless of whether access to a given feature is truly necessary for an application, once it is provided by the user, it is not requested again. Recent work has demonstrated that this all-or-nothing approach to system permissions is unrealistic, particularly for vision-based systems [66, 69]. In a perfect world, applications would execute sensitive operations only in accordance with end-user expectations for a given scenario, a concept known as “contextual integrity” [54]. Wijesekera et al. [90] showed that maintaining contextual integrity is not a simple problem; factors such as background operations, number of delivered notifications, the reason for the operation, and users’ personal privacy concerns all play into whether a user would allow a given operation in the moment.

A survey was conduct by Akter et al. [5] which elicits privacy preferences related to persons with visual impairments using camera systems to supplement their vision, an important use case for mobile augmented reality. The authors surveyed both persons with visual impairments and those without (considered “bystanders”). The bystanders expressed discomfort that users of such systems would be able to learn things such as facial expressions and behaviors that had the potential to be “misrepresented” by the system’s underlying algorithms. In another work [6], Akter et al. investigated how persons with visual impairments felt about unintentionally sharing privacy-sensitive images for different usage scenarios, such as different environments (home, office, or restaurant) and with different categories of human assistants (friends, family, or crowd-worker). In this article, the participants expressed vastly differing comfort levels with sharing captured images based on the environment, what the image contained, and who it was being shared with.

Liu et al. [45] propose an alternate framework for setting system permissions; they used permissions settings gathered from existing smartphone users, and from those settings, compiled “profiles” representing meta-preferences of like-minded users. They then presented new users with an interactive tool and predicted each user’s preferences based on a few targeted questions. Of the permissions set by the assigned profiles, only 5% of settings were changed back by the users. This suggests that alternate methodologies, other than binary approval or rejection of individual permissions, can be useful and effective for end-users.

In addition to researching new techniques for testing and verifying machine learning modules, there have also been recent efforts in applying machine learning concepts and techniques to distinguish between honest and dishonest applications on smartphones. Work by Burguera et al. [11] monitors calls made by Android applications to the underlying Linux kernel and applies K-means clustering to identify malicious applications. Other works have applied similar clustering approaches to other information about the applications under test, such as the requested permissions [82] or sequences of API calls [31, 48]. One study by Afonso et al. [2] uses machine learning to classify an app as malicious or honest based on a dynamically generated feature vector by tracing API calls. However, the authors mention that their approach (at the time of publishing) could be detectable by said app, which may lead to a change in the app’s behavior.

Using machine learning to categorize typical patterns and detect subsequent anomalies in runtime behavior is another popular research area. Abah et al. [1] created an anomaly/malware detection system that extracts features from running applications and uses these features to classify app as malicious or safe using a trained K-Nearest Neighbor classifier. Another study by Kurniawan et al. [39] analyzes phone sensor data for anomalies. The data they analyze is power usage, battery temperature, network, CPU, Bluetooth, and other hardware resources that is tracked by a mobile device for anomalies. However, this approach uses the Android Malware Genome Project dataset; at the time of writing this article, no such dataset exists for mobile AR applications. Other approaches utilize Generative Adversarial Networks (GANs) to preserve visual privacy and defend against attacks [93, 95, 96]. However, these approaches are focused on sanitizing datasets collected for offline processing, making them inappropriate for real-time augmented reality operations.

There has also been work to develop “trusted” neural networks that adhere to additional constraints beyond their initial functions. One approach proposed by Halliwell et al. [32] enforces saliency constraints on convolutional neural networks; in doing so, developers can trust that the internal layers of the network are leveraging the correct pixels of an input image to make classification decisions. Ghosh et al. [29] propose a parameterized approach to determine the “cost” of trusting a neural network’s output for a given set of conditions. Their work leverages steering directions for autonomous vehicles, where a dual-headed network calculates a proposed steering angle for a set of traditional inputs (e.g., camera frames), as well as a “danger” score for violations of a set of safety constraints (e.g., minimum and maximum allowable angles). The developer is then free to adjust the weights between these results to maximize accuracy while keeping the danger score within acceptable limits. Approaches like these require the participation of the network developer; they can do nothing if the developer fails to incorporate these additional constraints into their network, either through unintentional omission or malicious intent. Additionally, such measures only increase confidence that the function of the neural network itself can be trusted. They cannot control or verify that the context under which the function is being executed is appropriate.

Deep learning (DL) are one subset of machine learning systems that have gained in particular popularity, thanks to their unique applications in domains such as autonomous vehicles [7]. However, these types of systems also carry great risk, as errors in controlling logic can lead both to personal injury or loss of property. As such, interest in testing and verification of DL systems has increased in recent years. (This contrasts with machine learning-based testing, in which machine learning concepts are applied to testing of traditional systems, discussed above in Section 2.3.) The challenge is that testing practices of traditional software systems do not translate directly to DL systems. Where traditional software systems are “logic-driven” (e.g., behavior is controlled by logic flows manually encoded by their programmers), DL systems are “data-driven,” where behavior is governed by the architecture of the underlying neural network, the computations and activation functions being performed at each node, the weights between nodes, and the quality of the training data [100]. There are few “constituent parts” upon which to conduct unit testing, meaning the system must be tested as a whole [98]. However, simply feeding inputs to the system and evaluating the resulting output is an ineffective strategy; the search space of all possible inputs to such systems (as well as enumerating their respective expected outputs) is so large as to make brute-force testing prohibitively time- and resource-intensive.

Developing new strategies for testing deep learning systems, as well as metrics for measuring success, is a popular research area. Much effort has gone into developing metrics to represent logical coverage within a neural network. Proposed systems such as DeepXplore [56] and DeepTest [83] leverage a concept known as “neuron coverage,” where inputs are scored according to how many neurons are activated in their processing. This concept was inspired by traditional software testing metrics such as “code coverage,” in which testers seek to execute every line of code in a code base by the end of the test suite. Both approaches manipulate input images to maximize neuron coverage but differ in how the results are evaluated. DeepXplore evaluates the results by cross-referencing with other DL systems, while DeepTest applies metamorphic relations to detect if the output is within acceptable limits. DeepGauge [49] builds on this idea, proposing the concepts of \(k\)-multisection neuron coverage (how well a given input covers the activation range of a given neuron) and top-\(k\) neuron coverage (a measurement of the most commonly activated neurons in a given layer of the network). These approaches, however, only consider how a system responds to a given input in isolation and not how the responses differ between inputs; to address this, Kim et al. [37] propose a metric called “surprise adequacy” that measures how novel a given input is relative, not only to the network’s training data, but to the set of inputs overall. Retraining the network with “surprising” inputs is then demonstrated to increase network accuracy.

There has also been interest in developing property-based verification methods for DL systems, beyond simply verifying the accuracy of outputs for a given set of inputs. Pei et al. [57] treat the internal network of the system as a black box, validating a given input/output pair against a high-level safety property rather than trying to ascertain the nature of the network directly. Wang et al. [89] apply symbolic linear relaxation and constraint refinement to validate the internal activation functions of a network to enforce safety properties. The developers of DeepMutation [50] utilize principles of mutation testing, applying mutations directly to the training dataset, the network training program, and the resultant model to identify errant behavior. Test inputs are fed into both the original model and the set of mutated models; errors are detected when the output from a given mutated model for a given input differs from that of the original model.

While all of these approaches can provide some assurances into the ability of a given deep learning system to produce reliable output or to adhere to a given safety property, they all fail to address one particular problem: They do not ensure contextual integrity, that is, that the system under test adheres to the user’s expectation of appropriate behavior in a given scenario (discussed above in Section 2.2).

The ability for developers of mobile augmented reality applications to perform “hidden operations,” that is, unadvertised machine learning or computer vision operations on live camera frames, constitutes a serious threat to end-user privacy. Deng et al. describe seven key threats to user privacy, known as the LINDDUN model [20]: Linkability (the ability to determine whether two items of interest are related in the system), Identifiability (the ability to positively identify an item of interest within the system), Non-repudiation (the ability to conclusively tie an actor back to an action taken within the system), Detectability (the ability to determine whether an item of interest exists within the system), information Disclosure (the ability to expose personal information that should not be accessible), content Unawareness (the inability of an end-user to fully know the extent of data collected or inferred by the system), and policy and consent Non-compliance (the ability of the system to go against advertised privacy policies and end-user expectations).

By conducting a hidden operation behind the scenes of the advertised AR application functionality, the developer engages in three of the LINDDUN privacy threats in particular: information disclosure, content unawareness, and policy and consent non-compliance. First, the system exhibits information disclosure by leveraging a legitimate data stream to potentially learn personal data that it should not have access to (such as the user’s age, gender, or socio-economic status). The system also results in content unawareness by keeping secret the nature and extent of data gathered during the operation. The fact that the user is unaware of the hidden operation is fundamental to the operation’s success. Finally, the system exhibits policy and consent non-compliance by exploiting the permissions granted for the advertised AR functionality to execute the additional hidden operations behind the scenes.

There are a number of ways in which the adversary can implement their hidden operations. In this article, we only consider the adversary who utilizes existing vision and machine learning libraries to implement their hidden operations. This is because commercially available libraries provide the most convenient method for developing AR apps, since they require a much lower level of skill and domain knowledge compared to implementing such logic from scratch. Additionally, we assume that the adversary is completing the majority of their machine learning logic locally on-device; this is because augmented reality applications have a strict threshold of tolerable computational latency. Offloading the entirety of vision or machine learning operations to the cloud would incur too much latency to be feasible for real-time responsiveness as required for AR system output [46, 101]. Further, we assume that the hidden operations are also performed locally, as offloading frames for processing en masse risks getting caught by network monitors for transmitting large amounts of image or video data, without any guarantee that the files contain anything of interest.

The scope of the hidden operations privacy threat can be demonstrated through the following sample scenarios:

In the following sections, we discuss a selection of possible implementations for hidden operations in mobile AR and propose a design for improved testing and permissions of mobile AR apps.

Building a mobile AR application consists of two complementary pipelines, as described in Figure 3. The first pipeline reflects the development of computer vision functionality, while the second reflects the mobile application development. The first two steps of each pipeline can be executed in tandem with each other.

The first step in the vision pipeline is to identify the AR or CV operations that the application should support and which AR or CV library will be utilized to implement them ✓. Common vision operations include ground plane detection, facial recognition, object classification, and others. Once the desired vision operation and corresponding library have been identified, the second step in the vision pipeline is to collect or develop the assets necessary to accomplish the selected operation ①. “Assets” are resource files utilized by the system during runtime to complete the desired operation, and they can vary depending on the type of operation being executed as well as the library being used. For example, 2D image recognition with ARCore requires copies of the images to be recognized, while object classification with TensorFlow requires datasets of images representing the various classes to be recognized, which are then used to train a neural network.

The first step of the mobile pipeline consists of building the initial application skeleton ②, including establishing the system architecture, setting up scenes, connecting to device sensors, and other tasks. The second step involves importing the AR or CV library of choice ③, as well as supplementary tasks pertinent to the chosen library, such as registering a developer key. Once the vision assets have been assembled and the preliminary application developed, the developer can then integrate the assets into the application ④, typically by registering them with the library at application start-up. Finally, the developer can implement listeners to respond to results of the vision operations as they become available ⑤. Developers are free to respond to these results in whatever way makes sense for their application. Examples of potential responses include placing a graphical overlay in the user’s view to highlight an object, placing a label to give more information, playing a sound, delivering haptic feedback, or nothing at all.

We assume that the adversary is utilizing a commercially available vision library to build their AR application and implement their hidden operations. When implementing one of these operations, the adversary has three broad categories of libraries to pick from (as summarized in Table 1): integrated AR libraries; function-level computer vision and machine learning libraries; or general purpose vision and machine learning libraries. The developer may also combine any of these libraries together into a single app, both within and across categories. For example, an application that uses Vuforia to identify beverage labels can also leverage ARCore to detect and track face meshes, or an application that uses ARCore to place virtual superheroes on a tabletop could also use TensorFlow to identify other objects in the room and generate customized speech bubbles for the models. In doing so, the adversary can leverage the strengths of one type of library to compensate for the limitations of another. Next, we go into greater detail explaining the differences between these various vision libraries.

There are currently no publicly available, open source examples of known malicious AR applications; therefore, we developed three prototype applications to investigate the extent to which honest and dishonest MAR applications differ from each other. The applications were fully implemented in Java for the Android operating system as proofs-of-concept for increasing levels of maliciousness. The first application (A1) represents the “honest” application, in which the only operations being executed are in keeping with advertised functionality and are thoroughly communicated to the end-users. For the purposes of this article, we designed A1 to recognize faces and place an overlay on the camera feed, outlining the primary landmarks of the recognized face (e.g., eyes, mouth). The second application (A2) builds on the first, supplementing the honest logic with a complementary hidden operation, that is, one that builds upon the existing honest logic without informing users of its existence or purpose. A2 calculates the age and gender of the detected faces, but provides no indicator to the user of the operation. Finally, the third application (A3) supplements the honest logic with an orthogonal hidden operation, that is, one that is completely independent of the existing honest logic, but is similarly undisclosed to the end-users. A3 performs text recognition on frames taken from the live feed, an operation that is performed regardless of the results returned from the primary logic.

Based on the similarity comparison above, we make the following conclusions: First, we observe that the increasing availability and maturity of commercially available vision libraries makes it quite simple to conduct hidden machine learning operations without informing the end-user. Commercially available vision libraries give an adversary a host of options to incorporate into their application, catered to their needs and pre-existing subject matter expertise. These powerful operations can be used for a variety of different purposes, none of which are required to be consistent with the application’s advertised functionality or to be communicated to the end-user. As demonstrated above, in the case of low-overhead functionality such as text recognition, the introduction of new libraries may not even introduce new artifacts into the APK’s assets or libs directories. Additionally, because these operations are being conducted in the background, there are no impacts to the runtime UI of the application to alert the end-user to their presence. This has the effect of lowering the bar to create AR applications in the first place, while simultaneously making it easier to violate the privacy of end-users by exploiting these same libraries.

Second, we see that the OS-level permissions structures currently provided by mobile platforms are insufficient to protect against hidden machine learning operations. All three of the adversarial applications described above, honest and dishonest alike, achieved their functionality using the same list of permissions, e.g., access to the camera and write permissions to storage, which were requested by more than 90% of AR applications from the Google Play Store. Access to a given permission is requested on first execution of an application after install; once granted, it is never requested again, and the application has full access to any and all features related to that permission. Thus, an application will request its permissions once and then have no subsequent accountability on when or why it is accessing that functionality. Further, contemporary mobile operating systems make no requirements on developers either to request permission to perform machine learning operations or to make any visual or auditory indications to users when those operations are being performed. Based on this, in current mobile operating systems, there is no way to tell an honest application from a dishonest one based on the permissions they request.

Based on these observations, in the following sections, we propose and perform exploratory experiments on a system that seeks to mitigate the hidden operations threat. This system addresses the problem first by preventing developers from executing unvetted vision logic, and second by increasing user awareness of application behavior during runtime.

Our proposed system design consists of three principal components. The first component is a trusted verification and signing service for computer vision models. This service would be responsible for vetting vision functionality before it is allowed to be executed on the local device; it would offer a repository of pre-vetted, commonly used models (similar to the function-level organization of libraries such as Google’s ML Kit) and provide verification services for developers to submit their own custom models built with general purpose libraries such as TensorFlow. Signed versions of the vetted models would then be available for download, along with a unique key for each developer to use in authentication, thereby preventing the developer from modifying or replacing the model after download.

Figure 10 demonstrates the proposed workflow by which a developer would submit a custom model for verification. This workflow assumes that, in addition to the model itself, the developer provides all relevant meta-data describing the model’s purpose, operations, and targets to be recognized. The trusted service provider (such as Google for Android or Apple for iOS) would then perform their own independent validation, verifying that the model adheres to expected functionality and performs all advertised functions. Once the model has been verified, the service provider would return the signed version of the model as well as a unique key to the developer.

Once the developer has downloaded their model and associated key, they would then utilize our second principal component: a trusted computing partition on the local device in which to run the signed vision model, as shown in Figure 11. During runtime, the application would interface with the trusted partition to request operations from the custom model, providing the unique key to authenticate the request. The trusted partition would then interface with privacy-sensitive sensors such as the camera and microphone before returning the results to the application. By inserting this trusted computing layer in between the application and privacy-sensitive sensors such as the camera, the developer is able to perform approved operations using the signed model, but is unable to access the data feeds directly, removing their ability to perform additional operations on raw camera frames.

Finally, the device OS would leverage our system’s third and final component, an expanded permissions structure to communicate desired computer vision operations to end-users at runtime. This component leverages the meta-data associated with the application’s signed vision models and exposed by the trusted signing service. As such, the permissions could be structured in a number of different ways. For example, if an application is using pre-existing functionality from trusted function-level libraries such as Google’s ML Kit, then the OS publisher might require the developer to request permission from the user to execute that particular function (e.g., text recognition or image labelling). Alternatively, if the developer is using a signed custom model, then the OS publisher could provide the end-user with the results of that model’s verification process before deciding whether to permit the application to continue. The high-level operation permission can also be supplemented with a brief explanation of why the functionality is being requested. These prompts would supplement or replace the overly vague and unhelpful request for wholesale access to the device’s camera and will provide end-users with the opportunity to reflect and decide if an application’s request for a particular functionality is acceptable before engaging in use of the app.

There is a wide variety of established methods for testing and verifying expected behavior of software systems and machine learning modules that the trusted signing service publisher could utilize. Examples include but are not limited to such approaches as static and dynamic analysis and metamorphic testing. The purpose of this section is not to identify the perfect verification method, but to explore how different verification methods could be incorporated into this system.

The system proposed above is high-level and somewhat abstract; this is because we recognize the scope and complexity of addressing the hidden operations threat in full is too great for this article to cover satisfactorily. Our goal in proposing this system is not to present a single panacea solution, but rather to identify components that can be developed in synergy to mitigate the threat and its various facets. Full realization of this proposed system will require multi-disciplinary contributions from the worlds of computer vision, machine learning, systems security, and human-computer interaction.

The ultimate goal of the proposed system is to limit a developer’s ability to execute vision or machine learning logic without alerting the end-user. It does this through two main approaches. The first approach is by preventing developers from executing unvetted vision logic. By forcing developers to use trusted services to implement vision functionality (either by utilizing an existing function-level library or submitting a custom model for intense validation), we remove the adversary’s ability to execute custom machine learning logic without supervision. This mitigates the LINDDUN threat of information disclosure by adding another layer of review to the app’s machine learning components, preventing them from gathering more data from the user than what is permitted.

As mentioned in Sections 2.3 and 2.4, the testing of machine learning components and the systems that use them is a difficult problem [98, 100]. There are strategies that a given library publisher can employ to ensure the technical correctness of a given machine learning module [37, 49, 50, 56, 83], but these strategies cannot guarantee that an application developer will not use the module in ways contrary to user expectations. The second approach is, therefore, to increase user awareness of application behavior during runtime through the use of meta-data exposed by the trusted signing service and increased granularity of requested permissions. By providing dedicated permissions for computer vision functions and requiring end-users to explicitly approve or deny such permissions, we remove the adversary’s ability to perform such operations without informing the user. This mitigates the LINDDUN threats of user unawareness and policy and consent non-compliance by ensuring that the user is as informed as possible about the true nature of an application’s background operations. Once alerted to suspicious application behavior, it is then up to the user to respond appropriately, such as changing his or her own behavior or removing the application entirely.

Each application-under-test (AUT) was executed on a smartphone placed in a dock facing a computer screen. This method was utilized to engage the full camera frame delivery pipeline, rather than feeding a video or set of raw image files directly to the application. The computer then played a video showing a rotation of input images. The input video was split into three phases, showing one minute each of neutral images (e.g., recognized by none of the applications), then expected images (e.g., inputs that match the applications’ advertised functionality), and finally sensitive images (e.g., inputs that are potentially privacy-invasive and have nothing to do with the applications’ advertised functionality). For all AUTs, the expected functionality was facial recognition, while the sensitive functionality was one of either age and gender estimation or text recognition.

We tested four conditions: baseline, worst case, and stress tests for the facial and text recognition conditions. Input videos were generated for each condition, as described in Table 3. Results were collected and averaged over five trials from each application for each video. All tests were executed using a Samsung Galaxy S9 smartphone (Qualcomm SDM845 Snapdragon 845 processor, 4 GB memory) and a 15-inch MacBook Pro laptop (3.1 GHz i7 processor, 16 GB DDR3 memory). We simulated the proposed inspector system through a series of command-line scripts, executed through the Android Debug Bridge (adb) and the hardline connection between the smartphone and computer. Using this script suite, we collected a spectrum of runtime traces (CPU, memory, and storage utilization; display frame rate; active process information) as the application-under-test processed the input video. These traces were then utilized for analysis. All code files and data collection scripts used for these evaluations are publicly available on GitHub.¹

Our first set of tests utilized the adversarial applications described in Section 4.3, where A1 is the honest application, performing the expected operation \(\alpha\) (e.g., facial recognition). A2 and A3 are dishonest applications, performing complementary and orthogonal hidden operations \(\beta\) (age and gender estimation) and \(\gamma\) (text recognition), respectively, in addition to \(\alpha\).

We analyzed the resource consumption traces to determine whether any behavioral differences could be observed for the various categories of inputs. First, we examined the traces over time for different stress test conditions to determine if we could manually observe any changes in behavior. Next, we performed statistical analysis to determine the similarity of traces collected from different apps for the same sequence of inputs. Finally, we performed K-means clustering to group traces by input category in the attempt to distinguish malicious apps from honest.

Our second round of evaluations considered a group of commercially available third-party applications. Three applications were selected that explicitly exhibit facial recognition: Instagram [76], Snapchat [78], and SweetCam [79]. These applications were selected due to their popularity (over 1B downloads for both Instagram and Snapchat, and over 100M downloads for SweetCam) and high reviews (3.8 stars with 120M reviews for Instagram, 4.3 stars with 26M reviews for Snapchat, and 4.4 stars with 930K reviews for SweetCam). While we do not believe any of these applications to be malicious, we do expect them to adhere to the behavioral patterns described in Figure 12, where there are noticeable changes to resource consumption patterns when the application is processing an expected target, compared to “baseline” levels of consumption for both neutral and sensitive inputs.

The test conditions were the same for the commercial applications as the adversarial applications. Four input videos were used (Table 3) to cover the baseline (denoted as “noise-faces-signs” (NFS)), worst case (denoted as “noise-group-wiki” (NGW)), and stress test conditions (denoted as “noise-group-signs” (NGS) for faces and “noise-faces-wiki” (NFW) for text). Each application was configured to place a dog overlay (e.g., ears and nose) onto a person’s face when recognized. The applications were run on a smartphone positioned in a dock in front of a computer screen playing the input video to test the full camera pipeline. The smartphone was connected to the computer through a hardline, which collected resource consumption traces. CPU and memory consumption were tracked; display framerate was not trackable without either modifying the original application or significant instrumentation to the device. Results were averaged across five trials for each application processing each input video (n = 20 trials per app).

Given the proposed system described in Section 5, application marketplace publishers will have access to additional knowledge of what runtime operations a given AR application is supposed to be performing. In this section, we presented examples of how this knowledge can be leveraged to instigate noticeable behavioral changes during runtime in malicious applications. For the malicious prototypes, we were able to induce visible spikes in the resource consumption traces through stress testing of recognized target density and classify the honest/dishonest nature of a given application with relative accuracy using both statistical analysis and K-means clustering (62.5% and 67%–71%, respectively). Based on the results for the prototypes with known hidden operations (A2 and A3), we conclude that the additional meta-data provided by our proposed system shows promise in assisting with testing efforts when differentiating between honest and dishonest AR applications.

An interesting observation is that the tests conducted above demonstrated varying levels of success in identifying the malicious applications, depending on the type of hidden operation being conducted. The hidden operations of the complementary malicious application were more visible in runtime monitoring of the display frame rate; the hidden operations of the orthogonal malicious application were more visible in the CPU traces. Both applications responded to stress testing to induce runtime resource consumption spikes and displayed success in both detection by statistical analysis and K-means clustering. While the base prototype (A1) did not present observable changes in runtime behavior and scored lowest of the three in K-means clustering, we do not consider this to be problematic. An application that consumes consistent amounts of resources regardless of input (such as the commercial application, SweetCam) can be flagged for additional testing specifically because of its resistance to change.

A final observation is that, when applying our intuition to commercial applications, the expected behavior changes were not observed for all of the tested apps; some applications exhibited noticeable change to runtime behavior in response to different inputs, while others did not. While the commercial applications are presumed honest, the heterogeneity of runtime behavior patterns both supports our intuition and encourages future study on those applications that exhibit inconclusive patterns. Based on these results, and the varied success of the detection methods for the malicious prototypes, we can conclude that no single test is a catch-all solution; a variety of testing strategies must be incorporated to detect hidden operations in AR. The best approach for this remains an open problem. In the next section, we discuss the limitations of our proposed design and explore open problems for future research.

The success of the system proposed above depends on the mobile and machine learning communities coming together to develop standards and conventions for dealing with privacy problems in mobile AR and other camera-based systems. Here, we present a number of open problems in this area.

As observed previously, some tests are better suited to detecting different kinds of hidden operations over others. Even within the same application, performance and behavior can change, depending on the type of input being provided. Using the CPU and memory data traces collected from the commercial applications in Section 6.3, we applied the K-means clustering approach from Section 6.2.3 to see if the applications that demonstrated conformant behavior also responded well to clustering. Table 6 summarizes the results. The accuracy for assigning data points to the appropriate clusters (e.g., neutral, expected, sensitive) was understandably much lower for Instagram and SweetCam than for the malicious prototypes, as these apps exhibited relatively consistent resource consumption, regardless of the input condition. However, Snapchat, which exhibited the most promising resource consumption trends during runtime, failed utterly during clustering, with an accuracy of only 5%. Our intuition in this case is that, while the application exhibited the expected behavior for expected inputs when looking at the CPU trace on its own, minor lags in either the application’s response to an input phase change or the trace collection script in logging those changes were enough to throw off the clustering assignments.

It should be emphasized that the goal of this article is not to propose a single one-size-fits-all solution; rather, we have identified a range of application behaviors and identified approaches that show promise in identifying different patterns. There remains, however, much work to be done in improving these processes. Here, we present a number of open problems related to improving software testing methods, given the unique nature of mobile augmented reality systems and their runtime behaviors.