AccessLens: Auto-detecting Inaccessibility of Everyday Objects

There exist numerous standards and normative tools to help non-experts learn cumulative knowledge. The Americans with Disabilities Act (ADA) Standards for Accessible Designs [16] and the International Building Code (IBC) [28] represent comprehensive frameworks to alleviate mobility challenges. Increasing interests are in their interactivity, for instance, improving indoor access for older adults through interactive systems [11, 18, 42, 43, 55]. Homefit AR [55, 56] guides users through questionnaires to precisely locate issues with object types and recommends alternatives with better access. The closest prior work of ours is RASSAR [68], a mobile AR application to assess indoor accessibility upon standards such as low tables, narrow entryways, and dangerous items exposed. While these works respond to the needs of special interest groups (e.g., older adults and wheelchair users), a broader population is often excluded, since they have not experienced disabilities and could overlook contextual or situational disabilities. We are to provoke solutions with an emphasis on the engagement of a more diverse community in creating accessible and accommodating indoor spaces.

Fostering empathy is discussed in many disability studies to elevate awareness about the lived experiences of disabled people [8, 57, 58]. While simulating disabilities such as blindfolding [60], having colorblind effects [19], or trying wheelchairs [46] has gained popularity, disability advocates disparage simulated disability [8, 27]; as it is difficult to accurately replicate the real experiences [4]. Empathy alone may not suffice to sustain attention [27], simulations may inadvertently create biases or distress [53], resulting in perpetuated ableism [25]. More recent focus is on co-designing with people with disabilities (e.g., [7, 32, 33, 74]); e.g., citizens, healthcare professionals, and designers co-design personalized healthcare solutions  [33], sighted and blind participants design building navigation together [32]. Collective efforts to enhance the user experience can extend the impact beyond individuals with disabilities alone, encompassing different abilities of all  [48, 61, 63]. Unfortunately, there have been only little to no systems to engage non-experienced users to cultivate inclusivity. Our approach is motivated by “design for one, expand to all” and “learning from adaptations” [48] to promote awareness through noticing and better designs.

Visual perception of indoor places is a critical first step to improving one’s quality of life [65]. Various datasets were released to train detectors. Some early datasets such as MIT indoor scenes [59] and SUN RGB-D [66] have advanced techniques to train recognition models. Synthetic datasets such as HyperSim [62] can further advance recognition models with their variety and quantity. While there exist relevant datasets such as Gibson [76], offering a virtual visual navigation platform, PartNet [51] with focus on part of indoor objects, and BEHAVIOR-1K [34], data for embodied AI systems to foster human-robot interaction, none have centered interaction types to assess their accessibility and user contexts. We find ADE20K [77], which is a large-scale indoor scene dataset with hierarchical annotations of objects in images at the pixel level, promising. Refining the hierarchical taxonomy of objects and parts by ADE20K includes object categories and parts, we curate datasets by re-annotating potentially inaccessible objects to train and evaluate inaccessibility detectors.

While it is not feasible to replace all existing objects overnight [3, 35], 3D-printed assistive designs [5, 10] promises low-cost, custom solutions to redress everyday interaction challenges (e.g., [10, 14, 22]). These adaptations can range from magnifying cabinet knobs for improved grip (e.g., ‘ThisAbles’ [67]) to self-serving medicine dispensers [3]. Similar to the modular approach employed in the modern software engineering paradigm, wherein updates are selectively applied only where changes are necessary [54], the augmentation allows for unit-by-unit enhancements tailored to specific needs. Barriers to 3D printing have been significantly lowered [9], existing works studied motivations behind online communities sharing assistive 3D designs  [10] and proposed computational customization solutions (e.g.,  [14]). While documents based on similarity can classify shared designs’ objectives [36], current search relies on designer-created descriptions, often failing users to explore viable designs to rectify hidden inaccessibility that is not obvious to those without diagnosed disabilities. Discovering suitable designs heavily relies on keyword-based searches, relying on the textual information provided by the authors: titles, descriptions, and tags only. We propose novel metadata to categorize existing 3D assistive augmentations for better identification of solutions.

In sum, Table 1 summarizes the position of Accesslens.

We conducted an exploratory survey on Thingiverse [29], to gain insights into why individuals are motivated to create 3D assistive augmentations and modify existing physical objects to address specific contextual or situational interaction challenges. First, we listed several indoor objects that are very common around us, including door knobs, light switches, etc. Then we retrieved 3D designs that are for those objects, indicating 3D designs tend to augment targeted real-world objects from Thingiverse. We employed an iterative process of affinity diagramming, which was collaboratively performed by four of our authors. In the affinity diagramming process, we classified augmentations considering three primary criteria: (1) their intended objective, which refers to the barriers the augmentations aim to address, (2) the type of objects the augmentations target, and (3) any related motions or actions associated with their use. Our empirical findings revealed that even for objects that are under the same class (e.g., door knob/handle, light switch), the augmentations are much more diverse due to differences in the object’s type (e.g., single toggle light switch vs. rocker switch). This diversity emanates from shapes, motions, and objectives, which inspired us to develop AccessDB, our refined dataset with inaccessibility classes of AccessMeta. This iterative affinity study resulted in three high-level functions of adaptations as follows and example augmentations are shown in Figure 4.

We developed the prototype of the AccessLens and conducted a comparative study to assess its validity and advanced features over the baseline, MS Inclusive Design Guidebook [48]. Compared to other normative tools that are targeted to diagnosed disabilities, e.g., ADA Standards for Accessible Design [16], the MS guidebook is the foremost design guideline that argues accessibility as a universal daily challenge for all, encouraging recognizing exclusion, extending the inaccessibility concept to contextual from a permanent problem. Herein, the disability is discussed not as a personal health condition, but as ‘mismatched human interaction’ which we see the potential to rectify through augmentations. Thinking of solutions for those situational disabilities can allude to a design for one that can benefit all [61], empowering people to learn from diversity. AccessLens prototype (Figure 5) includes objects with detected potential accessibility challenges. Tapping on objects, the system displays relevant 3D augmentations depending on contextual needs. We provide the contexts through a catalog approach, helping users learn from viewing adaptations list, which also presents design implications to nurture people’s understanding of solutions.

Consideration #1: One-shot Image Input From the HCI perspective, allowing users to upload a single photo of an indoor scene would offer a more pleasant experience, considering that our target users might not know where to focus. While detection performance can benefit from multiple photos of the indoor scene, it is more friendly for users to take a single photo of the entire room or scanner view to check whether there exist any inaccessibility concerns. We target one-shot imagery of indoor scenes of interest as input, i.e., a panoramic scan of a bathroom, living room, and office space.

Consideration #2. Semantic Understanding of Parts To assist users with different needs, detecting part (doorknob from a door) and discerning the type of the object (doorknob vs. lever) is critical to articulate contextual barriers beyond simple object detection. The system must detect target objects and the parts where actual user interaction occurs, since each presents unique barriers with associated interaction types, for example, a knob for grab-pull vs. a knob for grab-rotate. Therefore, the image dataset must contain indoor scenes with part-level annotations.

Consideration #3: Recognizing Disability Attributes. Various contexts change the way that people with a wide spectrum of capabilities interact with everyday objects; for a graphic designer wearing a splint due to chronic wrist pain, a door knob is not accessible as it requires hard grasping to rotate. People are often frustrated with a panel with identical toggle switches; without labels, they are forced to recall targets or try to get the right one turned on, sometimes causing safety breaches. The disability context attributes of the objects might fortify the existing dataset. In sum,

We define “assistive augmentations” herein as attachments to legacy physical environments, addressing inexplicit barriers in varying contexts. Ever since numerous 3D printing practitioners have open-sourced their creations online (c.f., [10]), many were posted with voluntary textual descriptions with “assistive” to indicate the design intention. Some not originally intended to be assistive missing relative tags could also be used for access but makes shopping through millions of designs by searching exhausting. Navigating options is even more laborious due to ambiguity in language [36]. The structured rules or metadata to categorize assistive augmentations will broaden access to those designs, enabling users to explore easily.

Auto-detecting objects with their semantics and context from camera views (e.g., [50, 55, 68]) can assist visual perception for various interested groups and information processing, e.g., robotic affordance and different types of disability. Automation through a comprehensive dataset that provides a granularity of object classes is critical to infer necessary information from semantics. Yet, predicting contexts from images is more complex than detecting objects and instances; object attributes such as shapes (e.g., round, lever, cross-shaped) must relate their functional properties (e.g., grip, twist, pinch), to be able to derive their conceptual interaction types. Once interaction types are inferred regarding their visual and functional characteristics, those types can serve as clues to infer the original design intent as well as hidden barriers in various possible contexts. To train and evaluate our developed inaccessibility detector, we construct two datasets: AccessDB and AccessReal. Being built for semantic understanding of objects and their parts, ADE20K offers hierarchical annotations on object classes, such as closet - door - handle and oven - door - handle. AccessDB presents Inaccessibility Class (IC) to provide a nuanced understanding of diverse barriers that may manifest across various contexts, extracted from six distinct categories in ADE20K: button panels, electrical outlets, faucets, handles, knobs, and switches. The granularity of IC permits the identification of specific accessibility challenges, thus enabling tailored design solutions. Refer to Table 3 in the Appendix A.

(1) AccessDB is used to train inaccessibility detectors. We derive AccessDB from ADE20K [77], which contains > 20k images including diverse indoor scene photos with pixel-level annotations on objects and their parts. We re-annotated objects in ADE20K for 21 predefined inaccessibility classes (ICs) in addition to “unidentifiable” class for extremely small sized parts. We first select scene images sampled from “home”, “hotel”, “shopping and dining rooms”, and “workplace”, but excluded low-resolution images. We focus on 6 object categories that are often inaccessible (Figure 8): handle, faucet, switch, knob, button panel, and electric outlet. Three annotators are HCI experts in assistive designs, and annotators also cross-verify each other’s annotations for annotation quality. We obtained 4,976 high-resolution images exhaustively annotated with ICs as illustrated in Figure 10, which appears in extremely small regions of the image, posing a visibility challenge to detectors.

(2) AccessReal. Since AccessDB’s images are from the ADE20K dataset which was published five years ago (as of 2023 when this research is conducted), we are motivated to curate a new dataset for evaluation by collecting photos taken in ‘modern’ indoor scenes. To this end, we take 42 high-resolution photos (mostly 4032 × 3024) in diverse indoor scenes: bathroom, bedroom, kitchen, living room, and office (cf. Figure 9). We annotate them w.r.t the predefined 21 ICs (see data statistics in Appendix A Table 3), and end up with 428 annotated objects with ICs.

Our approach allows adapting any state-of-the-art detector architectures (e.g., GroundingDINO [40] and RetinaNet [37]; details in Appendix C). Figure 13 displays example detection results on AccessReal images, showing good qualitative performance in detecting small inaccessible objects. In evaluation, (Section 5.1), all participants showed solid trust in our detector’s performance. Detection result visualizations for sample images in AccessReal (Figure 13) also show that the detector accurately captures small objects. AccessDB and AccessReal datasets are open-sourced at https://access-lens.web.app/ to foster future research. While our work used one of the state-of-arts, any modules can be trained on our dataset. For technical specifications, refer to Appendix C.

Engaging with a building ADA coordinator at our institution sheds light on a collective effort in identifying/reporting. The ADA coordinator admitted that many staff lack accessibility expertise, so they hire external accessibility specialists to address issues on demand. Encouraging citizen science within our initiative could mirror successful collective intelligence models like Project Sidewalk [64]. By adopting a reporting system where individuals contribute to accessibility assessment within commons, accommodating potentially inaccessible physical environments but have not yet discovered by people with diagnosed disabilities before they encounter barriers. Experts’ recommendations to input disability types and validate possible conflicts can be applied to seek AccessLens at scale.

The term ‘disability dongles’ has emerged to criticize endeavors that employ innovative technologies but fail to address genuine accessibility needs [41], often targeting industry products that exploit accessibility concepts for superficial gains. AccessLens builds upon prior work on understanding and addressing existing issues of recognizing accessibility barriers [68]. Our approach utilizes the state-of-the-art technologies that are becoming more and more available to innovate an individual’s life (e.g., [6]), thereby assisting users make informed decision-making. Moving one step forward, AccessLens broadens its impact to a wider audience who do not experience diagnosed disability, provoking discourse about disabled contexts [48]. 3D printed and/or DIY solutions already become the major efforts made by numerous disabled selves, stakeholders, and altruistic enthusiasts [10, 14]. As AccessLens promises to incorporate more options for store-bought solutions and the industrial design industry (e.g., [55]), we anticipate more collaborative efforts across disciplines to advance people’s quality of life using technology as we detail in the following section.

AccessMeta links the object types with their needed interaction, seeking solutions that might alter interaction types (e.g., grab-rotate-to-open vs. push-open). Once detected, we see the future of AccessMeta and the dictionary expanding the search for similarly-functioning 3rd-party alternatives, such as buying door lever replacements from hardware stores or online markets. While some simple replacements like doorknobs might be as cheap as 3D printing, more complex fixtures such as refrigerator handles (as in Figure 12) are not trivial, necessitating the disassembly or replacement of the whole appliances. Although our study participants agreed on the less mental burdens with AccessLens recommendations, some were inclined towards store-bought products as they have gone through market testing (U3), given their perceived affordability and time cost for customization (U5). As AccessLens provides direct recommendations compared to “for store-bought ones, I might have to look for products on my own that solve the highlighted challenge for detected objects" (U2), offering users more options upon various rationale, control for materials (U5), easy-fix and remix (U6).

3D printing is a promising solution for custom adaptive interfaces to meet unique needs. One notable example is auto-filling numerics into parametric 3D designs [36] and in creating various branches of augmentations upon user’s needs to adapt common household items [14]. The current AccessLens prioritizes inaccessibility detection and assistive augmentation recommendations. As our work has been focusing on increasing awareness and low-cost solutions, dealing with fit [31] and parametric customization was considered orthogonal. However, we recognize the potential synergy with existing works facilitating customization (e.g., [24]), starting from the auto-detection and selection of a suitable design and culminating in real-world applications. We can further empower individuals to take proactive steps toward creating inclusive environments.

This work provides two challenging datasets, AcessDB and AccessReal for inaccessibility detectors. Communities’ interest in inclusive designs has grown, and advances to automate everyday surroundings (e.g., smart switches, thermostats with touch screens) create new challenges; touch screens often lack tactile feedback for people with visual impairments and can present more challenges for the elderly). To scale the dataset, this work elaborated on the re-annotation strategy of AccessDB in detail at our dataset website. We believe that the AccessMeta pipeline should remain open-ended and adaptable to accommodate emerging needs and novel designs. One approach to expanding the AccessMeta pipeline is involving a community in reporting problems and suggesting additional metadata categories. We can ensure that the system remains responsive to real-world needs, identifying new challenges.

We envision the use of AccessLens will help people become more aware of implicit inaccessibility and more actively engaged in improving access in public spaces, such as lecture rooms and shared dormitory community rooms. We have not observed positive behavioral changes in participants beyond the lab. We plan to conduct a deployment study to evaluate whether AccessLens raises people’s awareness and encourages collective actions, similar to how altruism motivates voluntary sharing of designers online for free. Expert interviews from diverse domains, including HCI, accessibility, visualization, and citizen science, will be conducted to critique the user interface and study design systematically, ensuring unbiased evaluation of AccessLens compared to other existing tools.

The literature of object detection commonly uses the standard metric of mean Average Precision (mAP) at interaction-over-union (IoU) thresholds ranging from 0.5 to 0.95, with a step size 0.05 [39]. We use mAP as the primary metric. Following other prior works [2, 17, 47], we also report performance with respect to the metrics of AP₅₀ and AP₇₅ [23], meaning the Average Precision (AP) at IoU threshold 0.5 and 0.75, respectively.

AccessLens supports detection for all object classes in the 3D assistive augmentation dictionary and ICs. Although any detector structure can be chosen, we utilized two different state-of-the-art methods, RetinaNet [37] for ICs with training on AccessDB, and GroundingDINO [40] for zero-shot detection without training for more common classes (e.g., sofa, table, cup, etc.) Specifically, we trained RetinaNet [37] with ResNet-50-FPN backbone with 3x LR schedule, implemented by detectron2 [75]. In training, we employed COCO pretrained weights retrieved from Model Zoo of detectron2. As Figure 8 illustrates, AccessDB contains 21 inaccessibility classes, and one extra class, unidentifiable instances due to their extremely small size to identify with human eyes. For training and validation of the detector, we randomly split the dataset, 85% training and 15% validation (2,029 and 359 images for training and validation, respectively). We used the AccessReal dataset for testing (42 images) to understand and compare how the detector works on AccessDB and more high-resolution images in AccessReal. For the ‘unidentifiable’ class, we still included it as an individual class in training but did not use it for evaluation. This is because, ‘unidentifiable’ objects are still in the 6 categories of our interests, so those might have overlapping visual features with other inaccessibility classes that the human eye could not capture due to the blurry images. By treating it as one class in training a detector, we can avoid unwanted penalizing of the other classes’ correct predictions.

Evaluation of the detector on validation and test sets was performed per each epoch. The detector achieved its best performance for the AccessReal dataset after around 51 epochs, yielding an mAP of 18.85 for the validation set and 14.86 for the test set. Additional performance metrics are provided in Table 5. AccessDB validation set showed the best mAP (19.86) at epoch 61, but after 51 epochs the detector started overfitting to AccessDB, resulting in the lower mAP (13.36) for AccessReal. Even though AccessDB and AccessReal both contain real-world indoor images, we could still see the domain gap between the two as the detector shows about 4 less mAP. We attribute this performance difference, in part, to the significantly higher resolution of images in AccessReal, which poses a challenge for a detector primarily trained on smaller images. Furthermore, AccessDB inherently exhibits a long-tailed distribution in terms of class counts (Detailed breakdown of the number of classes is described in Table 3). This distribution presents an additional challenge to the detector, particularly when recognizing classes with a relatively small number of objects, which may not provide sufficient data for the model to learn distinctive visual features. Despite the challenges, visual results created by our detector (Figure 13) showcase its ability to perform well on high-resolution indoor images. In the zoomed regions of Figure 13 (second and fourth images), results show that the detector successfully recognized our interested objects, including knob_rotate_round, faucet_handle_lever, electric_outlet, and handle_bar_small. Table 4 provides a breakdown of mAP for each inaccessibility class. The average mAP indicates that, as a whole, the detector performs better on AccessDB compared to AccessReal. However, it’s worth noting that the detector exhibits superior performance on AccessReal for certain classes, such as switch_toggle_multi, switch_toggle_single, and handle_lever. We hypothesize that for these classes, AccessReal may offer clearer object representations or exhibit fewer visual variations, possibly due to its smaller sample size, thereby contributing to improved detection accuracy.