Improving Accessibility with Computer Vision

Introduction

For individuals with visual impairments, navigating both physical and virtual environments can present significant challenges. Traditional assistive technologies, such as white canes and screen readers, have been invaluable, but they have limitations in providing comprehensive information about the environment and often require specialized hardware or extensive training.

At Cyrion Labs, we've been working on addressing these challenges through our VIZ (Virtual & Physical Navigation System for the Visually Impaired) project. VIZ uses advanced computer vision techniques to help visually impaired individuals navigate both physical and virtual environments with greater confidence and independence. In this blog post, I'll share how VIZ works and the impact it's having on users' lives.

The Challenge of Navigation for the Visually Impaired

Navigation involves perceiving and understanding the environment, identifying obstacles and pathways, and making decisions about how to move through the space. For individuals with visual impairments, these tasks can be challenging without access to visual information.

In physical environments, challenges include:

• Identifying obstacles and hazards in the path
• Finding specific locations or landmarks
• Reading signs, labels, and other text in the environment
• Navigating complex or unfamiliar spaces

In virtual environments, such as websites and applications, challenges include:

• Understanding the layout and structure of the interface
• Identifying interactive elements
• Accessing visual content such as images and charts
• Navigating between different sections or pages

How VIZ Works

VIZ addresses these challenges through a comprehensive navigation system that works in both physical and virtual environments. The system is built on our PhysNav-DG framework, which uses a novel adaptive approach for robust vision-language model and sensor fusion.

Physical Navigation

For physical navigation, VIZ uses the smartphone camera to perceive and understand the environment. The system:

• Detects obstacles and hazards in the path, providing real-time audio feedback to the user
• Recognizes landmarks, signs, and other environmental features to help with orientation and wayfinding
• Reads text in the environment, such as signs, labels, and menus
• Provides turn-by-turn directions to specific destinations

The system is designed to work reliably in challenging conditions, such as low light or crowded environments, thanks to the robust sensor fusion approach of PhysNav-DG.

Virtual Navigation

For virtual navigation, VIZ integrates with screen readers and other assistive technologies to provide enhanced navigation capabilities. The system:

• Analyzes the structure and layout of web pages and applications
• Identifies and describes interactive elements, such as buttons, links, and forms
• Provides descriptions of visual content, such as images and charts
• Helps users navigate between different sections or pages more efficiently

By leveraging semantic web technologies for automated UI element annotation, VIZ can provide more detailed and accurate information about virtual environments than traditional screen readers.

Impact on Users' Lives

VIZ is currently being tested with 50+ users across 3 cities, and the early results are promising. Users report a 40% improvement in navigation confidence, feeling more independent and able to explore new environments with greater ease.

The system has been particularly effective in helping users navigate complex environments such as shopping malls, airports, and public transportation systems. In virtual environments, VIZ has helped users navigate websites and applications more efficiently, with a 30% reduction in the time needed to complete common tasks.

Here's what some of our users have to say:

"VIZ has completely changed how I navigate the world. I can now go to new places with confidence, knowing that I have reliable assistance when I need it." - Maria S., VIZ User

"As someone who works with visually impaired individuals daily, I've seen firsthand how VIZ has improved their independence and quality of life. It's a game-changer." - Dr. James T., Vision Rehabilitation Specialist

Future Directions

While we're excited about the progress we've made with VIZ, we're continuing to improve the system based on user feedback and advances in computer vision technology. Future developments include:

• Integration with smart glasses and other wearable devices for a more seamless experience
• Enhanced object recognition capabilities for specific domains such as grocery shopping or public transportation
• Improved navigation in complex indoor environments through the use of 3D mapping and spatial audio
• Collaboration with website and application developers to improve virtual navigation capabilities
• Expansion to additional cities and user groups

Conclusion

Computer vision has the potential to significantly improve accessibility for individuals with visual impairments. Through projects like VIZ, we're working to realize this potential and create more inclusive physical and virtual environments.

We believe that by combining advanced computer vision techniques with user-centered design and continuous feedback, we can create assistive technologies that truly meet the needs of visually impaired individuals and enhance their independence and quality of life.

For more information about VIZ, check out our paper VIZ: Virtual & Physical Navigation System for the Visually Impaired, which will be presented at CVPR 2025 Demo.