March 6, 2024
In March 2024, CREATE students gathered with industry and community partners for a hackfest to explore and invent the future of AI and Accessibility.
The event featured invited speakers Heather Nolis, Ian Stenseng, and Shaun Kane and exciting workshops on building custom GPT and creating accessible Jupyter notebooks. Attendees brainstormed on ideas submitted by community members and formed teams to focus on specific needs. See the full lineup of brainstorming, hacking, and presentation sessions.
“Thanks for hosting a fantastic hackathon!”
After the event, one team wrote to CREATE Director Jennifer Mankoff, “to express our sincere gratitude for hosting a fantastic hackathon! The event was well-organized, provided a stimulating environment, and fostered a spirit of collaboration and innovation. We are incredibly grateful for the opportunity to showcase our skills and contribute to solving accessibility challenges using AI.”
A hearty congratulations to the organizers, including: CREATE Ph.D. students Kate Glazko, Jerry Cao, and Venkatesh Potluri; Tony Fast, an open-source designer working on CREATE research projects; and CREATE leadership and professional staff.
Attendees of the 3-day hackfest included those with no experience in coding or hacking, others with advanced experience in generative AI and building software or tools, and, at the center, attendees with lived experiences of disabilities who contributed their experiences and expertise to invent an accessible AI-enabled future.
Prizes awarded
While appreciation and congratulations go to all participants, these projects were awarded prizes:
First place ($1200): LookLoud.ai
Nishit Bhasin and Lakshya Garg
LookLoud.ai is voice-activated assistance technology, powered by GPT-4 Vision, and designed to make e-commerce accessible to everyone. Users can navigate, select, and buy products using simple voice commands.
The tool is based on Lolo, an AI assistant designed to make e-commerce accessible to everyone. Users can navigate, select, and buy products using voice commands for hands-free navigation. Lolo acts as the user’s mouse and keyboard, browsing and selecting items. For instance, a customer might say, “Lolo, find me an Apple iPhone 15 Pro (256 GB) rated at least 3.5 stars within a $1200 budget.” Lolo will then search, filter, and guide the user through the buying process using voice commands and auto-navigation.
Seamlessly integrating with current e-commerce platforms, LookLoud.ai eliminates traditional barriers, offering a shopping experience that is not only accessible but also enjoyable for individuals with disabilities. This technology can understand intentions, context, and user preferences, ensuring inclusivity is not just a buzzword but a tangible customer experience.
The LookLoud.ai project is now a startup called IncSkill and has evolved into a comprehensive AI-powered accessibility platform that helps enterprises enhance digital accessibility, automate compliance, and build inclusive experiences for employees and customers with disabilities. Congratulations to team members Nishit Bhasin and Lakshya Garg, who were honored with a 2025 Forbes 30 Under 30 for Social Impact award!
Second prize ($500): AI Posture Monitor & Intervention Alerts for Home Health
Max Smoot, Lige Yang, and Richard Li
The posture monitor is an AI tool that can monitor a person’s seated position, identify when they are in a posture that could cause injury or worsen an existing condition, and alert a caretaker with recommended corrections. The project was designed with input from the parent of a 2-year-old child with severe motor delays due to cerebral palsy and who requires continuous monitoring and intervention from a caretaker. Her needs exceed what can be supported by medical equipment and assistive technology alone.
We successfully demonstrated the full scope of:
- Monitoring an appropriately placed webcam using a computer vision-based pose detection model to detect compromised postured
- Relaying an image of an at-risk posture to a custom GPT for false positive screening and intervention recommendation
- Sending SMS alerts with the GPT recommendations to the caretaker.
We developed a Figma prototype to make this capability accessible to a broader audience of caretakers. With timely response from a caretaker, this monitoring system minimizes the time that an at-risk candidate spends in a compromised posture. Future project expansion might include tracking accumulated time in a compromised posture, monitoring relevant asymmetry, tracking trends over time, and providing feedback on the risk of the candidate’s initial seating position.
Third prize (three Alexas): Formflow Ai
Abdul Hussein, Abreham Tegenge, and Aelaph Elias
Formflow.ai reads PDFs, mail, and forms and gives an easy-to-read summarization, with the goal of helping people read and understand documents and forms.
The application uses AI to read PDFs, mail, and forms and gives an easy-to-read summary. To further enhance the accessibility, a text-to-speech feature can translate into many different languages. The web page also has the ability to zoom in and out to help with devices that might not have this capability. Our future goal is to turn this in to a mobile app so that users are able to take a picture of a document for even easier summarization.
Fourth place (swag): Clearview Assist
Dhruv Khanna, Ritika Rajpal, Minal Naik, and Menita Agarwal
ClearView Assist is a Chrome extension designed to bridge the digital accessibility gap experienced by users with low vision or blindness when navigating the web.
The tool simplifies cluttered web pages based on user tasks, allowing individuals to interact with digital content more efficiently and significantly reducing the time and effort typically required by screen readers to locate specific details. Through voice commands, users can articulate their browsing objectives. ClearView Assist dynamically filters out irrelevant page elements while highlighting and prioritizing content crucial to the task at hand. Powered by Azure Open AI, the extension intelligently parses web page structures to identify and present pertinent information. ClearView Assist ensures that vital information is not overlooked, optimizing the browsing experience for users with visual impairments through a streamlined, user-centric interface accessible via simple shortcuts.
Fifth place (swag): Student Success Portal
Mia Vong, Cameron Jacob Miller, Keyvyn Rogers, and Jerid Stevenot
The Student Success Portal (SSP) tackles challenges in supporting students with IEPs (K-12). Rising IEP numbers and teacher shortages create a gap where students fall behind and educators struggle. Existing systems lack flexibility for accommodations and efficient progress tracking, leading to burnout and litigation.
SSP bridges this gap with AI-powered assistance. It analyzes assignments, personalizes accommodations, and tracks progress against IEP goals. Real-time feedback and anti-bias training empower teachers. This not only reduces costs and paperwork, but also fosters student success and accessibility.
Future iterations aim for enhanced collaboration — with parents, students, and experts — along with data-driven goal setting, progress notifications, and streamlined resource referrals. With SSP, educators can effectively implement IEPs, ensuring every student receives the support they deserve.
Sessions, workshops and hack time
Friday
- Introductory session about the potential of AI for accessibility (also on Zoom)
- Invited speaker Ian Stenseng, Director of Innovation & Accessibility at The Lighthouse for the Blind, Inc. (also on Zoom)
- Brainstorming project ideas
- Learn from community members with lived experiences of disabilities to make sure your hack is solving a real accessibility need.
- Lunch (provided) and conversation, mentoring, team forming, idea hatching
- Invited speaker Heather Nolis, Principal Machine Learning Engineer of the Digital AI Team and Chair of the Accessibility Community at T-Mobile (ACT) at T-Mobile (also on Zoom)
- Optional Workshops and hack time
- Hack time
- Pizza dinner and opportunities to get feedback from mentors
Saturday
- Work time
- Lunch (provided) and opportunity to present for feedback from mentors
- Presentation of judging rubric
- Invited speaker, Shaun Kane, Researcher at Google AI and Director of the Superhuman Computing Lab at University of Colorado Boulder (also on Zoom)
- Hack time
Sunday
- Optional hack time
- How to present accessibly & sample pitch presentation (also on Zoom)
Monday
- Presentations to judges (also on Zoom)
- Judges deliberation
- Announcements, prizes, and closing keynote (also on Zoom)
Judges
- Laura Dorsey, Director of Licensing, Copyright & Software, UW CoMotion
- Richard Ladner, Professor Emeritus, University of Washington
- Devin Myers, Client Advocate, Hearing, Speech and Deaf Center
- Anisa Proda, Accessibility and Inclusion Manager at T-Mobile through PK Global, T-Mobile
Speakers
Ian Stenseng, Director of Innovation & Accessibility, The Lighthouse for the Blind, Inc.
Ian Stenseng, Director of Innovation & Accessibility, The Lighthouse for the Blind, Inc.
Ian Stenseng is the Director of Innovation & Accessibility for The Lighthouse for the Blind, Inc.
For the past twelve years, Ian has been a pivotal member of the Lighthouse for the Blind, Inc., where he has used his expertise in technology and creativity to enhance accessibility. He has focused on making tools, equipment, technology, and the built environment more accessible for individuals who are blind, DeafBlind, or who have additional disabilities. Ian's commitment to this cause is deeply personal, stemming from his familial connection to blindness; his father, who was born prematurely, is legally blind due to retinopathy of prematurity.
Before his tenure at the Lighthouse, Ian's career was marked by significant achievements in the IT sector, particularly with the Washington Office of the Secretary of State. There, he spearheaded special projects for the WA State Archives, working on advanced systems for the imaging, preservation, and access of historical materials. He played a key role in modernizing recording studios, establishing the first accessible computer lab at the Washington Talking Book and Braille Library, and contributing to the design and rollout of the Washington State Domestic Partnership program.
Ian resides in South Seattle with his partner, Jaime, and their twelve-year-old son, Mason. Outside of his professional life, Ian is deeply passionate about craftsmanship and loves to tinker with all things mechanical, recently building a cedar tiny home trailer in his driveway, a large format laser engraver in his home studio, and dedicating his free time to crafting fine handmade goods from leather and wood.
Heather Nolis, Principal Machine Learning Engineer of the Digital AI Team and Chair of the Accessibility Community at T-Mobile (ACT) at T-Mobile
Heather Nolis is a founding member of the AI @ T-Mobile team, who focused the conversion of cutting-edge analyses to real-time, scalable data-driven products. She began her career in neuroscience but once she realized how heavily that field relied on software built by other people and data analyzed by other people, she pivoted - deciding to make software herself. In addition to her role as Principal Machine Learning Engineer, she is also the chair of T-Mobile’s accessibility Employee Resource Group which has over 12,000 members.
Workshops
Accessible Notebooks Workshop
Though computational notebook platforms such as Jupyter are heavily used in many settings including classrooms, research, and storytelling, they are often not accessible to people with disabilities. This inaccessibility has the potential to exclude people with disabilities from educational, employment, and other information-seeking opportunities. Join us for a two-hour workshop to learn about the inaccessibility of notebooks and gain skills to author accessible notebooks. This workshop does not require or expect expertise in programming or accessibility to participate. All you need is a computer and the curiosity to understand what notebook accessibility is all about.
About the organizers
Tonyfast is a freelance developer, designer, and scientist with significant experience in open source and science software. They are a distinguished project jupyter contributor advocating for equity in computational literacy and digital accessibility in open science technologies.
Venkatesh Potluri is a graduating Ph.D. Candidate at the University of Washington. He investigates the inaccessibility of developer tools for blind and visually impaired developers participating in professional programming domains like data science and user interface design. He builds real-world systems and demonstrates new ideas to improve the accessibility of widely used developer tools.
Intro to GAI, GPTs, and More!
Welcome to "Intro to GAI, GPTs, and More!", an interactive workshop designed for beginners and enthusiasts eager to dive into the world of Generative Pre-trained Transformers (GPTs). We will explore ways to build Generative AI (GAI) tools at a range of coding skill levels, ranging from no-code GPTs to a quick introduction of the OpenAI API. As the demand for AI and machine learning solutions continues to grow, the ability to leverage these technologies becomes increasingly valuable. This workshop aims to demystify GPT/GAI and make it approachable through a practical, no-code approach.
Exploring GPTs Without Coding: Learn about no-code platforms and tools that allow you to interact with GPT models. Discover how to use these tools for a variety of applications, including content generation, data analysis, and creative projects, without writing a single line of code.
Hands-On Demonstrations of OpenAI API: Follow along to an introductory lesson on how to get started with the OpenAI API.
About the organizer
Kate Glazko is a first year Ph.D. student at the University of Washington. Kate is passionate about AI and accessibility and learning about all the ways in which emerging forms of AI like GPT, Midjourney, and more can help or hinder access in areas such as employment, creativity, and making. Prior to UW, Kate spent 6 years in the tech industry working in roles such as android engineer, IoT engineer, and product manager.
Brainstorming ideas
Relevant topics will be driven by community needs to increase access to technology, and to the world through technology. These topics could include, for example:
- AI’s use for generating plain language summaries of rights
- Accessibility of AI tools and interfaces
- Using AI to increase the accessibility of written and visual content
- Robotic control for access
- Tools for designing accessible physical objects
- Using AI to get feedback on the accessibility of things you’re making
- AI for embodied agent interactions
- AI applications for health and wellbeing
- Modalities for human/generative AI interactions such as voice or touch
- Guidelines or ideas around agents that may be used for accessibility
- What disability simulation might look like in the age of AI agents
- Best practices and pitfalls