jordansangalang.com

Tag: sign language AI

  • GPT-6 Astra is an impressive leap—and it still leaves Deaf signers behind.

    GPT-6 Astra is an impressive leap—and it still leaves Deaf signers behind.

    OpenAI just released GPT-6 Astra, and after watching the announcement video, the jump is real. It’s being called a generational leap: stronger computer use, better multi-step professional workflows, higher accuracy with fewer tokens in many cases, and clear gains on hard benchmarks like ARC-AGI-3 and ExploitBench. The model feels more capable at sustained, end-to-end work than what came before it. That part is exciting.

    But one limitation stands out immediately. Astra remains built around spoken and written language. Input and output still center on text and speech-adjacent interfaces. For the many Deaf people who use sign languages as their primary language, that creates a real barrier. We already use AI every day for work, learning, communication, and creativity. Leaving sign language out of the core interaction model means we stay on the outside of the most advanced systems.

    The technology to close that gap already exists.

    Google recently launched SL2T (sign-language-to-text) with the Pixel 11. It translates American Sign Language into written English in real time by analyzing handshapes, facial expressions, and body movement. The feature works inside Gboard and Live Transcribe so signers can search, message, draft, or talk to Gemini without switching to typing. Training data covered more than 100,000 hours across dozens of sign languages, and the system was developed with Deaf community input. It’s not perfect and currently limited to Pixel 11 and ASL-to-English, but it proves the direction is viable.

    SignCaption.ai has been building in this space as well. It is available on iOS, Android, and the web, so it works across phones, computers, and other devices. The platform handles automatic captioning of signed video, conversion between signed and spoken content, and related accessibility tools. Because it is web-based and mobile-ready, it already reaches more people without requiring a specific new handset.

    These systems show that real-time, usable sign-language interfaces are no longer speculative. They are shipping.

    What OpenAI should do next

    GPT-6 Astra should incorporate native sign-language support. That means:

    • Accepting video or camera input of signing as a first-class modality alongside text and images.
    • Producing signed output (via avatar or video) when the user prefers it, not only text or speech.
    • Supporting major sign languages with the same seriousness given to spoken languages, starting with ASL and expanding.
    • Designing the experience so Deaf users do not have to translate themselves into the hearing world just to use the most capable model.

    Accessibility here is not a side feature or a checkbox. Sign languages are complete natural languages. Treating them as optional add-ons after the fact keeps an entire community of AI users one step behind. The people building and using these tools daily—Deaf professionals, students, creators, parents—deserve direct access, not workarounds.

    Astra is a genuine milestone in capability and alignment. Making it fully usable by signers would make it a milestone in inclusion as well. The pieces are already on the table. The only question is whether the next version of the most advanced models will finally treat signed languages as equal.

    Here’s the video for those who haven’t seen Astra in action yet.

  • Google DeepMind Launches SL2T: Sign Language AI Now on Pixel 11 Phones

    Google DeepMind Launches SL2T: Sign Language AI Now on Pixel 11 Phones

    Google DeepMind just made one of the most meaningful accessibility advances in AI this year. On August 12, 2026, the company unveiled SL2T (Sign-Language-to-Text), a breakthrough model that finally brings high-quality sign language recognition out of the research lab and onto everyday smartphones.

    Starting with the Pixel 11, SL2T powers a new “sign-to-text” feature in Gboard and Live Transcribe. For the first time, Deaf and hard-of-hearing users can sign in American Sign Language (ASL) directly to their phone camera instead of typing. The phone turns those signs into English text in real time.

    What SL2T Actually Lets You Do

    Anywhere you would normally type on your Pixel 11, you can now sign:

    • Search the web
    • Draft messages, emails, or documents
    • Ask Gemini questions or give it tasks
    • Respond in face-to-face conversations using Live Transcribe

    Early testers told Google that signing in ASL felt faster and more natural than typing English. That alone is a big deal for people who use sign language as their primary language.

    How the Technology Works

    SL2T is deliberately designed with privacy and real-world usability in mind.

    An on-device model called MediaPipe Holistic tracks roughly 130 key points on the signer’s face, hands, arms, and torso. Only those geometric coordinates are sent to the cloud for translation. The original video is discarded immediately and never leaves the device.

    Unlike earlier systems that relied on intermediate “glosses” (fixed labels for individual signs), SL2T translates the sequence of pose landmarks directly into fluent English text. This approach better captures the simultaneous hand, body, and facial movements that give sign languages their full grammatical structure.

    The model was trained on more than 100,000 hours of signing data across over 50 sign languages, with about a quarter of that data in ASL. Training across many languages helped it learn shared patterns and improved performance.

    Performance and Real-World Readiness

    On the challenging FLEURS-ASL benchmark, SL2T scored 70 BLEURT in a zero-shot setting — significantly higher than previously reported results. It also performs well on one-handed signing (important when you’re holding a phone) and reaches 64% exact-match accuracy on mobile fingerspelling data.

    Google specifically optimized for practical issues: lower latency, reduced hallucination when no one is signing, support for left-handed signers, and one-handed use.

    Built With the Deaf Community, Not Just for It

    The project began with Sam Sepah, a Deaf Googler, and Deaf perspectives shaped data collection, testing, and evaluation throughout development.

    Google also established the AI Sign Language Advisory Committee (AISLAC), which includes major organizations such as the National Association of the Deaf and the World Federation of the Deaf. Together they published a joint impact report that clearly defines what SL2T 1.0 is — and is not — intended for.

    The report positions the current version as an assistive tool for informal, low-stakes situations such as messaging, searching, note-taking, and casual conversation. It explicitly states that SL2T should not be used as a substitute for certified interpreters in medical, legal, educational, or official settings, and it does not fulfill legal requirements for reasonable accommodations.

    What’s Coming Next

    ASL-to-English on Pixel 11 is only the starting point. Google plans to expand support to additional sign languages, improve handling of non-manual markers and multi-signer scenarios, add features like punctuation and editing commands, and eventually explore sign language generation.

    The longer-term goal is full parity: making sign languages as accessible and useful across digital products as spoken and written languages already are.

    Why This Matters

    An estimated 70 million people worldwide are Deaf or hard of hearing, and there are more than 200 distinct sign languages. While speech recognition and translation have advanced rapidly, sign language AI has lagged far behind — partly because of technical complexity and partly because of longstanding misconceptions about how sign languages work.

    SL2T does not solve every challenge overnight. Accuracy still drops in poor lighting, extreme camera angles, or with rapid fingerspelling of rare words. But for the first time, a capable sign-language-to-text system is available as a free, everyday feature on a mainstream smartphone.

    That is genuine progress. And for many users, it means their primary language can finally work with the devices they already carry every day.

    SL2T is available now on Pixel 11 in Gboard and Live Transcribe, with broader device support planned. No extra cost.

  • Day 18: Deaf Accessibility Wins – ASL Avatars & AI Translation Tools on the Web

    In the last post, I talked about the rise of ASL in performance, storytelling, and mainstream media. Now let’s shift to the accessibility lens — how ASL is appearing on websites and digital platforms.

    There are growing AI tools that translate text into ASL using avatars or real human interpreters. A few notable ones include:

    • Signapse.ai
    • Signly.co
    • GoPivot.me

    As far as I know, these are not Canadian-based or Canadian-led, and they don’t always use Canadian interpreters. However, they show what’s possible and could be adapted or built upon to enhance access here in Canada without reinventing the wheel.

    Ideally, we’d see ASL/LSQ translations become the standard on websites — seamlessly integrated so users don’t have to click away to a separate page or switch back and forth. Right now, this is still not widespread or the norm.

    With AI tools and technology readily available, making this the standard would be a significant accessibility win for Deaf Canadians.

    Where would you like to see ASL/LSQ translations implemented? Whether it’s government websites, e-commerce, banking, healthcare portals, education platforms, or something else — I’d love to hear your ideas. These conversations can lead to real partnerships and collaborations to push for better web access together.

    What about you? Have you come across websites or apps that do a great job integrating ASL or sign language translation? Or what would seamless web accessibility look like for you?

    Share your examples, ideas, or wishes in the comments!

    See you tomorrow for Day 19!