Tag: AAC

Posted on

Updating EmpTech

The history of EmpTech goes back a long way and the team has dispersed to take up different jobs. However, we are keeping this site because it has happy memories linked to Assistive Technology and we still have the same aims in life about supporting technology that works for everyone.

Welcome Communication and At

One project involves the development of Global Symbols. It has a repository of professionally designed communication symbols for education, healthcare, and accessibility. The number of symbol collections is growing as you can design new symbols with AI-powered tools that adapt to your specific needs and cultural contexts. It is also possible to edit them using Symbol Designer and create custom communication charts using Board Builder.

board with Kenyan and Pakistan symbols

Content offered on the Global Symbols Moodle site is free and has a Creative Commons Licence. In order to access the training resources and activities please create an account.

Unicef Logo Banner

Developed in Partnership with UNICEF

Courses based on themes

AAC Training (6) – Trening o PK na hrvatskom (8) – AAK Trening Srbija (8)

Courses based on prior skill / knowledge levels

AAC Guided Pathways (3) – Пътеводител на ДАК (5) – Патоказ кон ААК (3)

Extra Resources and Instructions

AAC Resources & Board Builder (1) – Symbol Voting Instructions (en, hr, sr) (3)

Churchill Fellowship Logo Banner

Supported by a Churchill Fellowship COVID-19 Action Fund

Communication using Symbols – A Carer’s Training Pack (1)

One module about Communication using Symbols with five steps that can be dipped into at any stage and need no prior knowledge. They include:

  • How we communicate – Managing conversations – Finding the right words and images – Adapting images and symbols for each individual – Creating opportunities to encourage conversations.

There are some activities, case studies and examples of communication boards and information charts with most steps and the entire Carer’s Training Pack can be downloaded

Posted on

Learning more about Generative AI and AAC symbols

The complexities of creating symbols for communication and the way they work to support spoken and written language has never been easy. Ideas around guessability or iconicity and transparency to aid learning or remembering are jut one side of the coin in terms of design. There are also the questions around style, size, type of outlines and colour amongst many other design issues that need to be carefully considered and the entire schema or set of rules that exist for a particular AAC symbol set. These are aspects that are rarely discussed in detail other than by those developing the images.

However, when trying to work with computer algorithms to make adaptations from one image to another a starting point can be image to text recognition in order to discover how well chosen training data is going to work. It is possible to see if the systems can deal with the lack of background and other details that normally help to give images context, but are often lacking in AAC symbol sets. The computer has no way of knowing whether an animal is a wolf or dog unless there are additional elements, such as a collar or a wild natural area around the animal such as a forest compared to a room in a house. If it is possible to provide a form of alternative text as a visual description, not disimilar to that used by screen reader users when viewing images on web pages, the training data provided may then work for an image to image situation.

There remains the need to gather enough data to allow the AI systems to try to predict what it is you want. The systems used by Stable Diffusion and DALL-E 2 have scraped the web for masses of images in various styles, but they do not seem to have picked up on AAC symbol sets! There is also the case that each symbol topic category within the symbol set tends to have different styles even though the outlines and some colours may be similar and humans are generally able to recognise similarities within a symbol set that cannot necessarily be captured by the AI model that has been developed. More tweaks will always be needed along with more data training as the outcomes are evaluated.

Comparison of symbol sets

The image above compares groups of symbols from the ARASAAC, Mulberry, Sclera and Blissymbolics sets.

The other problem is that most generative artificial intelligence (AI) systems using something like Stable Diffusion and DALL-E 2 are designed to provide unique images in a chosen style, even when you enter the same text prompt. Therefore each outcome will look different to your first or second attempt. In other words there is very little consistency in how the details of the picture may be put together other than the overview will look as if it has a certain style. So if you put in the text prompt edit box that you want “A female teacher in front of a white board with a maths equation”, the system can generate as many images as you want, but none will be exactly the same.

A female teacher in front of a white board with a math equation

Created using DALL-E 2

Nevertheless, Chaohai Ding has managed to create examples of AI generated Mulberry AAC symbols by using Stable Diffusion with the addition of Dreambooth that uses a minimal number of images in a more consistent style. There are still multiple options available from the same text prompt, but the ‘look and feel’ of those automatically generated images makes us want to go on working with these ideas in order to support the idea of personalised AAC symbol adaptations.

racing driver friend and astronaut

In the style of the professions category in the Mulberry Symbol set these three images had the text prompt of racing driver, friend and astronaut.

We would like to thank Steve Lee for allowing us to use the Mulberry Symbol set on Global Symbols and the University of Southampton Web Science Institute Stimulus Fund for giving us the chance to collaborate on this project with Professor Mike Wald’s team.

Posted on

AI for auto-translations; different languages for symbols

Over the last couple of months we have been testing the different AI automatic translation offerings to try and work out if we can translate symbol labels, with a chance to edit them online when they don’t make sense! This has been work related to an Augmentative and Alternative Communication (AAC) symbol repository – Global Symbols

Participants on the site who are registered AAC symbol developers can use Microsoft Azure’s cognitive translation services, but this does not work for all the languages we need.

Translation English to Dutch symbol labels

Microsoft Azure has 80 languages, but sadly not Macedonian or Montenegrin. This also means that when we use Weblate, an open sourse system using Microsoft, for menus and navigational elements on the website, there is also a problem. However, having tested the system with manual checks of other languages that we needed, Micorsoft appeared to provide a broadly satisfactory outcome.

When using Moodle we have found their Automated Manipulation Of Strings (AMOS) translation system can be used alongside the Google translation API which has the Macedonian language! Amazon also has Macedonian, but only 71 languages compared to 100.

Cost and the type of translation service required obviously affects choice and in our case we have been incredibly lucky to usually want a translation to go from English to another language, but in some cases it is important to have the option to reverse the situation as we have with one symbol set, where we need to go from Turkish to English. In this case Wikipedia offer a helpful chart, but do check the particular company sites, as they suggest.

Brewing tea!
Posted on

Working on Symbols and Concept Linking

View WebSci 2020 Presentation in a new tab

The WebSci 2020 virtual conference has a special theme on Digital (In)Equality, Digital Inclusion, Digital Humanism the first day of this virtual conference. This will gave us the chance to show the initial findings from our linking of freely available Augmentative and Alternative Communication (AAC) symbol sets to support understanding of web content.

There are no standards in the way graphical AAC symbol sets are designed or collated other than the Blissymbolics ideographic set that was “standardized as ISO-IR 169 a double-byte character set in 1993 including 2384 fixed characters whereas the BCI Unicode proposal suggests 886 characters that then can be combined.” Edutech Wiki.

Even Emojis have a Unicode ID, but the pictographic symbols most frequently used by those with complex communication needs do not have an international encoding standard. This means that if you search for different symbols amongst a collection of freely available and open licenced symbols sets you find several symbols have no relationship with the word you entered or the concept required.

symbols for up
Global Symbols used to show sample symbols when the word ‘up’ was entered in the search.

This lack of concept accuracy means that much work has to be done to enable useful automatic text to symbol support for web content. Initially there needs to be a process to support text simplification or perhaps text summarisation in some cases. Then keywords need to be represented by a particular symbol (from a symbol set recognised by the reader), that can be accurately related to the concept by their ISO or Unicode ID. Examples can be found in the WCAG Personalization task force Requirements for Personalization Semantics using the Blissymbolics IDs.

The presentation at the beginning of this blog will illustrate the work that has been achieved to date, but it is hoped that more can be written up in the coming months. The aim is to have improved image recognition to assist with the semantic relatedness. This automatic linking will then be used to map to Blissymbolics IDs. It is hoped that this will also enable multilingual mapping, where symbol sets already have label or gloss translations.

laptop coding

However, there still needs to be a process that ensures whenever symbol sets are updated the mapping can continue to be accurate as some symbol sets do not come with APIs! That will be another challenge.

Posted on

COVID-19, AI and our Conferences

conference seating

Much has changed for everyone since our last blog. Swami Sivasubramanian, VP of Amazon Machine Learning, AWS has written an article about the way AI and machine learning have been helping to fight COVID-19 and we can see how varied the use of this technology has been. However, we remain in a world that is having to come to terms with many different ways of working and travelling to conferences has been off the agenda for the last few months.

We have continued to work on topics covered in our papers for ICCHP that will delivered remotely, as will the one we submitted for WebSci 2020 . ISAAC 2020 has been moved to 2021, but who knows if we will get to Mexico but hopefully at least we may have some results from the linking of concepts for several free and open augmentative and alternative communication symbol sets.

As the months pass much of our work will be seen on Global Symbols with examples of how we will be using the linked symbol sets.

We are also trying to support the WCAG personalization task force in their “Requirements for Personalization Semantics” to automatically link concepts to increase understanding of web content for those who use AAC or have literacy difficulties and/or cognitive impairments.

mapping symbol sets
The future for freely available mapped sample AAC symbol sets to illustrate multilingual linking of concepts from simplified web content.
Posted on

AI and Inclusion projects related to Web Accessibility and AAC support.

Over the last few months we have been concentrating on projects related to automated web accessibility checks and the automatic  linking and categorisation of open licenced and freely available Augmentative and Alternative Communication symbol sets for those with complex communication needs.

As has been mentioned we presented these projects at a workshop in the Alan Turing Institute in November and work has been ongoing. It is hoped that the results will be shared by the end of March 2020.

Automating Web Accessibility Checks

Recent regulations and UK laws recognise the W3C Web Content Accessibility Guidelines (WCAG) as a method of ensuring compliance, but testing can be laborious and those checkers that automate the process need to be able to find where more errors are occurring.  This has led to the development of an accessibility checker that carries out well-known automated checks, but also includes image recognition to make it possible to see if the alternative text tags for images are appropriate. A second AI related check involves a new WCAG 2.1 Success Criteria 2.4.4 Link Purpose (In Context).  This is where “the purpose of each link can be determined from the link text alone or from the link text together with its programmatically determined link context, except where the purpose of the link would be ambiguous to users in general”.[1] 

A Natural Language Processing (NLP) model is used to check whether the text in the aria-label attribute within the target hyperlink object matches the content in the target URL. Based on the matching result, it is possible to determine whether the target web page or website fit the link purpose criteria. Despite previous research in this area, the task is proving challenging with two different experiments being worked on. One experiment has been designed to use some existing NLP models (e.g. GloVe), while another one is investigating the training of data with human input. The results will be published in an academic paper and at a conference.

AAC symbol classification to aid searches.

Global Symbols with a Cboard user

The team have also investigated issues for those supporting Augmentative and Alternative Communication (AAC) users who may have severe communication difficulties and make use of symbols and pictures on speech generating devices. A multilingual symbol repository for families, carers and professionals has been created to link different freely available symbol sets.  The symbol sets can be used to create communication charts for the AAC user but this takes time and finding appropriate cultural symbols is not always easy.  A system has been developed that automatically links and categorises symbols across symbol sets related to their parts of speech, topic and language using a combination of linked data, natural language processing and image recognition.  The latter is not always successful in isolation as symbols lack context and concepts are not necessarily concrete such as an image for ‘anxious’, so further work is required to enhance the system.  The Global Symbols AAC symbol repository will be making use of these features on their BoardBuilder for making symbol charts by the end of March 2020.

This project is exploring some existing Convolutional Neural Network (CNN, or ConvNet) models to help classify, categorise and integrate AAC symbols. Experiments have already been undertaken to produce a baseline by simply using the image matrix similarity. Due to the nature of AAC symbols, some of these similar symbols are representing different concepts, but some different symbols are representing the same concept across different symbols sets. The training data set has mapped symbol images labels and NLP models have been used to map the labels into the same concept across different symbols. This will help those supporting ACC users offer much wider symbol choices suitable for different cultures and languages. The Global Symbols API for searching open licence and freely available AAC symbols is already being used in the Cboard application for AAC users

[1] https://www.w3.org/WAI/WCAG21/Understanding/link-purpose-in-context.html