Stepping into the world of AI, Icon Recognition, Human Perception and AAC Symbols.

In the last few months we have been working on our Artificial Intelligence (AI) and AAC symbol project, finding how inconsistent the pictographic images may appear in some AAC symbol sets and the impact this has on the various stages of image processing such as perception, detection and recognition.

We have been researching how inconsistency can hamper automated image recognition after pre-processing and feature abstraction, but the advent of Stable Diffusion as a deep learning model allows us to include visual image text descriptions alongside image to image recognition processes to support our ideas of symbol to symbol recognition and creation.

Stable Diffusion

Stable Diffusion – “The input image on the left can produce several new images (on the right). This new model can be used for structure-preserving image-to-image and shape-conditional image synthesis.” https://stability.ai/blog/stable-diffusion-v2-release

Furthermore, with the help of Professor Irene Reppa and her project team researching “The Development of an Accessible, Diverse and Inclusive Digital Visual Language” we have discovered many overlaps in the work they are doing with icon standardisation. Working together we may be able to adapt our original voting criteria to provide a more granular approach to ensuring automatically generated AAC symbols in the style of a particular symbol set allow for ‘guessability’ (transparency) and ease of learning whilst also making them appealing based on a much more inclusive set of criteria. The latter have been used by many more evaluators over the last 8 years as is mentioned in this blog “When the going gets tough the beautiful get going.”

One important finding from Professors Reppa’s previous research was that when “icons were complex, abstract, or unfamiliar, there was a clear advantage for the more aesthetically appealing targets. By contrast, when the icons were visually simple, concrete, or familiar, aesthetic appeal no longer mattered.” The research team are now looking at yet more attributes, such as consistency, complexity and abstractness, to illustrate why and how the visual perception of icons changes within groups and in different situations or environments.

In the past we have used a simple voting system with five criteria using a Likert scale with an option to comment on the symbol and the evaluators have been experienced AAC users or those working in the field (which is small in number). On previous symbol survey occasions it has usually been the individual evaluator’s perception of the symbol, as seen in a text comment, that provided the best information. But, the comments have been small in number and the cohorts not necessarily representative of a wider population of communicators.

Symbol voting criteria

There is no doubt in my mind that we need to keep exploring ways to enhance our evaluation techniques by learning more from icon-based research, whilst being aware of the different needs of AAC users, where symbols may have a more abstract representation of a concept. This process may also help us to better categorise our symbols in the Global Symbols repository to aid text based and visual searches for those developing paper-based communication charts, boards and books as well as linking to the repository through AAC apps such as PiCom and Cboard.

Tenth Global Accessibility Awareness Day (GAAD) May 20, 2021

Over the last few years there has been a general move towards seeing how AI can help individuals involved with digital accessibility overcome some of the barriers faced by those with disabilities.  The use of machine learning can also provide access via assistive technologies that have been improved to such an extent that they are needing less and less human intervention.  Examples include automatic captioning on videos such as those presented on YouTube and speech recognition.

The question is whether we have really moved on from Deque’s 2018 “Five Ways in Which Artificial Intelligence Changes the Face of Web Accessibility”.

These included: 

  • Automated image recognition,
  • Automated facial recognition,
  • Automated lip-reading recognition,
  • Automated text summarization,
  • Real-time, automated translations.

Visiting the GAAD events page  is often a good way to find out as many companies and organisations world wide share what they have achieved over the year, such as Google with its Machine Learning for Accessibility where they discuss Voice Access, Lookout, and Live Transcribe along with Sound Notifications for Android on May 19, 8:15 PM and Microsoft with its AI powered 365 event and others also listed on the Access 2 Accessibility site. 

There is an AI for Accessibility Hackathon (Virtual) on May 24th – June 29th 9-10am BST (Beirut, Lebanon) run by the ABLE CLUB American University Of Beirut.  This competition is aimed at rallying talents and fostering the regional development of the innovative entrepreneurship community related to artificial intelligence while also increasing social inclusiveness.

AccessiBe.com uses machine learning and computer vision technologies for image recognition and OCR as it scans web pages for accessibility issues, just as our Group Design Project team used similar technologies on Web2Access to highlight alt tags that were possibly a poor representation of an image on a website and where overlaps occurred when zoom was used as well as a visualisation of a site on a mobile phone if it failed WCAG guidelines.  

However, still to come is Apple’s use of AI for screen recognition on iOS 14, where it “uses on-device intelligence to recognize elements on your screen to improve VoiceOver support for app and web experiences” such as detecting and identifying “important sounds such as alarms, and alerts you to them using notifications.”

So let’s all celebrate the improvements in digital accessibility that AI can bring, whilst making sure that one day there will be no need to have an AccessiBe YouTube video about “why web accessibility matters.”  It will just be something we can take for granted!  Equal Access for All. 

Web Page Accessibility and AI

computer with webpageOver the last year there has been an increasing amount of projects that have been using machine learning and image recognition to solve issues that cause accessibility barriers for web page users. Articles have been written about the subject. But we explored these ideas over a year ago having already added image recognition to check the accuracy of alternative texts on sites when carrying out an accessibility review on Web2Access.

Since that time we have been working on capturing data from online courses to develop training data via an onotology that can provide those working in education with a way of seeing what might cause a problem before the student even arrives on the course. The idea being that authors of the content can be alerted to the difficulties such as a lack of alternative texts or a need to annotate equations etc.

computer with presentationThe same can apply to online lectures provided for students working remotely. Live captioning from the videos are largely provided via automatic speech recognition. Once again a facilitator can be alerted to where errors are appearing in a live session, so that manual corrections can occur at speed and the quality of the output improved to provide not just more accurate captions over time, but also transcripts suitable for annotation. NRemote will provide a system that can be customised and offer students a chance to use teaching and learning materials in multiple formats.

We have also been discussing the use of text simplification that is making use of machine learning. The team behind EasyText AI have been making web pages easier to read and are now looking at the idea of incorporating text to symbol support where a user can choose a symbol set to suit their preference.

three sentences using symbols saying I read your red book today

Challenges to Implementation of AI and inclusion

treasure map

In no particular order as part of our roadmap we have been looking at the challenges facing aspects of inclusion for those who come under the umbrella of protected characteristics named in the UK’s Equality Act 2010

The list of challenges, for disabled people and those becoming less able due to age or debilitating illnesses, seems to grow despite the innovations being developed thanks to the use of clever algorithms and increasing amounts of data and high powered computing power. This is our first attempt at publishing our ideas…

road barriers

Challenges

Understanding the role and meaning Inclusion

  • Equity v equality

Disability is a Heterogeneous not homogeneous

  • Single ‘Disability’ classification not helpful as every disabled person can have very different needs
  • Small data for individual disabilities compared to big data for all (e.g. remove individuals whose data identifiable)

Skills and Abilities rather than deficit model

  • Looking at what an individual can do rather than focussing on the disabilities/difficulties

Designing for average rather than edge cases and outliers

  • Every disabled person may have very different needs compared to peers without a disability

Assumptions of Stakeholders

  • Changing attitudes
  • Lack of understanding – AI and ethics, data collection, algorithms, transparency  
  • Expectations of experts – will have a magic wand
  • Eugenics issues (e.g. Autism genetic correction)

Few disabled people involved in AI (Nothing about us without us)

  • Disabled people need to be involved in AI decisions
  • More disabled people need to understand AI

Capacity Issues

  • Resources – human, financial, tools
  • Policies and Procedures
  • Lack of general ICT as well as AT/AAC technologies that are regularly used in many settings

Cohesive Approach

  • Collaboration

AT and AAC Market

  • Small Market
  • Localisation issues

Lack of Competencies

  • Knowledge building

Black box non transparent Deep NN machine learning

  • Difficult to understand implications of AI DNN for disabled people

Lack of interest

  • Disabled people’s inclusion of little interest to Turing researchers and Turing research challenges and programmes (lack of knowledge due to lack of undergraduate courses, PhD supervisors, High impact Journals, Research funding etc.)

“We can only see a short distance ahead, but we can see plenty there that needs to be done.”

A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460.