Slides from my presentation during TDC Florianópolis 2016 on the .NET Track.

1. Assistive Context-
Aware ToolkitFelipe Pedroso

2. #IntelNoTDC
Agenda
• Stephen Hawking and Intel
• What is ACAT?
• Technical Details
• Architecture
• Creating Extensions
• How to?
• RealSense + ACAT
• SharpSenses + ACAT
• Summary and Call 4 Action

3. #IntelNoTDC
• Professor Hawking first met Gordon
Moore during a conference in 1997,
where he was offered a computer
with an Intel processor
• Intel upgrades his computer every
couple of years and also supports
him all his hardware requirements
• A cross-disciplinary team from Intel
Labs developed ACAT to enable him
to communicate and perform all his
computer tasks more efficiently
Stephen Hawking and Intel
Images Sources:
http://www.astrosurf.com/luxorion/hawking-hommage9.htm
http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/

4. #IntelNoTDC
• Acronym for “Assistive Context-
Aware Toolkit”
• Enable people with disabilities to
communicate through keyboard
simulation, word prediction and
speech synthesis
• Developed by Intel Labs together
with Professor Hawking
• Customizable and Extensible
What is ACAT?

5. #IntelNoTDC
Open Source
http://01.org/ACAT
https://github.com/01org/acat

6. #IntelNoTDC
Tech Details
• Runs on Windows 7 or higher
• Code entirely written in C# using Visual 2012 (works fine on 2015) and
Microsoft .NET Framework 4.5
• Microsoft Office Interop Assemblies are used to interact with Office apps
• Some tools used to develop the project:
• CodeMaid
• StyleCop
• ReSharper

7. #IntelNoTDC
Architecture
Applications
ACAT Core Library
ACAT Extension Library
Extensions
Extension Extension Extension ...

8. #IntelNoTDC
Applications
ACAT Qwerty ACAT Abc ACAT Tryout
Talk Window

9. #IntelNoTDC
ACAT Core Library - Components
Word
Prediction*
Management
UI Management
AnimationsPanels Widgets Themes
Text-to-Speech
Management
Talk Window
Management
*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)

10. #IntelNoTDC
ACAT Core Library - Agents

11. #IntelNoTDC
ACAT Core Library - Actuators
Actuator
Management
Image Sources: http://www.psfk.com/2013/02/intel-stephen-hawking.html & https://commons.wikimedia.org/wiki/File:Assistive_device.JPG
Actuators
Proximity Sensor Assistive Switch Camera

12. #IntelNoTDC
ACAT Extensions Library
• Allows developers to extend the functionalities from ACAT
• Dynamically loaded during startup using reflection
• Needs to implement specific interfaces/base classes
Extension Types
UI
Word Predictors
Agents
Actuators
Text-to-Speech Spell Checkers

13. #IntelNoTDC
Creating an Actuator Extension
• Create a new “Class Library” project and add references to “ACATCore.dll”
and “ACATExtension.dll”
• Extend the classes “ActuatorBase” and “ActuatorSwitchBase” to interact with
ACAT
• Implement the abstract methods and the use the callbacks (Init, Resume,
Pause, Dispose) to run the code of your extension (threads, config, etc)
• Call the methods OnSwitchActivated (Hold/Press), OnSwitchDeactivated
(Release) or OnSwitchTriggered(Click) to sinalize the interaction from the
actuator
• Deploy your DLL to the extension folder and update the configuration files

14. #IntelNoTDC
Intel® RealSense™
Intel® RealSense™ SDK
Scene
Understanding
Enhanced
Photography
Augmented
Reality
Person & Face
Tracking
Gesture
Recognition
Background
Removal
Object
Recognition
and Tracking
Speech
Recognition
3D Object
Scanning
Hand & Blob
Tracking
User Facing World Facing

15. #IntelNoTDC
Tracking (Position)
Landmarks (78 landmarks)
Pose / Orientation
Expressions
Recognition
Pulse estimation
Intel® RealSense™ - Face Module

16. #IntelNoTDC
• Objective: create an actuator driven
by facial expressions
• Components:
• RealSenseExpressionTracker:
RealSense SDK implementation to
perceive the expressions and notify the
Actuator using events
• RealSenseActuator: translates the
intensity of the expressions to a button
behavior (pressed and released)
Intel® RealSense™ - ACAT Plugin
RealSense Actuator Extension
ExpressionTracker
Actuator Switch
Actuator
Management

17. #IntelNoTDC
• The Camera actuator extension is a
socket client that keeps looking for
the “ACAT Vision” app, that has a
socket server
• Simple protocol and it’s possible to
customize using config files
• ACAT Vision is not yet open-source,
but its code will be released in
upcoming releases
Another way to connect an actuator
ACAT Vision (Server)
Camera Actuator (Client)
Actuator
Management

18. #IntelNoTDC
• SharpSenses* is an open source C#
library that simplifies the
development of Intel® RealSense™
SDK apps.
• Created and maintained by André
Carlucci (Intel Black Belt & Software
Innovator)
• Available on Github* and NuGet*
SharpSenses*

19. #IntelNoTDC
• The idea is to create a ‘fake’ server,
that replicates the communication
between “ACAT Vision” and ACAT
• To notify that the input was triggered,
just send the string
“gesture=trigger;action=CT;conf=75;t
ime=<TIMESTAMP>”
• ACAT also sends some events back
to the server (HideUI, ShowUI,
ExitApp)
SharpSenses* + ACAT
ACAT Vision
Camera Actuator (Client)
Actuator
Management
‘Fake’ Server
SharpSenses

20. #IntelNoTDC
Summary & Call 4 Action
• ACAT is a really nice option to allow people with disabilities to communicate
with others
• Open Source: You can fork and extend ACAT for your needs!
• Extensions: let’s help other people with new actuators, applications agents and
new features 
• RealSense and SharpSenses can also drive acessibility apps.

21. Q/A

22. #IntelNoTDC
References
• Stephen Hawking and Intel
• http://iq.intel.com/behind-scenes-intel-keeps-stephen-hawking-talking/
• http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/
• ACAT
• https://github.com/01org/acat
• http://01.org/ACAT
• RealSense: https://software.intel.com/en-us/realsense/home
• SharpSenses: https://github.com/SharpSenses/SharpSenses

23. Thanks!

24. BAckup

25. #IntelNoTDC
ACAT Server
SharpSenses + ACAT
SharpSenses
bool canRun = true;
AcatServer acatServer = new AcatServer();
acatServer.Start();
acatServer.AcatNotificationReceived += (sender, a) => {
Console.WriteLine("Notification received from ACAT: {0}", a.NotificationType);
if (a.NotificationType == AcatNotificationType.ExitApp)
{
canRun = false;
}
};
ICamera camera = Camera.Create(Capability.FaceTracking, Capability.FacialExpressionTracking);
camera.Start();
camera.Face.LeftEye.Blink += (s,a) => {
Console.WriteLine("SharpSenses: detected a blink from left eye.");
acatServer.SendGestureMessage("CT", "trigger");
};
while (canRun) { }
acatServer.Stop();