This document proposes developing an Android application to facilitate communication between hearing impaired and general people in Bangladesh through Bangla speech-to-sign language and sign language-to-text translation. It aims to recognize Bangla speech and convert it to animated sign language displays and develop a sign language keyboard to type Bangla text. The methodology involves using speech recognition APIs to convert speech to text, tagging parts of speech, looking up signs from an animated database, and displaying them sequentially via a virtual agent. It will also design a keyboard with buttons for Bangla characters and their corresponding signs.

1. 1
CHITTAGONG UNIVERSITY OF ENGINEERING AND TECHNOLOGY (CUET)
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
CHITTAGONG-4349
(PROJECT PROPOSAL)
Application for the Approval of B.Sc. Engineering Project
(Computer Science and Engineering)
Date: 09.10.2018
1. Name of the Student : Afif Bin Kamrul
Student ID : 1404065
Session : 2018-2019
2. Present Address : Room no-357, Dr. Qudrat-E-Khuda Hall, Chittagong
University of Engineering and Technology
3. Name of the Supervisor : Shayla Sharmin
Designation : Assistant Professor, Department of Computer
Science and Engineering, Chittagong University of
Engineering and Technology
4. Name of the Department : Computer Science and Engineering (CSE)
Program : B.Sc. Engineering
5. Date of First Enrolment in the
Program : March 18, 2015
6. Tentative Title : An Android Communication Platform Between
Hearing Impaired and General People.

2. 2
7. Introduction:
Communication is one of the major needs of human. People without hearing impairments can
listen and speak but deaf people cannot; instead deaf people use signs to communicate, these signs
are known as sign language. Sign language is a visual language which uses various body
movements as a method of communication. Like natural languages, there are different forms of
sign languages used in different countries around the world. There is a large gap between general
people and deaf society during communication. It is very hard for deaf people to communicate
with normal people. In maximum cases, sign languages are only used for direct visual
communication, such as video broadcasts or interpersonal communication. Unlike their natural
language counterpart, the sign languages are usually grammatically different. Many deaf people
may not know the natural language at all while many general people may not know how to
communicate in sign language. These differences make it very difficult for these two types of
people to communicate effectively with each other without a translator. Human interpreters are
inefficient and there are not enough of them to make sure every personal communication can take
place between the two communities of people.
According to “National Survey on Prevalence of Hearing Impairment in Bangladesh 2013”
published by WHO/SEARO/Country Office for Bangladesh and Ministry of Health and Family
Welfare (BSMMU), hearing impairment is the second commonest form of disability. One small-
scale study done in 2002 reported to WHO a prevalence of 7.9% hearing impairment (in better ear)
in Bangladeshi people. In addition, Bangladesh has a population of over 130 million by the
Population Census 20001 National Report (Provisional). Bangladesh Bureau of statistic, Dhaka,
Bangladesh, July 2003. Moreover, about 13 million people are suffering from variable degrees of
hearing loss of which 3 million are suffering from severe to profound hearing loss leading to
disability in accordance to Amin MN: Prevention of Deafness and Primary Ear Care (Bengali)-
Society for Assistance to Hearing Impaired Children (SAHIC), Mohakhali, Dhaka-1212,
Bangladesh. So, it’s a pretty serious issue to be prioritized for. This problem is causing economic,
social, educational and vocational problems both for the victims and the country. Such problems
could be avoided if there existed a digital translator between the normal society and deaf people.
Source: http://www.searo.who.int/bangladesh/publications/national_survey/en/

3. 3
The proposed research topic is a system for digitally converting Bangla speech to Bangla sign
language, to enable effective communication between the two communities. Another system will
be provided for converting sign to Bangla language by implementing a sign keyboard to ensure
two-way communication. For this to implement Android platform is selected.
Android is a operating system developed by Google for smartphones, based on Linux kernel and
other open source software and designed primarily for touchscreen mobile devices such
as smartphones and tablets. On the other hand, for developers, Android Studio is the official
Integrated Development Environment (IDE) for Android app development.
People not having physical impairment are featured with a user-friendly interface having built in
audio recognition system recognizing Bangla language. Users have to just open the application so
that they can talk in Bangla. The built-in cloud speech recognition system in android studio will
detect the speech of general people as Bangla text. Afterwards, the text will be further analyzed
and broken into root words using Bangla Language Processing. An animated database will be kept
providing the root words as name of corresponding animated clip. The Java program will have
fetched the appropriate animation for each word and play those sequentially. Thus, the deaf people
will understand easily what the person on the other end is trying to say. Therefore, the
communication medium from people without physical impairments to deaf people is established.
On the contrary, deaf people can understand sign language. There are specific signs for each of the
Bangla character. A keyboard will be designed for Bangla language where buttons of Bangla letters
with corresponding sign will be provided. While typing, corresponding Bangla word will be typed
on the screen so that sentence can be formed. The sentence will be recognizable to the general
users. A SMS system will be equipped by the program and so the users on the both ends can
communicate in between them via mobile number.
The above-mentioned system may be useful for deaf pupils, for parents of deaf children and for
any person who is in contact with deaf and need to learn sign language. Therefore, contribute in
reducing the language barrier between deaf and hearing people.

4. 4
8. Previous Work and Research:
There has been some notable work performed based on identifying sign language image or body
movement to develop a medium for deaf people. But there has not been a lot of work done on the
opposite. This kind of work is done by some honorable persons from India where they developed
a translator for Bangla text to sign language. In the process, they maintained a dictionary
containing about 1000 words against unique id, grammar id, path and filename. In the sentence
mode enabled, the corresponding Bangla word is fetched according to corresponding matched id
and individual video clips are concatenated one after another in the correct sequence to generate a
video clip that represents the sign language output corresponding to the input text. [1] Same type
of work is presented in another research [2] where input text is broken and analyzed into syntactic
and morphological level. Unicode which provides a unique character for every character, no matter
what the platform, no matter what the program, no matter what the language; is used for this
research. At first, unusual characters or special characters that are not used usually are removed
using syntactic analysis. Then, texts are ordered in series and according to the rule of Bangla
grammar through morphological analysis. Database is maintained for root words and it also
contains parts of speech of each word. Words are stored in a tree like structure where sign image
is only provided for parent word to avoid duplication.
A necessary research has been conducted in [3] where an empirical framework for tokenizing and
parsing Bangla three types of sentences: assertive, interrogative and imperative is shown. In this
work presented here, input sequence is taken at first. Then the program breaks the string into
individual words called tokens. A lexicon is maintained containing parts of speech of each word.
Some Context-Sensitive Grammar (CSG) rules are used to for processing tokens of input sequence
to generate parse tree or structural representation according to CSG rules.
A two-way communication is established in mobile platforms between a deaf/dumb person and a
normal person. In the process, an open-source speech recognition software called CMU Sphinx is
used to convert Bangla speech to Bangla sign language. The Sphinx defines Bangla phonetics,
words and grammars and recognizes speech input and convert it to phonetic text. Scope of words
are stored in dictionary, hence sentences are built and stored as Bangla language model file and
audio files are stored as Bangla acoustic model file and therefore using those required files, training
process is begun which breaks recorded speech into phonetics text. By converting texts to Bangla

5. 5
words, these are matched against the database and if match found, corresponding image are shown
on screen from database fetched from database. On the other end, impaired persons have to write
Bangla text which would be converted to speech by utilizing Google Translation Server. This is a
lacking due to the fact that impaired persons may not know the Bangla language which requires to
design a new keyboard for them. [4]
An essential keyboard design is introduced for deaf people [6] and Bangla layout is described. [5]
The swipe and press layout and sign keyboard both can be combined for the sake of deaf people
to communicate with normal people. The swipe and press keyboard layout would require less space
on screen. Both can be applied for android mobile. The sign language keyboard also support speech
to text and text to speech where text to speech allows a use of an algorithm where standard output
use a morphological analyzer module to determine the stem and part of speech of each individual
word. After preprocessing, a search is begun to find exact match of the word using hash value. If
not found, Levenshtein distance between those stems and the stem of the word under search is
calculated and minimum value of distance less than two is selected from database. If that word is
not found also, the word is broken into letters and signs are retrieved for letters.
Android app developed in different platforms can be communicated through specific methods. [7]
The inter-app communication can be established using predefined ‘intent’, built into the Java
package.
A tag-vector based approach can be useful for tagging parts of speech of Bangla word. [8] At first,
automatic word form is recognized by utilizing either word form method or morpheme method or
allomorph method. The process searches root lexicon and returns corresponding tag probability
vector. Another process [9] is implemented using hash table where root words and
suffixes/inflections are stored separately to determine parts of speech of the word.
Animation based teaching assistant is presented with the help of ‘WebSign’; which is based on the
technology of avatar (animation in virtual world). It translates transcriptions of natural hearing
languages to a real time animation in sign language. [10] Actual sign is played for each word from
dictionary.

6. 6
9. Objectives:
The main objectives of this project are as following:
a. To recognize Bangla speech and convert them to sign language for deaf people.
b. To develop a sign language keyboard that converts the sign into Bangla text.
10. Challenges:
1. To select a large scope of Bangla signs.
2. Creating animation for selected Bangla root words.
3. Creating database for storing the animated files for corresponding Bangla words.
4. Designing and applying signs for Bangla letters into the keyboard buttons.
11. Methodology:
Bangla Speech to Sign Language: Google provides a powerful speech-to-text conversion
platform and enables developers to convert audio to text by applying an easy-to-use API. The API
recognizes Bangla variant which is used in Bangladesh. Internet connection is required for
detection of Bangla language as it is not available from Google in offline mode. The Java program
will handle the texts generated from speech. In order to launch the speech recognition API, the
‘Intent class’ will be helpful. An ‘Intent’ is an abstract description of an operation to be performed.
The recognized Bangla speech will be processed as a Java string.
Bangla Sign Language Structure: Sign language follows some certain rules:
 An adjective generally comes after the noun.
 A “wh-” word comes at the end of a sentence. For example: “তু মি মি খেয়েছ?” will be
represented as “তু মি খেয়েছ মি?”.
 If a verb is in the past or in future tense, a tense marker gets added before the verb.
 In Bangla sign language, there is no sign for words like কিন্তু (“but”), এবং (“and”), নতু বা
(“otherwise”). These words get ignored.
 If signs are not available for the whole sentence, verbs are prioritized the most.
Lexicon: There is a unique code associated with each character in Bangla word which is called
Unicode. There is Unicode compatibilities in Java. A lexicon in other words simply a dictionary,

7. 7
with Bangla root words will be maintained as a database where a Bangla root word will be stored
with a unique id, a parts of speech (POS) tag and an animated clip.
Tokenizer: A tokenizer program accepting the text as an unbroken string will break the texts into
individual words called tokens, which will be stored for further access.
Parts of Speech Tagging: POS tagging is hard for Bangla as some words can represent more one
parts of speech at a time in a sentence. But only noun, adjective and verbs are required tagging for
this project.
Display Sign Language with the Help of Virtual Agent:
 First, the Bangla tokens will be analyzed to find direct match with the lexicon.
 If no match is found, unwanted suffixes, inflections, prefixes etc. will be separated from
root word.
 If the root word matches with dictionary (lexicon), an animated virtual agent will appear
on screen to display sign language following the structures of sign language.
 If the word is not available in dictionary, sign language of each character of the word will
be shown.
Animation: A wide variety of Bangla word will be selected for animation. Software suitable for
drawing image will be used for virtual agent displaying the signs. One, two or three images are
enough to depict the signs per word. This part is a bit tricky as images have to cover many root
words of Bangla language. In addition, software tool will be used to add the images to create the
animation. In order to display the animated file, android studio offers some effective features.
Sign Language Keyboard: Keyboard buttons will be designed using some software tools. The
buttons will not only hold Bangla characters but also the corresponding sign of Bangla characters
which enables the deaf persons to recognize the Bangla characters. Owing to press the key, the
Bangla text will be written.
SMS: The Bangla text converted from speech will be used to send SMS to any deaf user. It will
also be capable of sending the SMS to a general user as well. On the other hand, the sign language
keyboard for deaf people will directly type Bangla text and if the deaf people press the send button,

8. 8
the SMS will be sent to the appropriate number. It will also be able to send SMS to a deaf person
as well.
Speech to Sign
Language Conversion
with the Help of Virtual
Agent
Convert Bangla Text
Fig 1: Structural Representation of the Application
Speaking
Bangla
Fetching Sign
from
Database
Sign
Language
Output
Database of
Animated Signs
Fig 2: Speech to Sign Language Conversion
Launch Speech to Text
Launch Sign Language
Keyboard
Start
End
Splitting
Suffixes,
Inflections etc.
Splitting Words
from Sentence
(Tokenizing)
Parts of
Speech
Tagging
ছাত্র (Student)

9. 9
12. Required Resources:
Required resources are given below:
1. Personal computer.
2. Android smartphone.
3. Android Studio.
4. Java language to program in Android Studio.
Start
Image of Sign
Match with Corresponding Bangla
Character
Show Character
Fig 3: Sign Language Keyboard
End
Build Sentence
Select Sign
ছয়(Six)

10. 10
13. Cost estimate:
The costs that will occur to implement the proposed system is given below:
Personal Computer Tk. 55000
Android Smartphone Tk. 14700
Internet Cost Tk. 3000
Drafting Tk. 100
Binding Tk. 200
Paper Tk. 200
Total Cost Tk. 73200
14. References:
[1] Sarkar, B., Datta, K., Datta, C., Sarkar, D., Dutta, S., Roy, I., Paul, A., Molla, J. and Paul, A.
(2009). A Translator for Bangla Text to Sign Language. 2009 Annual IEEE India Conference.
[2] Ahmed Tanvir, “A Small Initiative to Convert Bangla Text to Bangla Sign Language”,
American-International University, Bangladesh.
Available:
https://www.academia.edu/30768422/A_Small_Initiative_to_Convert_Bangla_Text_to_Bangla_
Sign_Language
[3] Arefin, M., Alam, L., Sharmin, S. and Hoque, M. (2015). An empirical framework for parsing
Bangla assertive, interrogative and imperative sentences. 2015 International Conference on
Computer and Information Engineering (ICCIE).
[4] Shahriar, R., Zaman, A., Ahmed, T., Khan, S. and Maruf, H. (2017). A communication
platform between bangla and sign language. 2017 IEEE Region 10 Humanitarian Technology
Conference (R10-HTC).
[5] Islam, K. and Sarker, B. (2014). Designing a press and swipe type single layered Bangla soft
keyboard for Android devices. 16th Int'l Conf. Computer and Information Technology.
[6] El-Gayyar, M., Ibrahim, A. and Sallam, A. (2015). The ArSL keyboard for android. 2015 IEEE
Seventh International Conference on Intelligent Computing and Information Systems (ICICIS).
[7] Allison, L. and Fuad, M. (2016). Inter-app communication between Android apps developed
in app-inventor and Android studio. Proceedings of the International Workshop on Mobile
Software Engineering and Systems - MOBILESoft '16.
[8] Md. Shahnur Azad Chowdhury, Nahid Mohammad Minhaz Uddin, Mohammad Imran,
Mohammad Mahadi Hassan and Md. Emdadul Haque, “Parts of Speech Tagging of Bangla
Sentence”,
Available:
https://www.researchgate.net/publication/229038426_Parts_of_Speech_Tagging_of_Bangla_Sen
tence

11. 11
[9] Ismail, S., Rahman, M. and Al Mumin, M. (2014). Developing an automated Bangla parts of
speech tagged dictionary. 16th Int'l Conf. Computer and Information Technology.Ismail, S.,
Rahman, M. S., & Al Mumin, M. A. (2014). Developing an automated Bangla parts of speech
tagged dictionary. 16th Int'l Conf. Computer and Information Technology.
doi:10.1109/iccitechn.2014.6997347
[10] Jemni, M. and Elghoul, O. (2008). Using ICT to Teach Sign Language. 2008 Eighth IEEE
International Conference on Advanced Learning Technologies.