Methodology for the evaluation/assessment of competences

This project is funded with support from the European Commission. This document reflects the views only
of the author, and the Commission cannot be held responsible for any use which may be made of the
information contained therein. Project Nº 518536-LLP-1-2011-1-ES-GRUNDTVIG-GMP
“KEYROMA: Develop Key Competences in Social Skills for
Roma Women to increase their participation in the service
sector” Project
I.I.T.
GUIDELINES FOR ASSESSMENT OF COMPETENCIES
AND TRAINING NEEDS ANALYSIS
This project is funded with support from the European Commission. This document reflects the
views only of the author, and the Commission cannot be held responsible for any use which may
be made of the information contained therein.
Project Nº 518536-LLP-1-2011-1-ES-GRUNDTVIG-GMP

Guidelines for assessment of competencies and training needs analysis
This document aims at providing guidance and suggestions for the local teams involved in the
Key Roma project in order to perform the assessment of the competencies of the Roma
women member of the Local Learning Groups and the analysis of the training needs of the
group members.
It is assumed that a Local Learning Group consisting of Roma women has been set-up based on
voluntary participation in each project location.
The present guidelines need to be adapted and implemented with each Local Learning Group
considering the specific situation and the profile of group members, putting on the first place
the respect for the group members and the focus on empowering them to engage in a positive
change process, beneficial for themselves, their families, their community and the overall
society. These guidelines are meant to support partners in producing a report on
competencies of group members.
As specified in the project, the Local Learning Group is expected to evaluate and assess the
various competencies, with a focus on the transversal social competencies already managed by
Roma women participating into the project, acquired through formal, non-formal and informal
education. On this basis, the real learning needs of the groups concerning social and
communication competencies will be identifies, concentrating on those transversal
competencies required for labour integration into the service sector. This process will make
women aware of the potential that they have, increasing self-esteem and making them feeling
responsible of their personal education process. This will be the starting point for each group
of Roma women to understand on what kind of social competence they really want to focus.
Let us first clarify the meaning of the key concepts we are using: formal, non-formal and
informal education and learning and social competence.
Formal education refers to the structured education system that runs from pre-school and
primary school to university, and includes specialised programmes for technical and
professional training.
Non-formal education refers to any planned programme of personal and social education
designed to improve a range of skills and competencies, outside the formal educational
curriculum.
Informal education refers to the lifelong process, whereby every individual acquires attitudes,
values, skills and knowledge from the educational influences and resources in his or her own
environment and from daily experience (family, neighbours, marketplace, mass media, work,
play, etc.).
Social competence is a complex, multidimensional concept consisting of social, emotional
(e.g., affect regulation), cognitive (e.g., fund of information, skills for processing/acquisition,
perspective taking), and behavioural (e.g., conversation skills, pro-social behaviour) skills, as
well as motivational and expectancy sets (e.g., moral development, self-efficacy) needed for
successful social adaptation. Social competence also reflects having an ability to take another's

perspective concerning a situation, learn from past experiences, and apply that learning to the
changes in social interactions. Social competence is the foundation upon which expectations
for future interaction with others are built, and upon which individuals develop perceptions of
their own behaviour. The concept of social competence usually includes social skills, social
communication, and interpersonal communication.
Having social competence means possessing the social, emotional, and intellectual skills and
behaviours needed to succeed as a member of society and to adapt in appropriate ways to the
demands of social interaction, including interactions related to work environment. The skills
and behaviours required for healthy social development vary with the demands of particular
situations.
Measuring social competencies
Social competencies have been identified by the European Commission as one of the key
benchmark indicators to be targeted in order to improve prosperity and well-being in its
Member States (EU 2005). Social competencies can be broadly defined as the capabilities
enabling individuals ‘to live together in the world’ (Arendt 1958) comprising aspects of
interpersonal, intercultural, social and civic competencies. Social competencies reflect
adjustment in different contexts in society, requiring a focus on particular facets of social
competence, such as empathy, self-control, trust, respect for other people, or civic
engagement.
Behaviours which are functional in one context might be dysfunctional in another, implying
that the assessment of social competencies involves culturally based value judgments. These
values are not fixed, but are subject to change and such changes become more and more
visible in the current period, affecting all society and the work environment.
The assessment of social competencies can be done with a variety of methods, ranging from
self-ratings or self-reports of behaviour, values, and motivations; direct behavioural
observations (in natural situations or under experimental conditions) to the use by a specialist
of different assessment instruments. Such instruments can include behaviour rating scales (to
be completed by parents, teacher, employer, subordinates, or by the person whose
competencies are assessed); interviewing; role play; discussion based on hypothetical
scenarios; interpretation of video clips; social network analysis and socio-metric approaches;
as well as computer simulations.
A widely used instrument to assess personality characteristics is the ‘Big Five’ inventory,
focused on openness, conscientiousness, extraversion, agreeableness, and neuroticism
(OCEAN) (Costa and McCrae 1992; McCrae and Costa 2004). Openness is reﬂected in a strong
intellectual curiosity and a preference for novelty and variety. Conscientiousness is exempliﬁed
by being disciplined, organized, and achievement-oriented. Extraversion is displayed through a
higher degree of sociability, assertiveness, and talkativeness. Agreeableness refers to being
helpful, cooperative, and sympathetic towards others. Neuroticism refers to degree of
emotional stability, impulse control, and anxiety.
Other widely used self-reporting instruments include: self-esteem scales; measures of self-
efficacy or emotional intelligence; tests measuring cognitive and affective aspects of
dispositional empathy.

Concerns have been raised about the consistency and reliability of self-assessments as well as
biases in reporting (Hagerty et al. 2007). One problem with single item measures (such as
scales where participants tick a box corresponding to the level they evaluate they posess the
respective competence), although it is attractive, is that such approaches are only suitable to
assess constructs that are simple and unambiguous. Sometimes it is hard to capture the
complexity of a competence and there is the risk to miss important differences at the
individual level. Also, data collected in this way might be ‘contaminated’ by the context in
which they are collected. Besides this, recent research (Kurger and Dunning, 2009) has pointed
out that there is a general tendency for people who are more competent to underestimate
their competence, while people who are actually less competent tend to overestimate their
competence.
Psychometric scales comprising multiple items to measure a specific dimension, such as social
intelligence, social responsibility, assertiveness, or empathy, are considered by some
specialists as more reliable, yet often take longer time to complete, and without abbreviation
are not suitable for large scale surveys. The same applies to attempts to measure social
competencies on the basis of assessments in experimental settings, make belief scenarios, or
interpretations of video clips, which usually take more time to collect. An additional challenge
appears with these instruments in our case due to the specific situation of target group
members, with important cultural differences compared to the mainstream society in which
the respective tests have been validated and with sometimes limited possibilities of
understanding some items or of expressing properly the answers.
Ideally the measurement of social competencies should involve different assessment modes,
combining self-reports, rating scales completed by others, as well as observational data to
obtain reliable and valid measures. Instead of direct assessments, multiple measures could be
used as indicators of latent constructs, which would also facilitate comparative approaches of
assessment.
The High Level Advisory Group of Experts set-up by the European Commission identified in
their report on the social integration of disadvantaged ethnic minorities, published in 2007, 14
barriers which prevent members of these groups from fully participating in the labour market:
(1) Lack of education and training
(2) Lack of language skills
(3) Lack of recognition of skills and
qualifications
(4) Lack of access to professions
(5) Lack of access to citizenship
(6) Lack of integration policies
(7) Stereotypes, prejudices and negative
attitudes
(8) Lack of mobility and concentration in
certain areas
(9) Industrial change
(10) Disincentives through welfare systems
(11) Discrimination
(12) Lack of information
(13) Labour market competition
(14) Undeclared work
These barriers are particularly relevant for many Roma and some of them might affect directly
members of our local groups. Therefore, special attention should be paid to identifying for
each participant which barriers have a higher impact in preventing access to the labour
market. It might be that some barriers will not apply to some participants and this also needs
to be considered. For instance, some members of the group might have had experiences of
travelling to other countries over the past few years, thus acquiring significant life experiences
due to mobility and sometimes valuable language skills which could represent an important
asset for their future professional development.

One additional challenge is that, from our previous experience and based on the first
interactions with the members of our group, we anticipate that at least in some cases
participants will not consider social competencies as relevant or playing a decisive role in
finding a job. They will also expect that employers will not be interested in such competencies
but rather require formal qualifications. This is also related to possible low expectations of the
women in the group, based on their own previous experience and on what they see around
them in the community.
Therefore, besides the transversal social competencies, it is useful to make an inventory of
other specific competencies that women in the group might have, including competencies
related to certain job profiles. It is also important to provide opportunities for the women in
the groups to express their own projects for personal and professional development and
learning, including further education, qualification, etc. They will also need opportunities for
exchange and joint reflection with peers in order to overcome some of the limitations they
face in their concrete situation and to get fresh ideas about what they could do in the future in
their professional life. In this context, the assessment should also reinforce the self-esteem of
participants and avoid risks for them to feel looked down to in this process.
Considering all these challenges, our suggestion is to use a combination of qualitative
assessment methods which can be connected with specific life experiences and situations of
group members. Thus, the methods recommended for assessment of competencies of Roma
women in the project group include:
1. In-depth interviews with focus on life story and description of daily life experiences
2. Appreciative inquiry
3. Focus group
The results obtained with these methods will serve, on one hand, for the development of the
training activities to be implemented in the next phase of the project and, on the other hand,
for identifying employers potentially interested in hiring Roma women in the group.
The results can be structured in two categories:
1. Competencies that participants have and that can represent an asset for them on the
labour market
2. Social competencies that are important for a successful integration in the labour
market but that they lack or have in an unsatisfactory level.
The first category should be presented to each participant individually and this can be a way
to:
- Get useful feedback from them and possibly add useful element that are identified by
participants after looking at list of their competencies;
- Boost the self-esteem of participants and their confidence that they can envisage
professional progress.
The second category should not be communicated to group members and should be kept
confidential by the team. By communicating negative assessments they could be demotivated
or manifest defensive attitudes. But these elements can give useful hints for the development
of the training activities.

In order to reach to the two lists of competencies for each participant, we suggest taking an
inductive approach: collect qualitative information and process this information by identifying
elements to be put in each category.
This can be done by paying attention to competencies acquired by the Roma women in the
group through (the list is not exhaustive):
- Formal education
- Professional qualification trainings
- Non-formal education activities
- Work experiences, both formal and informal
- Interaction with others (e.g. learning how to do something from a family member)
- Daily household activities (e.g. cooking, gardening)
- Providing care to family members (children, elderly, ill people, people with disabilities)
- Participation in community events (e.g. organising wedding party, traditional festivals)
- Travelling and changes in place of living (locally, nationally, internationally)
- Media (e.g. radio, TV)
Thus, the output of this process can be for each participant two lists:
- Competencies acquired
- Competencies lacking
They can be presented in a descriptive way, as they result from the statements of participants,
and can be categorised and organised at a later stage.
1. In-depth interviews
Describing life situations is a simple and useful way to get information about what a person
knows, thinks, or knows to do. Asking specific question about their daily life will also put
participants in a comfortable situation, providing more reliable information than what can be
obtained by answering questions from a questionnaire.
Before the interview
You explain to participants that an open discussion of about one hour about their life is
necessary for making an assessment of their competencies, going beyond those acquired
through formal qualifications or through work experiences. You set up together a meeting,
specifying time and place. Make sure that the place is easy to reach for the participant an that
it provides good conditions for the discussion, including privacy and protection from
disturbances. Prepare a device for audio recording and make sure it has enough space for the
whole interview. It is recommended to test the recorder if you have not used it before.
During the interview
Ask questions slowly and clearly, giving the person time to answer. You can use some closed
questions (which prompt a respondent to give only a "yes" or "no" answer), but most should
be open questions like: "Tell me about..."; "Describe..."; "What was it like when...?"; "In what
ways...?"; "Why...?"; and "How...?"
Start with easy, friendly questions and work your way up to more difficult or sensitive
questions. The precise set of questions will not be the same for all participants, depending on
various factors, such as their age, education, specific life experiences, etc.

Here is a set of suggested questions (select only the ones which are relevant and adapt as
appropriate, both the content of the questions and the language used):
- Tell me about your childhood. Where were you born? Where did you spend your first
years? What do you remember?
- Who were the most important persons for you when you were a child? What did you
learn from them?
- As you grew up, were you helping with household activities? What where you doing?
Who taught you how? What did you like most?
- Who were your friends? What were you doing together? Did you have conflicts with
them? How were they dealt with?
- Where did you go to school? How was it? What teacher you liked most? Why? How did
you get along with the colleagues?
- Let’s talk now about your work experiences (not necessarily related to employment).
What was your task/activity? With whom were you working? Describe a successful
experience of interaction with co-workers, with your boss and with
clients/beneficiaries
- At home, what are your tasks? How did you learn how to do that? What is your
reaction if there is a disagreement between family members or between family
members and others? Did you take care of other family members (children, brothers
and sisters, elderly)? How? Could you give an advice to others in this situation, based
on your experience?
- Tell me about when you travelled abroad/to another region/city. Where did you go?
With whom? What did you do there? What did you find surprising or different there?
How did you manage to communicate with the people there? What are the most
important things that you learnt from this experience?
Listen carefully to what the person says; don't interrupt or correct. Maintain eye contact and
show interest by leaning forward and nodding.
As you listen to answers, other questions will come to mind. Asking follow-up questions will
help you get more information.
If someone is talking about an unhappy or painful experience, show that you understand how
they feel ("That's very sad"). If the person doesn't want to talk about something, that's okay –
just go to the next question.
It's okay for there to be moments of silence or emotion. A person's life is important, and
emotion is natural. Accept emotions as part of the process.
An interview shouldn't last more than about an hour. People do best when they're not tired. If
necessary you can schedule another time to continue. This also allows you to think about
answers, and come up with other questions based on the answers and things that interest you.
Don't forget to thank the person you've interviewed. Let her know that you are going to meet
again to provide feedback after the analysis of the interview and for a second short interview
(the appreciative enquiry).
After the interview

You do not need to transcribe the whole interview (this will save a lot of time and energy). You
just need to transcribe the sequences which reveal competences (things that the participant
knows or knows to do, as well as specific attitudes). It is preferable to do this shortly after the
interview, as it will be easier to remember details which can help understand possible difficult
sequences. Processing interviews as you do them will also help you in the following ones as
you gain useful experiences with each interview.
2. Appreciative inquiry
This is a brief activity which aims both to complete the information acquired through the in-
depth interview and to contribute to increasing the self-esteem and the confidence of
participants by pointing out various positive elements about them.
This can be done during the second meeting with each participant, when the conclusions of
the analysis of the in-depth interview are presented.
You can start by showing the conclusions you drawn from analysing the interview, pointing out
only the positive elements (what the person knows, what she knows to do and the positive
attitudes that she manifested in various situations). You can ask for feedback, confirmation of
the list of competencies identified and also ask if she wants to add something to the list and
illustrate each additional competence with a specific life experience.
After this, ask a brief series of questions focused on positive elements. The structure of the
interview can be the following:
- Give two examples of situations from your life when you felt happy or proud
- Describe two positive qualities of yourself and illustrate how they were shown in
various situations
- Imagine that you get a job according to your wishes and that you have worked for a
year. How would you like your boss to praise you at the end of your first year at work?
This type of questions is likely to help participants identify their strengths and picture
themselves in a positive and successful professional experience.
3. Focus group
Introduction: Focus group is a qualitative research method which takes the form of a group
interview, a facilitated discussion led by a series of questions around a specific topic. Below are
some instructions for the facilitator of the focus group.
Duration of the focus group: 90-120 minutes
The recommended number of participants in the focus group is between 8-10, in order to
ensure a good interaction and group dynamics and allow everybody enough time to express
opinions. Therefore, for our 15-20 participants, there should be two focus groups organised.

The division of participants in groups can be done based on various criteria but it’s important
to ensure some degree of diversity in each group. This way, participants will be more
stimulated by each other.
Opening
The moderator has to ensure that all invited participants are present and sit in a circle. After
having introduced himself/herself and reminded that the purpose of the discussion is to get
ideas useful for preparing a training programme and for facilitating the employment of
participants, the moderator introduces the rules for focus group conduction, emphasising the
following elements:
• Confidentiality of the information provided. The discussion is recorded (video and sound)
but the recorded material will only be used for elaborating the report and will not be made
public;
• The discussion will be approximately 90 minutes;
• All participants should listen to each other respectfully but they are encouraged to
formulate comments regarding what the others are saying. Interventions and interruptions
are possible;
• Participants speak clearly and loudly enough, one at a time.
• Because of the time limitation, the moderator will interrupt the participants in case of
diversion and will keep the conversation close to the main line.
• Participants express their personal opinions and wishes in a way that will not harm or
affect in a negative way the others.
When everyone has confirmed agreement with these principles and rules, the moderator
starts the focus group session.
The focus group discussion
INTRODUCTION OF PARTICIPANTS: If participants did not meet before, the moderator opens
the discussion, giving the floor to each participant for a short introduction.
NOTE: The moderator might need to pay special attention to encouraging some participants to
speak and will make sure that all get to intervene. It might be necessary to adapt the
formulation of the questions in order to ensure that they are well understood also by
participants with lower educational background.
The moderator will then address the following questions:
1. First of all, I would like to learn your opinion about employment. Is it difficult to get an
employment for you these days? If yes, what makes it difficult?
2. What kind of job would you like to have? Why?
3. Share experiences about when you tried to get a job. How did it go?
4. What do you think employers are looking for? What kind of employees they want to have?

5. What do you think you need in order to improve your chance to get the job you want?
6. What can you do to obtain that? Who can help?
7. Do you have experiences of working on the black market / without registration or official
documents? What are the disadvantages of doing this?
8. Have you thought about setting-up a small business or joining together with other people
to set up a social cooperative or another structure that could give you the possibility to work
and earn an income officially?
CLOSING OF DISCUSSION: The moderator has to give the opportunity to participants to ask
questions (if they have any). After that, the moderator has to thank for the participation and
specify again that the work together in the project will continue.
NOTES:
The suggested sequence of the questions is not mandatory. Changes and flexibility in the order
of asking are allowed. It is important to discuss all questions and each participant should have
the opportunity to express her opinion for each of the questions. In the case that participants
start the discussion on a question that is not asked yet, the moderator will not ask that
question again.
TAKING NOTES OF MAIN ELEMENTS IMMEDIATELY AFTER THE END OF THE DISCUSSION: The
discussion will be recorded, but it is important that the moderator does a short memo with the
main accents of the discussion; points out and writes key quotations that could be used in the
analyses; sketches out briefly his/her impressions of the participants – their involvement in the
discussion and their reactions to specific topics, as well as any other elements that are not
grasped by the recording.
Reporting
For the report on the competencies of the women in each group, we suggest the following
format:
1. Brief introduction describing the group and the process of assessing the competencies
2. Competencies inventory for each group member, using the following structure:
a. Level of formal education / basic reading and writing competencies in all
known languages
b. Acquired social competencies
c. Lacking social competencies
d. Other acquired competencies (e.g. technical skills, like cooking, computer
skills, specific types of knowledge)
3. Summary analysis with conclusions leading to the training needs.
The second section can have the format of a table, like the one below:

Name:
1. Formal education and basic competencies
a. Formal education 8 grades Comments (e.g. local school)
b. Reading skills in ……. Basic / average / good Comments
c. Wring skills in ….. Basic / average / good Comments
Acquired social competencies
a. competence Illustrated by
b.
c.
d.
Lacking social competencies
b.
Other competencies
b.
c.
Considering the explanation above, we do not provide a list of competencies to pick from, as
this can be a misleading process. We propose to each evaluator to go through a stage of
identifying from the interviews
- What participants know regarding social interactions (with members of their family,
with people in the community, with other people)
- What participants know to do in terms of effective interactions with other people
- Significant attitudes participants have towards the interaction with other people.
The analysis should focus on both correcting lacks of important social competencies and on
expanding and building upon competencies already acquired.
References
Council of Europe (2002) COMPASS – A manual for Human Rights Education with Young
People, www.coe.int/compass
Ingrid Schoon (2009) Measuring social competencies, German Council for Social and Economic
Data, Working Papers Series
European Commission (2007) Report of the High Level Advisory Group of Experts on the Social
Integration of Ethnic Minorities and their Full Participation in the Labour Market
FamCompass: Assessing and Validating Family Competences, http://www.famcompass.eu
Workalo - The creation of new occupational patterns for cultural minorities: the gypsy case.
CREA, University of Barcelona, http://www.neskes.net/workalo/teoriaan.htm
Kuger and Dunning (2009) Unskilled and Unaware of It: How Difficulties in Recognizing One's
Own Incompetence Lead to Inflated Self-Assessments, Psychology, no 1, 2009 pp 30-46

Methodology for the evaluation/assessment of competences

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Methodology for the evaluation/assessment of competences

Ähnlich wie Methodology for the evaluation/assessment of competences (20)

Mehr von EUDA_ERA

Mehr von EUDA_ERA (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Methodology for the evaluation/assessment of competences