1. Agent Architecture for
Simulating Norm Dynamics. Part II
Rosaria Conte
LABSS (Laboratory of Agent Based Social
Simulation), Roma, ISTC-CNR

2. Why to comply?
How to make norms happy?

3. Levels and dynamics of
immergence
 Norms may “sink” into
the mind
 Various levels
 Full norm processing
Violation/
 Internalization compliance
• Internalized N-goal Full norm processing
• Independent of norm
enforcement belief N-internalization
• Internalized goal
• Independent of N--belief N-incorporation
 Incorporation
• Automated behaviour
 Bidirectional dynamics

4. A non-rigid modular
architecture
NORM
RECOGNITION
INPUT N-B1 NORM DECISION: CONFORM
B-B2 ADOPTION: BEHAVIOR
N-
N-B3 N-GOAL
INTENTION
Socrates
acceptance of
sentence
Internalization

5. A non-rigid modular
architecture
NORM
RECOGNITION
INPUT N-B1 NORM DECISION: CONFORM
B-B2 ADOPTION: BEHAVIOR
N-
N-B3 N-GOAL
INTENTION
Stop at red light
Mind
commands Answer requests if
eye contact is
established
Internalization Incorporation

6. Norm internalization

7. Classic and current views
 Classic conditioning: external sanctioning system never
completely abandoned (Scott 1971).
 Internal sanctions (Parsons, 1952; Scott, 1971; Axelrod,
1986, Elster, 1989; Coleman, 1990; Dequech, 2009; etc):
 Related to emotions of discomfort, shame and guilt
 Sense of duty
 Thoughtless conformity:
 The more agents have conformed to a norm in the past,
the more they will redo it in the future (Epstein, 2007).
 Why, when, how???
 What about retrieving deliberation in norm-compliance?

8. In EMIL follow-ups
(Conte et al., 2011; Villatoro et al., 2010)
 Formulate hypotheses about
 effects, factors, and evolution of internalization
 reversible dynamics
 Implement essential mechanisms in EMIL-A.
No emotions at this stage
 Check hypotheses by simulation

9.  Internalized N-goal:
goal no more
relativized to N-
EMIL-Internalizer
enforcement belief.
 N-enforcement belief
(EMIL-IA)
still persists but agent
does not calculate it I N-bel1:general
when decide to n
execute N-goal Internalized
p N-bel2:pertinence N-goal
 Internalized goal: u
t
goal no more N-bel3: enforcement
internalized to
normative belief in
the general form. I N-bel1:general
 Internalizer has lost n
normative origin of p Internalized
N-bel2:pertinence goal
her goal. u
t
 N-belief still persist N-bel3: enforcement
but agent pursues
goal irrespective of it.

10.  Internalized N-goal:
goal no more
relativized to N-
EMIL-Internalizer
enforcement belief.
 N-enforcement belief
(EMIL-IA)
still persists but agent
does not calculate it I N-bel1:general
when decide to n
execute N-goal Internalized
p N-bel2:pertinence N-goal
 Internalized goal: u
t
goal no more N-bel3: enforcement
internalized to
normative belief in
the general form. N-bel1:general
 Internalizer has lost I
normative origin of n Internalized
N-bel2:pertinence goal
her goal. p
u
 N-belief still persist t N-bel3: enforcement
but agent pursues
goal irrespective of it. Bel4

11. Factors of internalization
 Why form a normative goal irrespective of external
sanctions?
 Several factors, among which
 Consistency with previously internalized norms
(McAdams, 2008) and temporal discount (see below)
 “Internal sanctions” (Scott, 1971; Axelrod, 1986, etc).

12. Consistency with previous
norms
 Why observe norms irrespective of
external sanctions?
 Consistency (McAdams, 2008),
 Successful educational strategies (King,
2008) link new inputs to previously
internalized norms
 Policymaking: Controversial efficacy of
frightening announcements (Goodall,
2005): Inability to act under
representation of delayed reward due to
hyperbolic discounting (Bickel and
Johnson, 2003; Rachlin, 2000), i.e. effort
hyperbolically decreasing with time-
distance from goal-attainment,
• leads to procrastination
• Better refer to consistent norms and
values

13. Internal sanctions
“ The internal sanction of duty is a feeling in our mind; a pain, more or less
intense, attendant on violation of duty; it is the essence of conscience.”
(John Stuart Mill)
When. ground the mechanism on salience.
How:
Anticipatory representations of consequences (supported by
emotions)
tend to replace exter:al sanctions.
Why: the higher the salience,
The more the sanctioners
The more frequent and severe the sanction is observed/received
The more justified the sanction is perceived.
The more often it is expected
The more often it is anticipated, simulated in imagination
But the confine between
what is experienced in anticipating future sanctions
current sanction becomes fuzzier over time…

14. Internal sanctions
“ The internal sanction of duty is a feeling in our mind; a pain, more or less
intense, attendant on violation of duty; it is the essence of conscience.” (John
Stuart Mill)
When. ground the mechanism on salience.
How:
Anticipatory representations of consequences (supported by
emotions)
tend to replace external sanctions.
Why: the higher the salience,
The more the sanctioners
The more frequent and severe the sanction is observed/received
The more justified the sanction is perceived.
The more often it is expected
The more often it is anticipated, simulated in imagination
But the confine between
what is experienced in anticipating future sanctions
current sanction becomes fuzzier over time…

15. Internal sanctions
“ The internal sanction of duty is a feeling in our mind; a pain, more or less
intense, attendant on violation of duty; it is the essence of conscience.” (John
Stuart Mill)
When. ground the mechanism on salience.
How:
Anticipatory representations of consequences (supported by
emotions)
tend to replace exter:al sanctions.
Why: the higher the salience,
The more the sanctioners
The more frequent and severe the sanction is observed/received
The more “justified” the sanction is perceived.
The more often it is expected
The more often it is anticipated, simulated in imagination
But the confine between
what is experienced in anticipating future sanctions
current sanction becomes fuzzier over time…

16. Internal sanctions
“ The internal sanction of duty is a feeling in our mind; a pain, more or less
intense, attendant on violation of duty; it is the essence of conscience.” (John
Stuart Mill)
When. ground the mechanism on salience.
How:
Anticipatory representations of consequences (supported by emotions)
tend to replace exter:al sanctions.
Why: the higher the salience,
The more the sanctioners
The more frequent and severe the sanction is observed/received
The more justified the sanction is perceived.
The more often it is expected
The more often it is simulated in imagination
But the confine between
what is experienced in anticipating future sanctions
What is experienced during the effective sanction
becomes fuzzier over time…

17. Dynamics of
N-internalization
N- Delibe
inter rative
N-salience N-salience N-
naliz compli
ation ance

18. Dynamics of
N-internalization
N- Delibe
inter rative
N-salience N-salience N-
naliz compli
ation ance

19. Natural experimental
hypotheses
Natural experiments
Internalized goals lead to a more vigorous goal-attainment (Bargh
et al., 2001) than originally inner goals because of effect of
prospect theory (Kahneman and Tversky, 1979; Abdellaoui et
al., 2007)
Internalizer shows higher intolerance for norm violation based on
confirmation bias (Nickerson, 1998; Sternberg, 2007)
Internalizers are much better at defending the norms than
externally-enforced observers based on cognitive dissonance
(Festinger 1957).
However, internalizers are sensitive to salience

20. Artificial experimental
hypotheses
Artificial experiments
Compliance is more robust if norms are
internalized because independent of
external sanctioning entities.
Less costly, allowed evolution
Sensitivity to salience makes internalization
flexible and allows for norm innovation

21. Effects of internalization in artificial societies
(Villatoro et al., 2010)
• Three interacting subpopulations :
• Strategic
• Normative (EMIL-A)
• Internalizers (EMIL-IA)
• NA may internalize the N.

22. Hypotheses
• Strategic and Normative agents lead to a
collapse of cooperation with low punishment
rates
• Internalizers maintain social order imposed by
social norms even if these are not defended
• Temporarily, allowing the system to recover from
possible failures;
• Permanently: when salience decreases internalizers
start dismissing the norm and do cost-to-benefit-
calculation

23. Experimental scenario
• Agents play a repeated Prisoner’s Dilemma.
• Norm specifies “Cooperation”.
• Agents can punish those that do not follow the norm,
with a given probability.
• Depending on salience value:
• Strategic punishment: Fine.
• Normative punishment: Fine + Deontic Message.

24. Simulation sequence
3rd Phase: Update values
• Norm salience
• Cooperation rate
• Internalization rate

25. Details
• Strategic Agents:
• Do not know about punishment. They only have a
benefit cost global perception of the interaction
• Normative Agents:
• They are initialized with a risk-taking value. They will
defect when they detect that probability of being
punished is below 30%.

26. Experimental design
• 100 Agents.
• Fully Connected Network.
• Results averaged on 25 runs.
• Different punishment distributions:
• Constant: agents are assigned a constant probability to punish.
• Linearly Increasing: probability continuously increases as the simulation
runs.
• Linearly Decreasing: the probability continuously decreases as the
simulation runs.
• Step Down: at a certain moment of the simulation, the probability of being
punished drops from 1 to 0, i.e. from total punishments to no punishment.
• Step Up: at a certain moment of the simulation, the probability of being
punished raises from 0 to 1, i.e. from no punishment to total punishments.

27. Results: Linearly Decreasing
Cooperation Rates Internalization Rates
Salience

28. Results: Step down
Cooperation Rates Internalization Rates
Salience

29. Experimental lessons
• Allowing different types of normative agents to interact
provides system policymakers with a tool that can help
predict the dynamics of prosocial behavior.
• Internalizers are endowed with a rich cognitive
architecture allowing high cooperation rates even when
punishment is low (and costly).
• Internalizers need a certain amount of strategic or
normative agents to dismiss the normative action when
necessary.

30. Conclusions
 Summary
 Building blocks of a rich cognitive
model of internalization
 Types and degrees of internalization
 factors favoring different types of
internalization
 First implementation (EMIL-IA)
 First experimental results on effect of
internalizaiton on cooperation rates
 But “Only the barest outline of
internalization is known" (Vygotzsky,
1978, p. 57).

31. Conclusions (2)
 Norme emerge from agents endowed with mental representations
and the capacity to act on them
 A simpler agent architecture does not allow for norms to emerge
in certain domains
 But to make norms happy, cognitive but non-fully deliberative
agent architecture is needed
 Further artificial experiments,
 more than one norm
 more interesting scenarios
 Necessity of cross-methodological experiments
 Implement incorporation
 Implement emotions

32. To practice (and discuss)
 Internalization with > one norm