A

1. The Promise & Threats of Narrative Research September 2011

3. To ensure its staff have the skills necessary to do this, it trains them

4. The suite of training programs includes four core units

6. We undertook six activities to put us in a position to evaluate this Organisation Development program suite We Did a document review of program materials Met with & explored the content and intent of the 4 programs with client design & presenting staff Did an international review of similar programs Ran a workshop exploring the organisational objectives for each program and the full suite. Set up Academic Directors and undertook initial observation of each program Established a narrative capture data collection and analysis process based on 4 captures roughly associated with the work of Kirkpatrick and Brinkerhoff We then began to collect data and run the Feedback Loops that would enable the Organisation to better achieve its goals

7. Hybrid Kirkpatrick – Brickerhoff framework

8. BRICKERHOFF’S Success Case Evaluation Method Low end High end. This method examines participants who, after being surveyed turnout to be the least impacted by the training, these he interviews to understand why. The other end of this method examines participants who, after being surveyed turnout to be the most successfully impacted by the training, these he interviews to understand why. The MBS Mt Eliza Impact Evaluation System simultaneously does the Survey, manages the full statistical analysis of the results, plots these in multiple formats and has the data on which these are based immediately available for analysis without any interviewing.

9. The 70-20-10 Development Philosophy This philosophy defines development as occurring in 3 main ways: through on-the-job experiences; relationships, networking & feedback; and formal training opportunities. The “70-20-10” refers to how you should allocate professional development:- 70% of all development should occur on-the-job,- 20% from relationships, networking and feedback, and - 10% from formal training opportunities.We believe that most companies stress the 10%...this creates a problem related to achieving outcomes from training. So we wanted to try to get a better balance through our learning about how the training programs worked for the organisation.

10. Question to discuss….What are your frustrations or challenges in evaluating complex initiatives?

11. A Children’s Party ……

13. emergence – not predicatable

14. principles NOT prescriptive rules

15. many possible outcomes- nor ONE right answer

16. Too many factors to make predictable connections and predict futures

19. narrative based – narratives hold and convey complex meaning

20. mass capture for patterns, insights and decision support

21. minimise role of ‘experts’ filtering data capture and analysis

24. Language tailored to target group

26. Language tailored to target group

27. Provide basis for tracking relativenature of experiences over time

28. Established as double negative slider barsDemographics General research questions

30. Improving workplace effectiveness

32. Provoke people's memories

33. Allow the question to be answered in the third person

34. Do not privilege a certain type of experience over another

37. Signification - Polarities: Opposing opposites to uncover layers of meaning Ideal state Absence of… Excess of… The questions are put in terms relative to two extremes of the area under consideration – an “absence of” on the left hand side and “excess of” on the right hand side. By sliding the ball to where they think their experience lies in relation to the two extremes, greater insight into the responses can be achieved when compared to a simple “good / bad” linear scale.

38. Signification – Multiple Choice: The person who provides the experience decides what it means Multiple choice questions are also used to get the context of the experience from the participant. These questions can then be used as the lens through which the aggregate of all responses can be viewed. For example, multiple choice question could be used to view all responses from participants whose intent in telling their experience was to “inform” and about the personal outcomes associated with “flexibility and adaptability”

39. SenseMakerTM A look under the cover

40. Hybrid Kirkpatrick – Brickerhoff framework

41. What did we want to know more about?

42. What insights did we find? Collaborative learning Overwhelmed withinsufficient time toshare experiences Mostly useless & theyshould have shut up Level of engagement (others) Intense &obsessive Detached & indifferent Performance as an adult learner Just along forthe ride Accepting everythingwithout question Facilitator / Presenter Great fun & entertaining,but I did not learn Too intimidating &dismissive for me to learn Group functionality - collaboration Having no concernfor anyone but themselves So polite no onedared to challenge Ideal State Safe learning environment Over protected Too scared totry anything new Quality of Facilitator / Presenter Indifferent Inhibiting Level of engagement (self) Avoiding beingnoticed Centrestage Boundary management – the social contract A free for alldominated by a few Limited & restricted by too many rules Self responsibility Someone else’s fault Learnedhelplessness

43. What insights did we find? Cohort 1 Self responsibility Cohort 2 Cohort3 Cohort 4 All cohorts Learnedhelplessness Someone else’sfault Roleplaying I think that the roleplays had the most impact - all well and good to theoretically teach new consulting skills but experience in putting them into practice I found far more effective in making the messages stick. While it seemed a little forced initially, once into full swing I found it valuable to get to deal with different possible behaviours of clients in interview and group meeting/presentation scenarios and then receive feedback on how it all went. It was also obviously easier to try things a little different and outside my confort zone in a controlled environment. Role plays The aspect that had the biggest impact on my learning was the mock interviews with partners. At first I felt nervous and a little intimidated but as we went through the 4 different interviews I realised that it was a safe environment in which to make mistakes and developed more confidence to give different interview techniques a go. I learnt a lot about how to manage difficult clients and will be less concerned about this if the issue arises in practice. By the end, I was very happy with my presentation and felt that I had overcome my fears to an extent.

44. What insights did we find?3 months back in workplace: Behaviours

45. What insights did we find?3 months back in workplace: Behaviours

46. Question for Group Discussion: What challenges in using this approach do you think we faced?

48. Measuring program impact over time is not appropriate for all organisations

49. Upfront commitment to actively participate in collection, analysis and outcomes

50. A core team that changes needs new members being effectively inducted

51. Transparency and access to staff essential

52. Process is not a “one shot” deal

53. Need for assessment of program across all four stages to measure impact using feedback loops to improve programs

55. Senior leadership sponsorship and support

56. A core team that will stay with the project

57. Transparency of programs – access to all staff

58. Commitment of time and resources for analysis and intervention

60. Viv Read viv@emergingoptions.com.au

61. Chris Fletcher chris@emergingoptions.com.au

62. Reference sites

63. www.emergingoptions.com.au

64. www.cognitive-edge.com