F S K N Phase I Program Evaluation 03 03 10

Spring
FSKN Phase I
Program
Evaluation
Authors External Evaluator Formative Feedback and
Dr. Christine Geith Dr. Flora McMartin
Dr. Leslie Bourquin Recommendations for Phase II
Karen Vignare

FSKN Phase I Program Evaluation 2010

Table of Contents

Executive Summary ....................................................................................................................................... 4
Building Open Educational Resources to Support Learners in Food Safety through Industry University
Collaboration................................................................................................................................................. 6
1.0 Introduction ........................................................................................................................................ 6
2.0 Food Safety Knowledge Network (FSKN) ............................................................................................ 6
2.1 Building the FSKN Pilot Group and Supporting Documents ........................................................... 6
2.2 Developing FSKN Curriculum Materials .......................................................................................... 8
2.3 Creating Processes for OER Development ...................................................................................... 9
2.5 Using OER in the Food Industry (Corporations) ............................................................................ 10
2.4 Challenges in Implementing OER in a Corporate Setting ............................................................. 10
.
3.0 FSKN Pilot Trainings .......................................................................................................................... 11
3.1 Chennai, India – September 2009 ................................................................................................. 11
3.2 India August 2009 – January 2010 ................................................................................................ 11
3.3 Cairo, Egypt – November 2009 ..................................................................................................... 12
3.4 Shanghai, China – December 2009 ............................................................................................... 12
3.5 Overall Summary ........................................................................................................................... 12
4.0 OER Support Technology .................................................................................................................. 12
5.0 Findings and Recommendations ....................................................................................................... 13
5.1 Project Management Observations and Recommendations ........................................................ 13
5.2 External Evaluator’s Observations and Recommendations .......................................................... 15
5.3 Summary and Conclusions ............................................................................................................ 18
Appendices .................................................................................................................................................. 19
A. Timeline of Meetings .......................................................................................................................... 19
B. Basic Level Competency Framework .................................................................................................. 20
C. Inventory of FSKN Resources .............................................................................................................. 30
By Topic | English ................................................................................................................................ 30
By Topic | Chinese .............................................................................................................................. 31
By Event | Pilot Programs ................................................................................................................... 31
By Event | Coca‐Cola Food Safety Conference (December 2009) ...................................................... 32

2 Michigan State University


D. OER Policy & Procedures and FSKN Toolkit ........................................................................................ 33
1. Michigan State University Open Educational Resources (OER) Policy & Procedures ..................... 33
2. Food Safety Knowledge Network (FSKN) Toolkit ............................................................................ 37
E. DC Roundtable Forum Attendees ....................................................................................................... 39
F. Status Update on Pilot Projects – February 2010 ............................................................................... 45
Chennai, India – September 2009 ....................................................................................................... 45
India Online Pilot – August 2009 – January 2010 ............................................................................... 46
Cairo, Egypt – November 2009 ........................................................................................................... 46
Shanghai, China – December 2009 ..................................................................................................... 47
G. OER Technology .................................................................................................................................. 48
1. Review of the FSKN Correlate Module ........................................................................................... 48
.
2. Integrating DiscoverEd with: Software Specification Document Draft 0.5 ..................................... 49
H. Logic Model and Program Charts ....................................................................................................... 54
FSKN Phase I Logic Model ................................................................................................................... 54
Overall Initiative Program Chart ......................................................................................................... 55
Phase I Planning Grant Program Chart ............................................................................................... 56



Executive Summary
This report summarizes the experiences and progress made during the Food Safety Knowledge Network
(FSKN) planning grant period. The findings and recommendations herein are based on data collected
over the course of the grant period (November 2008 – January 2010), including evaluation reports of the
project activities (attached as appendixes to this document), formative evaluation of the planning
process and outcomes as conducted by the project PIs and staff, as well as observations from the
external evaluator.

The purpose of the planning grant was to create an international network of people and resources
within the food industry to enable and support the development of industry‐led Open Educational
Resource (OER) for members of the Food Safety Knowledge Network. Led by a collaboration between
Michigan State University (MSU) and the Global Food Safety Initiative (GFSI), the resulting OER and FSKN
will be the basis for the creation and provision of high‐quality, low‐cost training and education around
issues associated with food safety for developing countries.

The funded activities can be grouped around collaboration between university and industry support for
FSKN, development of OER materials, and development of a competency‐based OER delivery platform.
While collaboration is critical for building support and awareness, there must be clear, consistent, and
constant communications to keep the partnership moving forward with stated goals. Developing
materials to support FSKN was similar to other curriculum efforts, but making OER policies, procedures,
and licensing understandable to corporate partners required creating documentation and having
knowledgeable staff to answer questions. The OER platform used the principles of open source and built
on work from the Creative Commons DiscoverEd project to provide content resources organized by
topic and competency.

The following were outputs of the planning grant:

Competencies
• 14 requirement areas
• 90 competency objectives
• 264 assessment questions

Resources
• 13 topic areas covered
• 8 different file output types
• 3 languages (English, Chinese, Arabic)
• 200+ individual outputs
• 6 partnerships with content providing organizations



Pilots
• 5 pilot trainings (1 still in progress)
• 320+ total pilot participants
• 66 participating companies

Media
• FSKN Website‐ http://www.foodsafetyknowledgenetwork.org

OER Documentation and Software Code
• 2 source codes and documentations (FSKN DiscoverEd and Drupal Correlate Module)
• 1 OER toolkit for potential content partners
• 1 OER policy/procedure form (for MSU)

Meetings and Presentations
• 1 roundtable forum
• 4 FSKN Technical Working Group meetings
• 4 FSKN Competency Subgroup meetings
• 2 informal planning meetings
• 5 conference presentations



Building Open Educational Resources to Support Learners in Food Safety
through Industry University Collaboration
1.0 Introduction
This report summarizes the experiences and progress made during the Food Safety Knowledge Network
(FSKN) planning grant period. The findings and recommendations herein are based on data collected
over the course of the grant period from November 2008 through January 2010, including evaluation
reports of the project activities (attached as appendices to this document), formative evaluation of the
planning process, outcomes as conducted by the project PIs and staff, as well as observations from the
external evaluator.

The purpose of the planning grant was to create an international network of people and resources
within the food industry to enable and support the development of industry‐led Open Educational
Resources (OER) that will provide individuals access to learning resources that are aligned with
competencies. These competencies are based on recommendations from an industry committee, which
provided targeted corporations with the basic level requirements for food manufacture. Led by
Michigan State University (MSU) and the Global Food Safety Initiative (GFSI), the resulting OER and FSKN
will be the basis for the creation and provision of high‐quality, low‐cost training and education around
issues associated with food safety for developing countries and developing businesses.

While the industry‐led collaboration commenced the OER effort, several other important activities were
part of the planning grant. To create the resources to support the competencies, OER processes and
workflow, use of DiscoverEd, and the creation of an open source competency database module all were
technologically combined in a powerful website. The findings and the resulting codes and
documentations from this project can serve as a guide and model for other endeavors focused on
creating and searching topic‐specific OER.

This report, organized chronologically, discusses the activities that were accomplished during the grant
period and the resulting outcomes associated with each activity. Observations and reflections on the
activities and formative evaluation from the participants provide the basis for program improvements.
Many of these improvements and next steps have been incorporated into the future plans for the
network.

2.0 Food Safety Knowledge Network (FSKN)
2.1 Building the FSKN Pilot Group and Supporting Documents
Prior to the start of this grant period, MSU met with the Global Food Safety Initiative’s (GFSI) Global
Markets Working Group and created a subgroup known as the FSKN Working Group. The Global Markets
Working Group is an officially recognized and sanctioned part of the GFSI Technical group. The FSKN
pilot group is a subset of the Global Markets Working group along with MSU. The members of FSKN
include: Kevin Swoffer (Chair of Technical Committee), Jan Kranghand (Chair of Global Markets and Sr.
Department Manager Quality Assurance, Metro Buying Group), Marc Cwikowski (Quality Manager for



External Engagement, Coca‐Cola), Catherine Francois (Director of GFSI), Bizhan Pourkomailian (Senior
Food Safety Manager, McDonald's Europe), Dominque Berget (Corporate Food Safety Director, Danone),
Dr. Leslie Bourquin (MSU), Dr. Christine Geith (MSU), Dr. Deepa Thiagarajan (MSU), and Karen Vignare
(MSU). The purpose of the FSKN is to further the goals of the GFSI through the development of

• voluntary food safety requirements for less developed businesses and small suppliers
• strategies for implementing the requirements internationally
• a set of competencies (associated with each requirement) to guide the development of training
processes, curriculum, and assessment directed at the employees of less developed businesses
and small suppliers.

To accomplish these goals, the Working Group began to develop a unique partnership between the
corporate and academic sides of the food industry. Working with its stakeholders at multiple meetings
held from December 2008 to January 2010 (see Appendix A for a chronological list of all meetings
associated with the project) the group co‐developed a set of competencies to be used both for the
Global Markets Working group and FSKN. Two separate committees working side by side co‐developed
the corporate and individual competencies. The FSKN group also met separately to validate the initial list
of competencies. In Amsterdam and Paris, the group devised and validated the initial set of associated
competencies with the Basic Level Requirements for Food Manufacture for less developed businesses
and small suppliers (see Figure 1).

Figure 1. GFSI Basic Level Requirements for Less Developed Businesses and Small Suppliers



In May 2009, the Global Markets Working Group approved the 14 Basic Level Food Manufacture
requirements for corporations, allowing the FSKN to begin work on the competencies 1 . The Basic Level
Requirements represent the first 30 percent of the total knowledge that the Global Markets Working
Group is addressing. The rest will be addressed in the Intermediate Level Requirements and the GFSI
Guidance Document. During this period, MSU initiated the platform to support the work of the FSKN.
The task of the FSKN, using the building blocks of the Basic Level Requirements for Food Manufacture
for less developed businesses and small suppliers, was to create a set of competencies for individuals
(statements of knowledge, skill, and behavior) associated with each requirement. The group identified
90 competencies that articulate the knowledge and skill sets an individual food safety manager should
have upon completion of training at the FSKN Basic Level. (See Appendix B for a listing of the individual
competencies.) The group was also tasked with designing and pilot testing training curriculum (learning
resources, materials, and assessment tools) regarding the competencies and requirements; the end goal
being that the training and resources would be made widely and freely available as Open Educational
Resources (OER).

2.2 Developing FSKN Curriculum Materials
From May through August 2009, the FSKN pilot team (Bourquin, Thiagarajan, and Swoffer) focused on
developing learning resources linked to the competencies. These initial learning resources served as
seed materials for all of the pilot training programs. The long‐term goal is to continue to complement
and localize these resources with additional ones from an ever expanding FSKN network. Examples of
these materials can be found at http://foodsafetyknowledgenetwork.org (see Figure 2).

Figure 2. FSKN Home Page and Competency Correlation Page

1
While the corporate set of requirements identifies 14 areas of knowledge, the FSKN, after beta testing them,
found significant overlap in the areas of control of food hazards. As are result, they combined them into a final set
of 13 requirements.



The learning resources (organized by the GFSI requirements) were first produced in English and made
available in the following formats: flash video, audio (mp3) video (mp4 for iPod), text transcriptions of
lectures, and presentation slides (Power Point, PDF, OpenOffice). Training manuals were also produced.
All formats of these resources (except for the transcripts and training manuals) have been translated
into Chinese and are currently being translated into Arabic. In addition, ten stand alone online learning
modules are in review and seven modules have been created from their storyboards in the Connexions
OER Repository (http://cnx.org). The learners from the first online pilot (see Section 3.0) preferred the
video files for iPod. (See Appendix C for an inventory of FSKN resources.)

While these resources were being created, the corporate members of the FSKN sought out training
venues for testing the curriculum materials in face‐to‐face delivery. Four pilot training programs were
planned and implemented. Three were conducted face‐to‐face in India, China, Egypt, and one was
conducted online with an audience based in India. Another online pilot is currently in progress also with
an audience based in India. In the online pilot, the make‐up of the FSKN gave the group the opportunity
to leverage the resources from other MSUglobal projects as well as build on the previous training efforts
of the corporations involved. (Details and findings from the pilot training programs are discussed
in Section 3.0).

2.3 Creating Processes for OER Development
While the FSKN pilot team began creating competencies, MSUglobal began creating instructional
documentation and materials to support the development of OERs. To accomplish the goals of
publishing OER, the internal team designed OER processes and procedures for collecting and creating
content. For creating content, it was integral to think strategically about instructional use of materials.
This meant moving beyond the original course proposed and into the level of individual learning
resources. Thus learning resources could be combined to create a full course for Basic Level Food
Manufacture, or they could be used separately if a learner desired to work on only a certain set of
competencies.

This kind of flexibility supports a scaffolding process, which allows learners in an online environment to
focus on the competency needed. However, the process also allows workshop trainers to use a
complete package or focus on a limited number of competencies. Building the separate sets of materials
was initiated through a capture process. The FSKN pilot team, with grant funding from another source,
had an opportunity to prepare a workshop for India (the Chennai workshop is discussed later). In
preparation for the workshop, MSU drafted an internal OER policy and procedures document
(see Appendix D). While the FSKN pilot team was well aware of OER, the policy and procedures were
designed to help new members of the FSKN and others at MSU to understand OER.

To speed initial capture of materials, the use of proprietary capture software Camtasia and Relay were
used to record presenters. The capture process allowed for four immediate file output types—Power
Point, flash video, audio mp3 and video mp4. The audio files were immediately transcribed to produce
text transcripts files. In addition, the Power Point slide presentations were made available as PDFs and
OpenOffice documents.



While capturing presentations is important for providing a base of resources for learning, it should be
noted that the internal team immediately started cataloging existing resources from training providers,
universities, and non‐governmental organizations to align with the FSKN. The FSKN pilot team reviewed
those lists and agreed on content that should be added to the website. The internal team prepared not
only the policy document to explain OER, they also created an FSKN Toolkit document to explain how to
contribute content (see Appendix D). Documenting the creation and publishing/sharing of OER materials
was developed as part of the planning grant.

2.5 Using OER in the Food Industry (Corporations)
The planning grant also provided funds to support the expansion and development of an international
knowledge network, the Food Safety Knowledge Network. Early on in the project, the FSKN pilot team
identified significant differences in the ways international food companies and academic institutions
approach training and education, especially with regards to OER. OER is a nascent educational concept
within academia as well as in US and international business and government agencies. The FSKN,
through this planning grant, was uniquely placed to introduce the concept to the international
community committed to food safety and to educate that community about its benefits and potential to
inform and reform training practices and opportunities.

Introducing the conceptual underpinnings and practices of OER informally, through discussions at
regular meetings of the GFSI members and FSKN, was not a strategy that sufficiently educated the
corporate members of the group. So, to more formally introduce OER to food safety leaders and to
initiate a discussion about how it might be used, the FSKN Pilot Team hosted a roundtable event in
Washington DC on September 28, 2009, attended by 65 food safety experts (see Appendix E for a list of
attendees).

The purpose of the event was to educate this group of stakeholders (members of US academic
institutions, certification bodies, international donor organizations, training organizations, corporations,
and US government organizations) about the goals and projects of the FSKN and to begin to ‘build out’
the network by inviting others to participate.

The timing of the event also made it possible for the FSKN pilot team to share the results of the first
pilot training held September 7, 2009 in Chennai, India. Holding the event after the first pilot effort,
allowed the team to give the group a concrete example of the kinds of training resources created and
tested, and to validate the curriculum with potential users of such resources. The team was also able to
demonstrate the OER side of the project, thereby introducing the audience to the concept and potential
benefits of OER to corporate training practices.

2.4 Challenges in Implementing OER in a Corporate Setting
The collaboration between MSU, GFSI, and the Global Markets Working Group brought to light the
differences in legal liability and a number of challenges such as communication that are unique to an
international collaboration among academic institutions, retailers, manufacturers, food service
companies, and service providers. While there was a common understanding around the requirements



and the competencies, there was not agreement around how best to translate these into curriculum
resources. The industry has developed since 2000 a requirements document which provides information
on what is required, but those documents were not provided as instructional materials by the
association. The creation of training and instruction was not something GFSI did and thus MSU was able
to provide faculty experts and instructional designers to create curriculum.

The corporations involved in the pilot projects (and GFSI) are bound by certain legal restrictions that
academic institutions do not experience and which call into question the nature of OER in this context.
European Union law holds food companies liable for any illness or injury caused by their products. They
are also liable for the work performance of the food professionals they employ. These liability issues,
once brought to light, have called into question whether these organizations can fully participate as
partners in FSKN to make the curriculum they use ‘open’ in OER terms, the terms commonly understood
by most institutions of higher education in the US that are protected from liability claims associated with
the information they provide.

The members of the collaboration have continued to work together to find a solution to this problem,
since all agree on the need for high quality, effective training and training resources in this field.
Currently, the curriculum resources will be made available on the FSKN website, but to avoid liability,
they will not carry a GFSI endorsement of any sort.

3.0 FSKN Pilot Trainings
From August 2009 to January 2010, the Food Safety Knowledge Network project conducted four pilot
training programs (three face‐to‐face trainings and one online training). The following sections provide a
summary of each training session (see Appendix F for more details regarding the assessment results).

3.1 Chennai, India – September 2009
• Face‐to‐face, three‐day training program conducted in partnership with USAID and led by: Dr.
Leslie Bourquin, Dr. Deepa Thiagarajan, and Kevin Swoffer .
• 74 participants from 25 Indian companies and 6 public sector organizations participated.
• 72 participants completed pre‐ and post‐tests to assess their knowledge. Overall performance
scores improved by 19% between pre and post‐tests. Analysis of scores by each of the 13 areas
under the Basic Level for Food Manufacture competency framework showed varying increases
in knowledge by competency.

3.2 India August 2009 – January 2010
• Online training program conducted in partnership with METRO Cash and Carry. Online
curriculum identical to Chennai training and available in varied of formats, i.e., flash video,
audio, video for iPod, PDF, etc.)
• 63 participants from 30 METRO Cash & Carry participated.
• 26 participants completed both pre‐ and post‐tests to assess their knowledge. The pre‐ and
post‐tests were the same tests used in the Chennai training program. Overall performance
scores improved by approximately eight percent between pre and post‐tests.



• Participants also completed surveys at the end of the experience to determine their satisfaction
with the online nature of the training. There was strong overall support for the value and
effectiveness of the online training. Participants also noted preference for audio resources over
all others.

3.3 Cairo, Egypt – November 2009
• Face‐to‐face training program conducted in partnership with Makro (METRO) and UNIDO. Train‐
the‐trainer approach used with Kevin Swoffer training the trainers. Training materials and
curriculum same as those used for Chennai training – materials translated into Arabic.
• 36 participants completed pre‐ and post‐tests to assess their knowledge. Overall performance
scores improved by 16 % between pre and post‐tests.

3.4 Shanghai, China – December 2009
• Face‐to‐face 4‐day training program/conference conducted in partnership with the Coca‐Cola
Company, and led by: Dr. Leslie Bourquin, Dr. Deepa Thiagarajan and Kevin Swoffer. FSKN
training modules based on the materials used in Chennai – materials translated into Chinese. In
addition to FSKN training materials, the entire training program included topics dealing with pre‐
harvest food safety, other prerequisite programs, and the ISO 22000 food safety management
system.
• 142 participants from the Coca‐Cola Company participated, including representatives from
bottlers, Coke customers, suppliers, and sub‐suppliers.
• 113 participants completed pre‐ and post‐tests to assess their knowledge. Overall results
demonstrated an improvement in scores by 6%. Note, these scores are not directly comparable
to other trainings because a short version of the test was used (70 questions instead of 132 in
previous versions of the test).

3.5 Overall Summary
The FSKN pilot activities in India, Egypt and China, all demonstrated the effectiveness of the training
programs to improve participant knowledge regardless of educational attainment, previous training, or
organization type. Groups with lower performance on the pre‐test tended to show the greatest
improvement. Analysis of the results from the testing is continuing in order to learn more about how to
improve the training and refine the learning materials.

The file formats the OER were created in played a significant role in the efficient reuse and remixing of
the materials. The editable file formats allowed the resources to be localized and adapted for various
audiences. Presentation slides were efficiently localized and translated for specific groups and videos
were bundled with audio to provide Flash videos in both English and Chinese.

4.0 OER Support Technology
While the website was not officially launched until mid‐February, 2010, preparation and planning was
done early in the project. MSUglobal and its MSU partner, Virtual University Design and Technology
(vuDAT), used open source technology to host the demonstration site for the OER network and created



an open source tool to map the OER to the competency frameworks
(http://foodsafetyknowledgenetwork.org).

Content management open source platforms were immediately investigated as potential solutions.
While many options exist, Drupal was chosen for its large community of developers and extensive list of
modules developed for various website features and functionalities. Drupal served as the base to build
from for accomplishing the tasks described in the planning grant. However, Drupal in and of itself did
not provide a competencies database module nor did it provide a method for searching and pulling
content to the website.

As a result, vuDAT programmers created an open source module that provides a way of creating a
competency database and correlating resources to individual competencies. The current website
demonstrates only the top level of competencies, but the module was built so that it could go five layers
into a hierarchy. This allows for much deeper detail within competencies and eventually a learner could
choose only a sub category of a competency to learn. While this level of sophistication was not quite
necessary when the planned three parts of FSKN become a reality, this will be a useful tool in the future
(see Appendix G for more details).

The other challenging part of developing the OER technology was determining how to best integrate
Creative Commons’ DiscoverEd search tool into the competency database. DiscoverEd provides an
enhanced search experience for users looking for educational resources by looking for materials from
selected curators and displaying valuable metadata on the resources. The competency database only
includes competencies while learning resources are culled to match a competency. This required writing
code to customize DiscoverEd so it would work with Drupal. It also required that both an RSS feed and
Open Architecture Index (OAI) be incorporated. These tools pull with an RSS or OAI standard from our
own and other websites materials to support learning of the competencies. In addition by adding OAI
especially for MSU materials, we have begun an education process which demonstrates the use of OAI
for content repositories. If a provider does not want to or cannot implement the OAI standard, they
have the option of uploading to a website that does use OAI (see Appendix G for more details.)

5.0 Findings and Recommendations
MSU has updated the logic model and program charts in Appendix H. Those charts show in an
abbreviated format the accomplishments and challenges faced during this planning grant.

5.1 Project Management Observations and Recommendations

University and Industry Collaboration and Support for FSKN
Observation
The intent of FSKN was to use OER as a method of low‐cost learning in an industry led collaboration with
MSU. This collaboration remains focused on the goals, but the collaboration encountered unexpected
challenges in working with a large association. While the director remained in place, the members and
goals of the GFSI changed as well as the members and goals of the working committees over the life of
the grant. However, by working closely with individuals of corporations who are members of the



association, the team was able to use the materials for training suppliers even before the formal launch
of FSKN, to use the materials for training suppliers in emerging markets.

Recommendations
1. All partnerships require clear, consistent, and constant communications to maintain successful
working relationship. An industry and academic partnerships often speak about issues in specific
lenses, it is critical to write things down and provide examples.
2. It is important to engage companies with a need for training to be part of the project team as
they are likely to help pilot the materials and be the first to implement the OER approach.

Development of HighQuality/LowCost Training and Educational Curricula
Observation
Building FSKN OER curriculum is similar process to assembling any type of curricular materials, yet two
instructional design skills are also needed: understanding of OER licensing and workflow processes and
using a scaffolding approach built with either competencies or taxonomies as the framework.

Recommendations
3. The launch of Open Education Resources requires documentation and staff who can be clear
about Creative Commons licensing and who can build processes and workflow.
4. Building learning resources that allow learners to self‐select what information they are
interested in knowing requires instructional design staff members who understand how to
integrate scaffolding into the learning resource.

Development of an OER Delivery Platform
Observation
Many tools and technologies currently exist to create an open website using a repository and search.
Yet, there are very few examples of websites for specialized content. Employing a competency
framework and customizing DiscoverEd from ccLearn showcases the information technology
programming and processes which need to be adopted to effectively launch FSKN. While many other
websites use open source content management systems, competency frameworks or taxonomies, and
search, no other websites we found do all three for a specific content area.

Recommendations
5. Review open source software technologies and modules to see what exists from previous
projects and include staff members who understand open source, languages, and architecture
to choose a platform that can be customized.
6. Work closely with Creative Commons to determine what improvements are underway and then
revise code to write customization for search technologies.
7. Build an open module to work with platform which provides a way to linearly organize a
framework, like a competencies or taxonomies.



5.2 External Evaluator’s Observations and Recommendations

University and Industry Collaboration and Support for FSKN
Observation
The collaboration between MSU and GFSI, embodied by the FSKN, was critical to the development of
valid and agreed upon Basic Level Requirements and individual competencies and the eventual
implementation of pilot training projects. Involvement by the GFSI lent credibility to the work of the
FSKN and provided links to representatives from leading industries (Coca Cola and METRO Cash and
Carry) as well as other international organizations such as UNIDO. These links proved essential in terms
of setting up and running pilot training programs.

Recommendations
1. Build out from initial partner set to involved more corporate and international organizations
such as the World Trade Organization or the World Bank.
2. Continue to use, refine, and expand on this planning and development process of stakeholder
involvement to establish requirements and competencies for the Intermediate and Advanced
Requirement Levels.

Observation
Involvement of corporate and international stakeholders at the planning level make it possible to quickly
create venues for training, both face to face and online. Without this involvement, it would have taken
much longer and required significant effort on the part of the FSKN to set up and run trainings.

Recommendation
3. Find ways to recruit and bring into the process new stakeholder representatives to create new
training opportunities at the higher levels and/or to extend Basic Level trainings.

Observation
Stakeholder involvement in the planning process proved to be an effective means for educating the food
industry about OER. As a result of participating in the process, and learning about OER first hand, the
industry representatives became advocates for OER.

Recommendations
4. Find ways to involve first set of industry representatives to recruit new members to the planning
process.
5. Develop a means to involve industry representatives in educating their peers about OER. These
representatives can be powerful allies and can speak directly to the challenges and
opportunities offered by OER and food safety training.

Development of HighQuality/LowCost Training and Educational Curricula
Observation
The FSKN team members, Prof. Leslie Bourquin and Prof. Deep Thiagarajan of MSU, and Kevin Swoffer,
consultant, designed the original training curriculum, made modifications for each instantiation of the



training, led the bulk of the in‐person trainings and implemented the assessment of the trainings from
design to analysis. This process is not sustainable over the long run.

While during the pilot phase of the project, it is important to establish and maintain high quality control
in the development of the curriculum and its implementation and assessment; this type of work takes
an enormous amount of effort and resources to maintain over the long‐term. The barriers reflected in
this model, i.e., lack of adequate resources or reliance upon volunteerism or people who will undertake
it as a ‘labor or love’ may doom growth of the effort.

If the training is to grow in scope and become more available to more food industries, this training
process must be modified to allow others to learn the process and implement it on their own, thereby
reinforcing the purpose of the materials being OER.

Recommendations
6. Develop and make available learning materials for trainers (beyond the training manual),
supporting their efforts to conduct training on their own. There is much to be learned from the
online training conducted in India and the train‐the‐train model utilized in Egypt. These models
should be examined carefully and lessons learned applied to developing materials for trainers,
not just trainees.
7. Develop online social networking and a community site for trainers. This type of support will
help make materials more accessible as well as be a valuable aid to others who wish to develop
their own materials. It is also an essential component to ensuring quality of trainings, not just
the materials.

Observation
The assessment of the pilot trainings offers much needed insight into the learning of food safety
competencies; however, it is but one step in the process of determining the impact of that learning.

Recommendations
8. Examine the validity and reliability of the shorter, 70‐item assessment instrument as compared
to the 132‐item instrument. Should the 70‐item instrument stand up, make it available as the
main assessment instrument.
9. Based on findings from Recommendation 8, consider creating an item bank of tested
assessment items (or instruments) for use by trainers. The item bank could provide trainers with
more flexibility in terms of matching assessments to the workshop curriculum and how they use
it. (Please note, in this recommendation and others, the evaluator assumes reuse of the OER is a
goal, thus derivative versions of all materials will be used. This has already been the case as the
curriculum developers changed materials for each pilot training conducted during the planning
phase.)
10. Design and conduct a longitudinal study regarding the impact of trainings on changes in practice
of the participants.



11. Expand assessment of training to include issues such as: improved or increased access to
learning materials, changes in what and how it is learned. Include specific industry concerns in
the assessment, e.g., is training more efficient and what is the return on investment of such
training?

Observation
In order to grow, the project (and website) need to attract trainers who will make derivative works
based on the existing materials or develop new works to meet the standards but sensitive to their
unique environment or audiences. Currently, there is no mechanism to ensure the quality of these
materials. High quality materials are of special significance given the need for credentialing in the field,
the high health and safety risks involved to the customers of the training participants, and the liability
issues associated with GSFI that have already been identified.

Recommendation
12. The high standards in this field do not seem to readily meld or mesh with the philosophy and
goals of OER. The program leaders, in collaboration with industry leaders need to make creating
an overlap between the two a priority.

Explore multiple methods of review to devise a quality control system for this audience and
within this environment. Several models exist: for example, professional societies, academic
associations and trade organizations all employ multiple models for quality assessment. While
perhaps not directly applicable in this situation given some of the challenges, they should be
examined for potential use. For example, the peer review system employed for health related
professional societies (medicine, dentistry, etc.) may be particularly pertinent.

Development of an OER Delivery Platform
Observation
The site is still in the development phase of its life‐cycle. As such, not much is known about how its users
and potential users interact with it or hope to interact with it.

Recommendations
13. Implement user testing of the site. This should be done in several stages, the first being in‐
person observation of the site by intended users (or proxies of such users.) The next stage might
involve creating a user panel of intended users for more longitudinal study of the site. This panel
would be a useful way to track use as users become more sophisticated in their use of the site.
14. Implement a web‐tracking system to identify how all users access materials and use the site.
This is also an essential piece in conducting a gap analysis to determine those content areas that
require more materials or in determining features that might require refinement or identifying
new features to improve user experience.



Sustainability
Observation
Like all OER projects the most significant challenge is that of sustainability. The question of how the
project will continue to exist and grow must begin to be addressed, even at this early stage in its
development.

Recommendation
15. The collaboration with the GSFI has led to the development of requirements at both the
industry and individual level. Curriculum and training methods have been developed that
demonstrate increased knowledge about the requirements that may show to have a strong
return on investment in the training. The benefits to industry and the value to their own training
efforts may be the starting point for exploring funding of the project by industry. Clearly, liability
issues must be addressed, but these are not insurmountable. Key to the process will be
demonstrating, in and on business terms, the value of the project to their continued success in
food safety.

5.3 Summary and Conclusions
The FSKN planning grant demonstrated the use of OERs within an industry‐university partnership. To
complete Phase II of the grant, it will be critical to recognize that cultivating partnerships takes time.
While many content partners, including corporations, are willing to contribute resources, it is very time
consuming to explain OER, secure and create learning materials, and have them reviewed. The process
is indeed quite doable but until the project has developed a viral pull, it is critical to maintain and
support a coordinator like MSU to keep FSKN growing and thriving.

It is noteworthy that the FSKN project has generated a very high level of interest among donor agencies
and others working in emerging markets. We anticipate that the immediate next steps for the FSKN
project include 1) formal launch of the FSKN website, 2) expansion of FSKN training activities to other
regions, and 3) in conjunction with the GFSI Global Markets Working Group, expansion of the program
to Intermediate Level for Food Manufacturing and Basic Level for Primary Production.



Appendices

A. Timeline of Meetings

December 2008 (Paris) Official FSKN working group launched

February 2009 (Barcelona) Informal meetings of FSKN at GFSI conference

February 2009 (Amsterdam) Competency Development Meeting

March 2009 (Paris) Competency Development and FSKN Update Meeting

May 2009 (Chicago) Informal FSKN meetings at GFSI Technical Committee meetings

June 2009 (Lansing) Development Team working on Individual Competencies

August 2009 (Paris) Official FSKN Update Meeting

September 2009 (Chennai) FSKN Workshop

September 2009 (Online) Metro E‐Learning pilot Launched

September 2009 (DC) FSKN OER Launch meeting

November 2009 (Egypt) FSKN training with UNIDO

December 2009 (Shanghai) Coca Cola Food Safety Conference (includes FSKN) training

January 2010 (Paris) FSKN Update meeting



B. Basic Level Competency Framework

Competency Area Food Safety Manager Competency
and Criteria Competency

Specifications Technical and legal The individual will describe the principles and
knowledge reasons for meeting established specifications
The organization shall ensure of raw materials, ingredients, product contact
that specifications are materials and finished goods.
adequate, accurate and shall
Legal knowledge & The individual will explain how to access and
ensure compliance with
customer apply information (legal and customer
relevant food safety and
requirements requirements).
legislative requirements.
Technical and legal The individual will explain the process for
knowledge updating specifications based on changes in
legal and customer requirements.
Technical and legal The individual will maintain knowledge of food
knowledge safety requirements which will affect
specification information.
Writing skills The individual will construct written
specifications for raw materials, ingredients,
product contact materials and finished goods.

Traceability Technical and legal The individual will describe the principles and
knowledge reasons for product traceability.
The organization shall
establish a system which Legal knowledge & The individual will explain their awareness of
enables the identification of customer the regulations and customer requirements
product lots and their relation requirements associated with product traceability.
to batches of raw materials, System development The individual will collate data on origins of
primary and consumer unit raw materials, usage of ingredients and
packaging materials, product contact materials.
processing and distribution
records. Records shall include: System development The individual will design a system (i.e. lot
identification of any in coming identification, finished product coding) that
product or ingredient, identifies the relationship and linkage between
complete records of batches finished goods and raw materials, ingredients,
of in‐process or final product, product contact materials and process
and record of purchaser and conditions.
delivery destination for all System development The individual will describe how to manage
product supplied. the system (i.e. implementing and record
keeping).



Incident Management Technical and legal The individual will describe the principles and
knowledge reasons for incident management.
The organization shall have an
Technical knowledge The individual will identify product
effective product withdrawal
issues/complaints with respect to consumer
and recall procedure for all
safety.
products, which shall include
the provision of information to Technical knowledge The individual will analyze the significance of
the customer. product issues/ complaints with respect to
consumer safety.
Technical knowledge The individual will explain the actions required
in the event that product issues/complaints
warrant action by the company.
Legal knowledge & The individual will describe their awareness of
customer the regulations and customer requirements
requirements associated with the management of incidents.
System development The individual will design a product quarantine
system for product within control of the
company.
System development The individual will explain how to manage a
product quarantine system for product within
control of the company.
System development The individual will design a product recall
system.
product recall system.

Control of Non‐Conforming Technical and legal The individual will describe the principles and
Product knowledge reasons for control of non‐conforming
product.
The organization shall ensure
that any product, which does Legal knowledge & The individual will identify the regulations and
not conform to requirements, customer customer requirements for non‐ conforming
is clearly identified and requirements product.
controlled to prevent Legal knowledge & The individual will identify the regulations and
unintended use or delivery. customer customer requirements for control and
requirements reporting of non‐conforming product.
Technical knowledge The individual will explain the significance of
product quality in relation to non‐conforming
product.
Technical Knowledge The individual will analyze the food safety risks
associated with non ‐conforming product.



Technical Knowledge The individual will explain the options
available for disposition of non‐conforming
product.
System development The individual will explain how to design a
product quarantine system.
system necessary for proper evaluation and
disposition of non‐conforming product.

Corrective Action Technical and legal The individual will describe the principles and
knowledge reasons for corrective actions as they relate to
The organization shall ensure process, product and system non‐
that corrective action be conformities.
undertaken as soon as
possible to prevent further System development The individual will describe a process for
occurrence of non‐conformity. corrective action that provides links to other
system processes.
Legal knowledge & The individual will identify the regulations and
customer customer requirements associated with
requirements corrective actions.
Technical knowledge The individual will identify the possible causes
of non‐conformities.
Technical knowledge The individual will identify the options for
corrective action based on sound risk‐based
decision making.
Technical knowledge The individual will analyze the significance of
food safety risks in determining corrective
action.
Technical knowledge The individual will explain how to check that
corrective actions taken have been effective.
Technical knowledge The individual will explain the difference
between correcting a problem and taking
corrective action to prevent the problem from
reoccurring.

Personal Hygiene Technical and legal The individual will describe the principles and
knowledge reasons for effective personal hygiene (i.e..
The organization shall have hand washing, personal cleanliness, illness,
processes and procedures in injury, personal behavior, and visitor control).
place to ensure the
implementation of employee Legal knowledge & The individual will explain how to access
and visitor personal hygiene customer regulations and customer requirements
requirements governing effective personal hygiene (i.e. hand



practices. Such practices shall washing, personal cleanliness, illness, injury,
result in the sanitary handling personal behavior, and visitor control).
and delivery of safe and
Technical knowledge The individual will explain when to apply
quality products to customer.
proper hand washing techniques.
The Codex Alimentarius
Commission's Technical knowledge The individual will identify employees with
recommendation on personal symptoms of illness (i.e. fever, vomiting,
hygiene shall be followed. diarrhea, sore throat) so they can ensure that
ill individuals who might pose a risk to food
safety are not allowed to work with food.
System development The individual will design a system for
employees and visitors to notify the
organization of any symptoms of illness and
injury that may preclude them from working in
direct contact with food.
System development The individual will operate a system for
employees and visitors to notify the
organization of any symptoms of illness and
injury that may preclude them from working in
direct contact with food.
System development The individual will design a system to enforce
procedures in relation to illness and injury,
personal cleanliness, personal behavior which
could result in the contamination of food and
food safety provision for visitors.
System development The individual will operate a system to enforce
procedures in relation to illness and injury,
personal cleanliness, personal behavior which
could result in the contamination of food and
food safety provision for visitors.

Facility Environment Technical and legal The individual will describe the principles and
knowledge reasons for maintaining the site and facility to
The organization shall ensure ensure the production of safe food.
the site or facility shall be
maintained so as to prevent Legal knowledge & The individual will explain their awareness of
contamination and enable the customer regulations and customer requirements
production of safe products. requirements governing the facility environment.
Technical and legal The individual will describe the criteria to be in
knowledge place within a food premises to ensure food
safety, with particular reference to: power
supplies, fuel supplies, lighting,
heating/chilling (environmental), toilet



provision, washing (personal hygiene)
facilities, and fabrication maintenance.
Technical and legal The individual will describe the requirements
knowledge in relation to surfaces and finishes for: work
surfaces, food equipment, fittings and utensils,
walls, floors and ceilings.
Technical and legal The individual will describe the requirements
knowledge for effective workflow arrangements to avoid
cross contamination.
Technical and legal The individual will describe the need for
knowledge regular and reliable maintenance, including
prevention of contamination during
maintenance.
knowledge suitable storage facilities in order to avoid
cross contamination and temperature abuse.
knowledge suitable access and egress arrangements with
particular reference to product incoming and
outgoing, worker access, and emergency
services.
knowledge suitable waste storage and/or disposal
facilities.
System development The individual will design a system to assess
the effectiveness of the maintenance of the
site and facilities.

Cleaning and Disinfection Technical and legal The individual will describe the principles of
knowledge and reasons for cleaning and disinfection.
appropriate standards of Technical and legal The individual will describe regulations
cleaning and disinfection shall knowledge governing cleanliness requirements.
be maintained at all times and Legal knowledge & The individual will describe the regulations and
throughout all the stages. customer customer requirements associated with use of
requirements cleaning and sanitizing chemicals.
Technical and legal The individual will describe the appropriate
knowledge use (i.e. concentration, contact time,
temperature, pH, contamination level,
microbial targets) of chemicals to ensure
appropriate levels of cleanliness.



System development The individual will design a system for suitable
cleaning and disinfection i.e. develop (i.e..
who, what, when, frequency, how).
System development The individual will operate a system to
manage (i.e.. implementing, monitoring,
corrective actions) and maintain (i.e.
effectiveness, continuous improvement) a
documented operational procedure for
identifying that appropriate cleaning and
disinfection procedures are in place.
Technical knowledge The individual will explain how to assess (i.e.
visual, record review, or other methods) the
effectiveness of cleaning and disinfection
practices that have been applied.

Product Contamination Technical and legal The individual will describe the principles and
Control knowledge reasons for ensuring product contamination
control.
Legal knowledge & The individual will explain regulations and
appropriate facilities and
customer customer requirements governing the control
procedures are in place to
requirements of product contamination.
control physical, chemical, or
biological contamination of Technical knowledge The individual will explain the steps taken to
product. Appropriate controls minimize the risk of contamination by physical,
shall be in place to minimize chemical and biological contaminants.
incidence of contamination by
foreign bodies. System development The individual will design a system to ensure
the control of any possible product
contaminants.
System development The individual will operate a system to ensure
the control of any possible product
contaminants.

Pest Control Legal knowledge & The individual will describe regulations and
customer customer requirements relating to pest
The organization shall ensure requirements control.
controls are in place to reduce
or eliminate the risk of pest Technical and legal The individual will explain the hazard posed by
infestation (including rodents, knowledge specific food pests and the controls required
insects and birds). to reduce the risk to the product.
Technical and legal The individual will explain the preferred
knowledge habitat of relevant food pests with particular
reference to the company's facility.
Technical and legal The individual will explain the environmental,



knowledge physical and chemical methods of preventing
and controlling pest infestations of the facility.
Technical and legal The individual will explain the system used to
knowledge monitor the effectiveness of pest controls and
define corrective actions.
System development The individual will design a system to minimize
the risk of pest infestation of the product and
facility through the inspection and control of
raw materials, factory cleanliness,
maintenance and specific control measures.
System development The individual will operate a system to
minimize the risk of pest infestation of the
product and facility through the inspection
and control of raw materials, factory
cleanliness, maintenance and specific control
measures.

Water Quality Technical and legal The individual will describe reasons for
knowledge ensuring water quality.
Legal knowledge & The individual will explain the regulations and
the quality of water, ice, or
customer customer requirements associated with water
steam in contact with food
requirements quality, ice and steam.
product shall be suitable for
intended use at the facility. All Technical and legal The individual will explain the characteristics
food contact water, ingredient knowledge of properly constructed water wells (where
water, and water used in applicable) and plumbing systems necessary
cleaning and sanitizing for delivery of potable water, ice, and steam.
operations shall be
determined to be from a System development The individual will design a system to ensure
potable source. the quality of water, ice and steam.
System development The individual will operate a system to ensure
the quality of water, ice and steam.

Control of Food Hazards: Technical and legal The individual will describe the principles and
General & Specific knowledge reasons for controlling food hazards.

The organization shall reduce Technical and legal The individual will explain the nature of food
the risk of unsafe food by knowledge safety hazards (biological, chemical, physical
taking preventative measures and allergens) and factors influencing the
to assure the safety and likelihood of their occurrence (e.g. conditions
suitability of food at an which influence food safety hazards commonly
appropriate stage or stages in associated with specific ingredients used,
their operation by controlling specific processes undertaken, and specific
food hazards. The organization foods produced) (customer requirements and
legislation).



shall control food hazards Technical and legal The individual will explain the recognized and
through the use of systems knowledge   known control measures and how to apply
such as HACCP. They shall: these to control hazards relevant to specific
undertake a risk analysis to products (take into consideration local
identify all possible hazards, regulatory compliance and customer
identify any steps in their requirements).
operations which are critical
to the safety of food, Technical and legal The individual will design the monitoring
implement effective knowledge   procedures necessary to ensure control of
procedures at those steps to food safety hazards relevant to the product.
defined limits as appropriate Technical and legal The individual will perform corrective actions
for food safety, monitor knowledge   when control measures are not achieved.
control procedures to ensure
their continuing effectiveness, Technical and legal The individual will operate a system for
and review control procedures knowledge   maintaining comprehensive records in relation
periodically, and whenever to the control of identified hazards.
the operations change.

Control of Food Hazards: Technical and legal The individual will describe the principles and
Allergens knowledge reasons for controlling food allergens.

The organization shall ensure Legal knowledge and The individual will be able to explain
there are adequate control customer regulations and customer requirements
measures in place to prevent requirements   governing the presence and control of
cross contamination of allergens.
allergens. All ingredients Technical and legal The individual will explain the recognized and
known to cause food allergies knowledge known control measures and how to apply
in the product shall be clearly these to control food allergens within the
identified and communicated facility (take into consideration local
to the customer. regulatory compliance and customer
requirements).
Technical and legal The individual will design the monitoring
knowledge procedures necessary to ensure control of
food allergens.
Technical and legal The individual will perform corrective actions
knowledge when control measures are not achieved.
Technical and legal The individual will operate a system for
knowledge maintaining comprehensive records in relation
to the control of allergens.

Personal Skills Interpersonal Skills The individual will describe how to effectively
interact and communicate verbally with
company management, regulatory agencies,
customers, suppliers, and employees.



Team Management The individual will describe how to effectively
Skills manage collaborative activities of
interdisciplinary teams to address matters
pertaining to food safety and quality.
Teaching , Employee The individual will describe how to effectively
Development teach employees about appropriate practices
critical to maintenance of food safety. (This
teaching could include topics such as
appropriate personal hygiene, effective
methods for cleaning and disinfection of
equipment and food contact surfaces,
methods for control of potential food
contaminants, methods for monitoring of food
hazard controls, etc.)
People Management The individual will describe how to effectively
Skills manage employees to ensure they are
following appropriate practices necessary for
production of safe food.
Reading The individual will interpret appropriate legal
Comprehension and technical documents pertaining to
product safety, legality and quality. (This will
include documents such as food laws,
regulations, guidance, best practices related to
food safety, customer requirements and
customer specifications.)
Systems The individual will describe how to develop or
Development modify the organization’s food safety control
systems to comply with new requirements
related to emerging laws and regulations,
relevant guidance, customer requirements,
emerging science and the organization’s Basic
Level requirements.
Analytical Problem The individual will identify problems, consider
Solving options for resolution and adopt the correct
action to resolve problems pertaining to the
organization’s Basic Level requirements.
Writing The individual will describe how to effectively
write technical and non‐technical documents
and communications related to relevant food
safety requirements or practices and customer
requirements.
Observation The individual will describe how to effectively



monitor employee practices to ensure
compliance with company policies and food
safety requirements.
Assertiveness The individual will describe how to effectively
communicate food safety requirements to the
organization’s management and employees,
and ensure these requirements are addressed
and applied through policies and practices.
Self Motivation The individual will describe how to
independently access and apply knowledge,
such as new research, information on
outbreaks/incidents or other findings relevant
to the safety of the organization’s products.
Leadership The individual will define the correct decisions
to safeguard the safety of the consumer, even
if the outcome of these decisions might carry
economic costs, potentially cause other
adverse consequences, or otherwise be
discouraged by management.


F S K N Phase I Program Evaluation 03 03 10

F S K N Phase I Program Evaluation 03 03 10

Recommended

Recommended

More Related Content

What's hot

What's hot (12)

Viewers also liked

Viewers also liked (9)

Similar to F S K N Phase I Program Evaluation 03 03 10

Similar to F S K N Phase I Program Evaluation 03 03 10 (20)

More from Karen Vignare

More from Karen Vignare (15)

F S K N Phase I Program Evaluation 03 03 10