1. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
1
Vincent Ostberg, student no. 049096670
PPA 402 Policy/Program Planning and Evaluation
Ryerson University
Instructor: Peter Constantinou, Ph.D.
EVALUATION OF FNTI/RYERSON STUDENT SUPPORT SERVICES
PURPOSE
To evaluate the student support services provided to students in the Public
Administration and Governance program with level 1, level 2, and level 3 courses in
comparison to student support services offered to Aboriginal students offered by
Lakehead University enrolled in Political Science. The specific purpose of the research is
to determine whether the student support and mentorship offered by First Nations
Technical Institute (FNTI) and Ryerson University staff enables the students to achieve
student success in completion of the three levels. This evaluation must use equivalency
of programming to be effective and it is pertinent that the students are in similar
programs although they are in different institutions.
There will also be a program available to the FNTI/Ryerson course instructors
and support staff to partake in additional training. The evaluation will disclose the
results of the program effectiveness for support services provided to FNTI/Ryerson
students in the Public Administration and Governance program for all levels.
BACKGROUND
The First Nations Technical Institution is an education service provider that
caters to mostly First Nations students that are adults and working professionals who
are upgrading their education while working in their current professions. FNTI is in a
partnership with Ryerson University to deliver distance education program like Public
Administration and Governance and other programs by other educational institutions. A

2. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
2
number of students have already worked in various administrative jobs from First
Nations tribal councils, local bands, and regional non-governmental organizations.
The seventh student intake has produced approximately seven or eight graduates
who will receive their diploma as of spring 2012. This intake of students started in the
fall of 2006 with nearly thirty students that utilized video conferencing in three sites:
Thunder Bay, Timmins, and southern Ontario First Nations. All but two Thunder Bay
students dropped out of the program within one to two years, some students stopped
from the Timmins site but returned later, and most of the southern Ontario students
have completed all three levels.
Upon completion of eight courses in level 1 the student earns a certificate. Upon
completion of eight courses in level 2, the student receives an advanced certificate, and
upon completion of seventeen courses, one which is called the Practicum amounts to two
courses, the student earns a degree in Public Administration and Governance from
Ryerson University. As a result of the seventh intake with its graduation numbers and
students who dropped out, FNTI has hired the consulting firm to conduct a review on the
student support services offered by FNTI and Ryerson University.
In addition, the contractor also contacted Lakehead University officials for
permission to conduct a study concerning Aboriginal student support services with their
institution and to compare it with FNTI/Ryerson. Also, the contractor will contact the
department head overseeing the FNTI/Ryerson Public Administration and Governance
program to arrange for FNTI/Ryerson staff to partake in proposed program to improve
student support services. The contractor will ensure that the initial draft of the report

3. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
3
and research will include FNTI’s goals and objectives for student support services and its
attribution to student academic success.
PROPOSED EVALUATION METHODOLOGY
DESIGN 5: Comparing student academic marks/student support services
The contractor will use Design 5 for comparing First Nations students who are taking a
comparable program called Political Science that is offered by Lakehead University
which is located in Thunder Bay, Ontario. The FNTI/Ryerson staff and students will be
the experimental group and the students at Lakehead University in varying years from
year 1, year 2, and year 3 will the experimental group. The LU students will be assessed
with the FNTI/Ryerson students to compare their academic success that is attributed to
the student support services that are provided. By doing so, the review will be more
powerful because it shows a wider scope of evaluation by comparing three groups of
students.
The students who will be used in the research will be the intake of students who are
starting level 1 in the fall of 2012 with FNTI/Ryerson as well as Aboriginal students who
are taking year one of Political Science at the Lakehead University. In addition, level 2
students in FNTI/Ryerson will be compared to year 2 Aboriginal students in Political
Science, and level 3 FNTI/Ryerson students will be compared to year 3 Lakehead
University students.
The effectiveness of the student support program offered by Lakehead will be evaluated
by the following components:
 Participants attitude to their learning environments

4. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
4
 Learning and academic performance. Students from both institutions will sign
consent forms for the release of academic transcripts.
 Student support offered by FNTI staff and Ryerson University instructors to enable
students to pass their courses evaluated through questionnaires.
 First Nations students who are taking the Political Science program in Lakehead will
also be issued questionnaires to evaluate the student support services offered by
Lakehead University.
A summative evaluation will be completed following the research which will occur
over six months. If FNTI/Ryerson’s course calendar is on par with Lakehead students,
the evaluation will use students who are registered for two to four courses from
September 2012 to the end of February 2013. Students will be issued a questionnaire at
the beginning of the evaluation and another questionnaire will be available at the end of
the research.
First Nations Technical Unit/Ryerson University staff members and students will be
the experimental group also known as the E-group. Lakehead University students will be
known as the non-equivalent control group also known as the C group.
Definition of Design 5: This is a time series evaluation with a non-equivalent control
group. Two groups of people are measured regularly before the program of which the
FNTI/Ryerson group will receive the additional training for student support. By utilizing
students and staff at six different levels, the contractor will increase the power of the
evaluation.

5. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
5
DESIGN 1: Evaluation of a new program aimed at improving student support
services
For assessing the staff at FNTI/Ryerson and Lakehead University in the Public
Administration and Governance and Political Sciences program. Random questionnaires
will be distributed to each location to randomly select which staff will be evaluated.
50% of the instructors are evaluated at FNTI/Ryerson University are to be evaluated
who are delivering courses in all three levels. They will be evaluated in self-reflection
questionnaires and to be evaluated by the students. Lakehead University staff and
students will also be issued the same questionnaires which are to be completed by
random by 50%. There will be no evaluation on the staff by their student in the Political
Science program.
50% of the staff selected at random with FNTI/Ryerson who are delivering the Public
Administration and Governance program, including instructors and program
coordinators, will be assessed and take the opportunity to work with the Ryerson
student support services division for four sessions on how to mentor students with
academic writing, online library research, research, and mentoring of staff and students.
The staff selected within FNTI/Ryerson will consist of an E-group, the ones who take this
supplementary training and the C-group consisting of staff who don’t take the training
offered by Ryerson University. A summative report will be included within the final
report.
Definition of Design 1: True control with pretest-post-test design, both groups are
randomly assigned but one group will utilize student support training, this is considered
to be a powerful test in evaluation.

6. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
6
DELIVERABLES/WORK PLAN
Evaluation Work Plan
The consultants will prepare and submit a detailed evaluation plan and the proposed
methodology for FNTI’s review and approval at the start of the contract. The work plan
should describe activities and provide a schedule for implementation.
The consultants must also hold two working sessions with the FNTI/Ryerson personnel
and Lakehead University personnel and students. The initial working session should be
held upon arrival in location as selected by FNTI to solicit input for the evaluation plan.
The contractor will provide periodic briefings and reports to FNTI on an as needed basis.
Team Composition and Duration
The duration of the evaluation mission should be up to six months. The consulting
firm will provide up to five key personnel for this task. The team should include a team
leader and up to four additional education specialists. The team leader is responsible for
the overall management of the contract team's evaluation effort.
Specific responsibilities should include: consulting with FNTI’s board of directors at
the beginning and key intervals of the program evaluation; coordinating and supervising
team members and major evaluation activities, including documentation review,
interviews, development and distribution of questionnaires, compilation of data, analysis
and formulation of recommendations; oversight of the team work plan and ensuring that
all the tasks required of the team are satisfactorily accomplished; completing and
submitting the final report.
All team members should have at least 5 years work experience in public
administration and possess strong proficiency in research and data compilation and have

7. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
7
First Nations cultural sensitivity. They must have a minimum academic training at the
graduate or at the Masters level.
Two consultants will work with FNTI/Ryerson staff and students and two
consultants will work with Lakehead staff and students. The team leader will fly back
and forth and meet with the field consultants in the preparation of the final report. The
field consultants will be on site to conduct research and it should be noted that the
consultants will receive funds for hotel, travel, and meals along with their fees.
ANTICIPATED PROBLEMS
The anticipated problems that could affect this research are the students who are in
the C group and E group could drop out of the program. Another problem that we may
incur is whether there are First Nations/Aboriginal students in the Political Science
program at the varying years. We also anticipate that FNTI might find our proposal a bit
costly if we implement design 1 for evaluation of FNTI/Ryerson staff and Lakehead
University staff’s Aboriginal student support services. We also anticipate that there
could be setbacks as staff members may get sick, may find other employment, or feel
defensive about the evaluation process. Our research team may also suffer setbacks for
reasons beyond their control outside the evaluation process.
SOLUTIONS TO THE ANTICIPATED PROBLEMS
If there are no First Nations/Aboriginal students in Lakehead’s Political Science
program, we could substitute for another program and try to make it as close as possible
to the program offered by FNTI/Ryerson University. Another option is to compare
student support services by utilizing another institution like Confederation College who
have their own Public Administration and Goverance program. The difference is that

8. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
8
Ryerson offers a degree to their graduates and Confederation College offers a diploma to
their graduates. With regard to the staff that may feel defensive about the evaluation
our team are prepared to assure them that this is not a performance evaluation and that
they understand that the purpose of this evaluation is made clear to them. If one of our
field consultants experiences a setback due to personal reason, the team leader will fill
in.
COST-EFFICIENTCONSIDERATIONS
In the event that FNTI finds the cost of implementing design 1 is to costly we will
change the design to design 6. Design 6 will use the same requirements as Design 1
except that there will be two evaluations on the E group with no true control group
which would be Lakehead University Aboriginal support services staff and instructors.
Further options to reduce cost will be to shorten the evaluation from six months to four
months. By selecting Design 6 with Design 1, the team leader will eliminate four trips to
the GTA and save on hotel and travel costs. FNTI also has the option of selecting either
dual design model at 2/3 the cost although the evaluation will be reduced from six
months to four months.
Definition of Design 6: This is a before and after design which the FNTI/Ryerson group
receives the training where two evaluations occur, at the beginning and end of the
program. Although the power is decreased in the evaluation, this is still a viable option.
FINAL REPORT
The final report will list the action taken by the consultants and the work plan will be
drafted at the start of the evaluation but if there are revisions done to the evaluation, this
will be documented and the final work plan and modifications will be documented and

9. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
9
the final report will include a revised budget from the original as well as a revised tabled
work plan.
The report will contain information as to what FNTI’s goals were in achieving
student success to their students and the evaluation will also show whether the
implemented program to the FNTI/Ryerson staff made any improvements to student
support services and student success. The consultants will determine whether FNTI met
its goals and objectives in the final report. It can be argued that student success is
attributed to student support services and the evaluation will provide a summative
report on the findings. The final report will be made available to FNTI/Ryerson as well
as Lakehead University or another institution in lieu like the Confederation College in
Thunder Bay, Ontario.

10. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
10
PROPOSED PROJECT SCHEDULE
Timetable for conducting and completing research
TIME LINE SEP
–
FNTI
SEP-
LU
OCT-
FNTI
OCT -
LU
NOV -
FNTI
NOV -
LU
DEC -
FNTI
DEC -
LU
JAN
-
FNTI
JAN -
LU
FEB -
FNTI
FEB
-
LU
ACTIVITIES
Interview FNTI/Ryerson
and LU staff: Political
Science instructors and
Aboriginal student
support services
X X X X
Development of
questionnaires
X X X X
Distribute
questionnaires to
students
X X X X X X X X X X X X
**Evaluate
FNTI/Ryerson and
Lakehead University
staff members –
randomly assigned - for
Aboriginal student
support services –
**Design 1 to be used,
alternate will be Design
6.
X X O O O O X X
Monitor student
academic transcripts
and marks
X X X X X X X X X X X X
Prepare report for
FNTI/Ryerson
X X X X X
Prepare report for
Lakehead University
X X X X X
Final report for
FNTI/Ryerson and
Lakehead University
X
Project team:
Team Leader: Vincent Ostberg
Thunder Bay Consultants: Sean Batise, Lydia Big George
Greater Toronto Area (GTA) consultants: Barbara Miller and Byron Millette

11. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
11
Proposal fees:
1. Team leader fee: $500 X 24 days $12,000
2. Four field consultants: $300 each per day X 6 trips X 2 days $14,400
3. Travel for team leader: 12 return trips from Thunder Bay
to GTA X $900 $10,800
4. Travel for field consultants in Thunder Bay:
6 trips for Timmins based field consultant (FC) $6,000
Mileage is issued for 600 km for Thunder Bay FC $300
5. Travel for field consultants in GTA:
2 trips for two field consultants travelling to FNTI locations $2,000
One field consultant is based in Brantford and the other in Hamilton.
Trips will be taken to Brantford, Orillia, and GTA, this includes overhead costs of travel.
6. Hotel costs for team leader is $130 per night X 24 nights $3,120
7. Hotel costs for GTA field consultants: 12 nights X $130 $1,560
Totals costs of project
Option 1, Full evaluation at six months with Design 5 and 1: $50,180.00
Option 2, Partial evaluation of option 1 at four months at 2/3 rate: $33,453.33
Option 3, Full evaluation at six months with Design 5 and Design 6: $44,420.00
Option 4, Partial evaluation of option 3 at four months 2/3 rate: $29,613.33

12. Evaluability assessment of the Public Policy program at FNTI/Ryerson University
12
EVALUATION OPTION SELECTED
First Nations Technical Institute selects Option ___. By selecting this option, FNTI agrees to
pay 50% of the project fees up front and will pay the 35% of the project midway through
and the final 15% will be paid to the contractor/team leader upon completion of the
evaluation and delivery of the final report. The total amount payable will be
$_____________________.
Date: Sunday April 8, 2012.
Signed by:
__________________________________________________________________
First Nations Technical Institute Chairperson
__________________________________________________________________
First Nations Technical Institute Executive Director
__________________________________________________________________
Vincent Ostberg, Ostberg Consulting Services, Ltd.