1. EGLS3
EGLS3 - Evaluation for Greater Learning Student Survey System
Implementation at Houston Community College
Dr. Alan Ainsworth Dr. Mac Adkins
Former HCC Faculty President
Senate President SmarterServices LLC
Mario Heredia Jeff Worford
Director Research & Lead Developer
Support Services SmarterSurveys™ Product
February 23, 2012
1

2. EGLS3
Houston Community College
A comprehensive community college serving the Houston area
Over 75,000 students credit, non credit in Fall 2011
One of highest number of international students of all community colleges in the country
2

3. EGLS3
Student Evaluation of Instruction – History
Paper-Based Student Evaluation of Instruction (SEOI) survey has been in place for 27
years, revised incrementally.
For Face to face classes, the SEOI was the HCC’s paper method to administer the student
evaluation
For online classes Distance Education used their own online survey
Structure. Four part-time staff plus a supervisor working for two months.
Large work room
Given the current technology, Institutional Research sought
-Online administration methodologies
-New Instrument
3

4. EGLS3
HCC’s SEOI Survey Paper Distribution System
IR manually prepared and routed more than 5,000 packets and 100,000 SEOI instruments each
semester.
Each packet contains the SEOI instrument and instructions for faculty and proctors.
The packets were sent to their respective campuses administrators.
Campuses administrators were responsible for distributing the materials to the Instructors and collecting
from the proctors (who administer the survey to the students).
Once IR received the packets, entered them manually in the system to track the return rate.
IR cleaned the packets, scanned the surveys and generated data file. The report was run in SPPS.
The final report was saved on disks and sent to the College Administrators for manual distribution.
Students’ hand written comments were sent back to faculty.
Process is cumbersome, labor intensive and costly.
SEOI’s means report results were not used systematically in the evaluation process.
Return rates have been an issue, but in Spring 2010 the rate of return increased to 95% and the
response rate was of 73%.
4

5. EGLS3
Student Evaluation of Instruction (SEOI)– Paper Base Instrument
5

6. EGLS3
EGLS3 Student Evaluation Survey Instrument
Houston Community College
EGLS3 – Evaluation for Greater Learning Student Survey System
Organization and Explanation of Materials
Strongly Disagre Strongly
Agree
Agree e Disagree
1. My Instructor explains difficult material clearly.    
1. My Instructor communicates at a level that I can
   
understand.
1. My Instructor makes requirements clear.    
1. My Instructor identifies relationships between and
   
among topics.
Learning Environment
Strongly Disagre Strongly
Agree
Agree e Disagree
1. My Instructor establishes a climate of respect.    
1. My Instructor is available to me on matters pertaining
   
to the course.
1. My Instructor respects diverse talents.    
1. My Instructor creates an atmosphere in which ideas
   
can be exchanged freely.
Self-Regulated Learning
Strongly Disagre Strongly
Agree
Agree e Disagree
1. My Instructor gives assignments that are stimulating
   
to me.
1. My Instructor encourages me to develop new
   
viewpoints.
1. My instructor arouses my curiosity.    
1. My Instructor stimulates my creativity.    
Overall Opinions
Strongly Disagre Strongly
Agree
Agree e Disagree
1. I like this instructor.    
1. I am interested in this subject.    
1. I think the classroom was appropriate for this class.    
1. I would recommend a course taught by this
   
instructor.
General Information
1. This class is:
 A Required Course
 An Elective
 I am not sure
1. What grade do you expect to earn in this course?
A
B
C
D
F
6

7. EGLS3
Important EGLS3 Features from UNT’s SETE
These are the factors or elements of teaching effectiveness that are being measured:
Factor 1: Organization and explanation of materials
Factor 2: Learning Environment
Factor 3: Self-regulated Learning
Four questions are used in the EGLS3 validity analysis. Scale scores (the weighted responses)
are reported for each of the three domains. An overall score is also computed
The following are sample scale ranges for EGLS3
7

8. EGLS3
SEOI-EGLS3 Survey Instrument Comparison
Student Evaluation of
Instruction(SEOI) EGLS3 - Survey Instrument
• Paper-Based • Is implemented online to allow
• Home Grown Survey flexibility, transparency, and
• Included non-instructional Questions accountability
• Library and course content questions • Provides a measure of teaching
• No Validated effectiveness as perceived by students.
• Means report • Allows college-wide comparison in key
• IR cleaned the packets, scanned the areas
surveys and generated data file. The • Facilitates student evaluations of their
report was run in SPPS. instructors (not the course or the
• The final report was saved in a disk and content)
sent to the College Administrators. Hand- • The EGLS3 scale scores can be used to
written comments returned to faculty. make assessments for continual
• Process was cumbersome, labor intensive improvement of the institution
and costly. • Results available on line for individual
• The results were not used in a systematic faculty.
evaluation process. • User built their own reports
8

9. EGLS3
Early Stages
Vice Chancellor of Planning and Institutional Effectiveness has Faculty and OIR representatives
visit University of North Texas to discuss SETE.
HCC Administration and Faculty Senate decide to move forward with assessing SETE for
possible online evaluations.
9

10. EGLS3
Faculty Governance and Endorsement
Faculty Senate President requests the evaluation of SETE for replacement of SEOI by
Educational Affairs Committee
Faculty Senate sees reports of SETE and votes unanimously to support pilot program
for Spring 2011
10

11. EGLS3
Seeking Additional Endorsement
Approval of Pilot from Vice Chancellor of Instruction, Deans’ Councils.
Forums held for information and feedback
11

12. EGLS3
Student’s evaluation process, the transition from paper to online
IR Partnership with Faculty Senate
IR Responsibility: Develop and implement the Operational plan
Faculty Senate President: Navigating Faculty Governance and Endorsement
Pilot Implementation Spring 2011 . Two Locations were selected
Outcome: Response Rate 20%
Follow up: Student Focus Groups
Findings: Login process cumbersome
12

13. EGLS3
Solutions for Log-In Process found with PS Integration
HCC - SmarterServices People Soft login Integration of EGLS3 - Student Center
13

14. EGLS3
Student’s evaluation process, the transition from paper to online
HCC - SmarterServices People Soft login Integration of EGLS3 - Faculty Center
14

15. EGLS3
Integration and Responsibility of IR in EGLS3 Implementation
Institutional Research liaison between SmarterServices and the HCC departments participating
in the EGLS3.
Develop timeline and monitor progress
Develop and review documentation with SmarterServices, IT/Costumer Support
Develop and review communication plan with Communications Department.
Several media channels used to notify the institution: meetings, email, paper, social media,
web page, plasma screens, posters, etc.
15

16. EGLS3
Fall 2011 - Full Implementation Facts
Only Regular Term and Second Start Classes surveyed, including Distance Education and
Dual Credit
Data submitted: 6,460 classes with 140,800 enrollment.
System runs smoothly
23,005 responses with a Response Rate of 16%
Technical Issues
Pop –ups blocked in College Computer Labs and other computers.
System Slow due to lack of services running on the web server.
Reporting
Available on our Web Site
Faculty, Administrator access
Administrators can Build their Own Reports
Follow up
Faculty Focus Groups
16

17. EGLS3
FALL 2011, Implementation
Full Implementation Fall 2011. District-Wide
Steering Committee in place
Teams and strategy are developed
Operations Strategy : How do we implement? What processes and procedures need to be
developed
Communication Strategy: How do we communicate? What and where? What are the
benefits? How is it different? How does it relate to our vision?
Feedback Strategy : How will this improve student learning? How will it improve teaching
effectiveness?
17

18. EGLS3
Spring 2012-Next Steps
Spring 2012 Next Steps
Immediate: Increase response rate
As survey reports became available, faculty and administrators will be more involved in
the process
Develop committees to work to integrate this students evaluation with HCC Faculty/Staff
evaluation system PEP.
Redevelop PEP
Research Project
18

19. EGLS3
About SmarterServices
Incorporated in 2002.
Served over 500 educational institutions including K-12, technical colleges, career colleges, community
colleges, private colleges, universities, and corporations.
Some of the other solutions provided by SmarterServices include SmarterMeasure, SmarterFaculty,
and SmarterProctors.

20. EGLS3
About UNT’s SETE™ Instrument
Benefit from extensive research at the University of North Texas.
Control for bias of 24-variables
Student grades
Time of course
Delivery system
Track improvement with a standardized effectiveness score (1-1000 scale)
Useful for benchmarking and longitudinal analysis of faculty effectiveness
Meet state reporting guidelines

21. EGLS3
Student’s Perspective within SmarterSurveys™

22. EGLS3
Student’s Perspective within SmarterSurveys™
18 Standard SETE Questions
16 Likert scale questions (4 per Category)
2 General Information questions

23. EGLS3
Student’s Perspective within SmarterSurveys™
Custom Question can be added to SETE.
Custom questions can be Likert, open ended, multiple choice, etc.
HCC choose to add 5 open ended questions to their evaluation

24. EGLS3
Student’s Perspective within SmarterSurveys™
Confirmation pop-up is presented to Student’s for verification upon submission of a survey.

25. EGLS3
Student’s Perspective within SmarterSurveys™
After completing a survey, the course is marked as “Completed”
Any remaining surveys can be taken immediately or at another time

26. EGLS3
Faculty’s Perspective within SmarterSurveys™
HCC choose to suppress results until the end of the survey period.
Faculty can still see their completion percentages, survey status, and results release date.

27. EGLS3
SETE™ Data Processing Flow
Faculty and Student Attributes
+ Grades
EGLS3 Scores Generated
Data Processed Faculty ID, Course ID & Scores
Survey Responses

28. EGLS3
Faculty’s Perspective within SmarterSurveys™
Once survey results open, faculty can view course responses and also pull aggregate reports for the term.

29. EGLS3
Faculty Reports within SmarterSurveys™
Individual Course Response Distribution Report – Per Course
Header and Likert Responses

30. EGLS3
Faculty Reports within SmarterSurveys™
Individual Course Response Distribution Report – Per Course
Open Ended Responses

31. EGLS3
Faculty Reports within SmarterSurveys™
EGLS3 Score Report – Per Term

32. EGLS3
Dean Reports within SmarterSurveys™

33. EGLS3
Dean Reports within SmarterSurveys™

34. EGLS3
34

35. EGLS3
Questions….
Contact Information:
Alan Ainsworth 713.718.8709; alan.ainsworth@hccs.edu
Mario Heredia 713.718.8627; mario.heredia@hccs.edu
SmarterSurveys™ Info 877.499.SMARTER (7627);
info@smarterservices.com
35