Acumen Website Reports

1. February 29, 2012
1
Part II: External Evaluator’s Report
A) NSF Gk-12 Programmatic and Project Goals
1) Goals and Measures
Fellows’ Goals Measures
Enhanced understanding of one’s own
research subject area
Presentation skills protocol & number of
presentations made
Its societal & global contexts
Presentation skills protocol & number of
presentations made
Improved communication skills
Presentation skills protocol & number of
presentations made
Improved leadership skills
Presentation skills protocol & number of
presentations made
Experience working on teams Summer program survey, exit survey
Enhanced teaching capabilities
Direct observation of teaching inquiry-based
lessons
Teachers’ Goals Measures
Professional development in STEM content Summer program content tests
Professional development in hands-on science
activities
Student Watershed Watch training and
implementation observations
**Establish long-lasting professional
relationships with the Environmental Science
Learning Community Teachers’ record of collaborative activities.
Students’ Goals Measures
Energize students to pursue STEM careers Attitude towards science survey
**Increase participation in hands-on
environmental research
Items will be added to Student Survey to
explore amount of hands-on research students
have experienced over the previous year.
**Increase participation in SWW and science
fairs SWW & science fair participation numbers
**Increase interest in science and science-
related careers Student Science Interest Survey
University/Community Goals Measures
Create strong and enduring partnerships with
K-12 schools.
Participation of local schools in LEC
sponsored poster session
Transform graduate programs Feedback from graduate advisors & fellows
Enhance the impact of graduate education on
society Combination of all outcomes
**Indicates project only goal

2. February 29, 2012
2
2) Explanation of Measures and Instruments
As in previous years, an in-depth case study design was used to examine outcomes.
Fellow data was gathered using three instruments utilized in previous years—a performance
assessment was used for direct observations of the fellows as they presented their research to a
variety of audiences (mostly high school students) throughout the year, an exit interview for
fellows graduating from the program, and fellow content knowledge gain was measured during
the summer program using a pretest/posttest of relevant content.
Teacher gains were also measured using the content test developed by the faculty teaching in the
summer institute. Teachers will also complete the Science Teacher Ideological Preference Scale
in April to examine preferences for inquiry vs. non-inquiry teaching practices and changes over
time will be reported next year.
Project effects on students were again examined using our Student Attitudes towards Science
instrument. Matched comparisons between treatment and control classrooms over the past three
years indicated that this process did not provide equivalent groups and therefore comparisons
were not providing valid findings. This year we eliminated the control classrooms and only
examined treatment pretest/posttest gains on the Student Attitude about Science instrument
developed for this project. This survey was based upon Leopold Klopher’s (1971) categories of
affective behaviors in science education that cross behaviors with phenomena to discover to what
extent students in high schools internalized positive aspects of science and whether classrooms
that host GK-12 Fellows can affect this change. Internalization occurs when a value or
phenomenon becomes a part of the individual’s identity. Our survey specifically targeted
favorable attitudes towards science and scientists, enjoyment of science, the development of
interests in science and science-related activities, and the development of an interest in pursuing
a science-related career. We calculated reliability indices on this measure using Rasch analysis
because it allows us to look at both person and item reliability. Person reliability was 0.72 and
item reliability was 0.99.
The Student Watershed Watch survey implemented last year was eliminated because the survey
proved to lack reliability and validity. However, we added a teamwork assessment this year to
gain a better understanding of the dynamics between fellows, teachers, and project personnel.
Impact on university faculty will be explored in April/May again using our project-developed
survey that explores each faculty member’s interest in working with K-12 education.
B) Executive Summary of Findings
1) Fellows: Fellows made 40 research presentations and published 19 scholarly articles
during the past year. As in previous years, fellows presented their research to local high school
science classes and were observed by the evaluator to assess fellow presentation skills when
working with the general public. Most fellows scored in the proficient range and those who were
completing their second year with the project did much better than those who were first year
participants. Two fellows presented information that was too technical for the audience but in
general the presentations were easy to follow and relevant (last year this was a problem area).
One fellow was thrown into a difficult situation because the teacher of the class frequently

3. February 29, 2012
3
interrupted and rushed the fellow through the presentation. While the presentation suffered
slightly from the teacher’s actions, the fellow was able to make the best of the situation and
exhibited a professional demeanor.
The 2011 Summer University of Toledo Lake Erie Center GK-12 content was measured using
the course, Evolutionary and Ecological Adaptations of Aquatic Fishes. This course was selected
because it presented new material to both fellows and teachers. On the first day of the summer
program, all participants (16) were given a pretest. Out of a possible 100 points, the mean score
for fellows on the pretest was 50.31. The posttest was administered at the conclusion of the
course and fellows showed a statistically significant increase in content mastery scoring a mean
of 80.93 (p < 0.004).
Fellows were asked to provide an example of something they learned during the summer that
they would incorporate into the high school classroom this coming year. All but one fellow
provided a variety of thoughtful, appropriate applications.
Feedback from the summer program (both from fellows and teachers) continued to reflect a
positive learning experience. Many changes made this year based upon feedback from last year
appear to have improved the experience for both the teachers and the fellows. Most agreed that
the field trips were a good use of their time and they learned about new places to take students.
However, several commented that the information provided during the field trips, as well as the
assignments, were overwhelming and detracted from the experience.
When asked what was particularly valuable about the summer experience, the Stone lab,
working onsite at Lake Erie, the University of Michigan museum, and the topic of evolution
were mentioned by several respondents.
Fellows were also asked how they would change the summer experience to make it more
meaningful. A general theme that emerged focused on homework and written assignments.
Fellows viewed these assignments as busy work and did not feel they contributed to the summer
experience.
Five of the six fellows leaving the project at the end of this year have completed the exit survey
to date. Three fellows hoped to eventually become faculty at universities and the other two
preferred working in private industry. While none of the fellows felt that the GK-12 experience
changed their plans for the future, they all acknowledged that the experience either better
prepared them for their careers or reinforced their decision. While four of the five prefer to work
alone, most did feel that their opinion of what it is like to work on a team changed. They
acknowledged that in groups a clear leader needs to be identified, members need to accept
responsibility for their roles, and a facilitator to keep the team on track is vital. As with the
fellows last year, most agreed that the GK-12 experience has made them more adept at sharing
their research with the general public and have gained an understanding of the importance of this
endeavor. One fellow noted that sharing research with the general public is “a major
responsibility of our profession--raising up the next generation of scientists.” To present to high
school students, the fellows eliminated scientific jargon and many statistical and/or data details.
Overall, they realized that this audience needed to gain an overall understanding of what they
have been doing rather than detailed findings. All felt that the interaction between scientist and
general public to be mutually beneficial.

4. February 29, 2012
4
As in previous years, the fellows felt that the typical undergraduate science classroom could
benefit from more interactive teaching that included less lecture and more hands-on student
work. They also agreed that to recruit more scientists, stereotypes about the profession need to be
challenged and “people need to understand that these are the careers of the future and that they
aren't reserved for ‘nerds’.”
Their classroom experience in the high schools was challenging but they all acknowledged that
even though students may seem uninterested or have a short attention span, quality teaching
strategies that engage the students often results in students rising to the occasion and active
sessions that run out of time. All of the fellows appreciated the opportunity to be a GK-12
fellow. One fellow’s comment summarizes the consensus: “I've learned a lot about myself and
what it means to be part of a scientific community. A community of scholars that is able to both
challenge you and frustrate you but in the end, make you better at your craft.”
Teachers: As with the fellows, teachers showed a statistically significant gain in content over the
summer with similar pretest and posttest scores—50 and 80.93 respectively (p < 0.005). In
general, teacher reactions to the summer program were similar to the fellows. However, the
application of one topic (limnology refresher and corer on Sand Lake) proved to be challenging
in particular for teachers. Overall the summer program provided teachers with an opportunity to
learn new content, to experience relevant hands on science investigation, and to become better
acquainted with the newest additions to the project.
Teachers will complete the Science Teacher Ideological Preference Scale in April and any
changes over the past year will be reported in the next report.
Teamwork: In total, 15 of the 16 participants completed the Teamwork Survery (8 fellows and 7
teachers). The purpose of this survey was to help us understand the relationship of the
teacher/fellow partnership. We also wanted to better understand group dynamics as a whole.
Overall, the teachers were more positive about the relationships than the fellows. Items that
focused on open communication and respect within the fellow/teacher partnership showed the
strongest degree of agreement. Overall, the teachers felt that their partnership was strong and that
all aspects of the project contribute to its success. Only one teacher felt that the group of 16
participants as a whole struggled to identify issues and find solutions. Fellows were also positive
about their relationship within the partnership but a few respondents felt that some group
activities were not as productive as they could be. The GK-12 partnerships reflect hallmarks of
teamwork (mutual respect, open communication, a balanced relationship). In one case, it appears
the fellow may need to ask his/her teacher to provide more information about what is occurring
in the classroom (i.e., knowing where the teacher is going next). It may be helpful to all of the
fellows if the teachers shared their long-term learning goals for their class and how what they
contribute fits into this plan if this is not already being done. While meeting as a whole seems to
be less productive than the teacher/fellow partnerships, the criticism is minor.
Students: Only pretest scores (August 2011) have been reported to date. Posttests are scheduled
to be collected in April. The sample consisted of 455 students in the participating teacher/fellow
classrooms. While approximately 67% reported having participated in some type of science
competition prior to this year, only 17% felt they had engaged in scientific inquiry over the past
year and only 13% felt that their science teachers in general relate science to student lives and
environment. Of a total of 68 possible points on the survey (a higher score indicating a stronger

5. February 29, 2012
5
value placed on science), student mean score was 31.99 (slightly below an expected mean of
34). Responses to one item fell outside the acceptable two standard deviation range—“Scientists
have trouble relating to non-scientists.” Students agreed with this statement at an unusually high
rate. Since this is a major goal of the GK-12 program, comparisons in April could provide
evidence of the effect interacting with a fellow can have on student understanding of how
scientists communicate.
Information on student participation in science activities through participation in science fairs
has not yet been reported
4) University faculty & sustainability: Faculty participating in GK-12 as well as other faculty
from the same departments will complete our survey that examines amount of time their graduate
assistants spend on a variety of activities and the ideal distribution of a graduate student’s time
over several related activities and responsibilities might be. Results will be reported in the next
annual report.
As in previous years, this project continues to reach its goals for fellows, teachers, students, and
higher education.

6. February 29, 2012
6
C) Recommendations
This project continues to improve based upon formative assessment findings and through the
identification of best practices. Slight changes are recommended to ensure that data collection
adequately reflects program results. First, a sample fellow presentation (some have been
recorded) should be shown to the fellows over the summer or early in the fall semester so that
they have a clear idea of what is expected with regard to presentations. Second, the project team
should consider making explicit connections between teachers and the science community so that
the teachers might improve their networking with this valuable resource. The Student Watershed
Watch survey as is should not be repeated. However, it is recommended that a team of teachers,
fellows, and the evaluator work together to refine the instrument as there is value in developing a
reliable, valid instrument to measure the effects of the program. This instrument will not only
contribute to the verification of project goal attainment but can also be used by the Student
Watershed Watch program to gather data that could make a compelling case for community
support. Finally, while those faculty who work on GK-12 support graduate student interaction
with K-12 teachers and students, faculty who do not have this or a similar connection remain
unsupportive. It is recommended that the GK-12 faculty, teachers, and fellows share their results
with the University community.
Sevian, H. and Gonsalves, L. (2008). Analyzing how scientists explain their research: A rubric for measuring the
effectiveness of scientific explanations. International Journal of Science Education, 30 (11), 1441-1467.