- The document analyzes student usage of the Blackboard site for an economics course and performance on an exam. - It finds that students performed better on questions that were seen beforehand compared to unseen questions. - Increased usage of the announcements section of Blackboard, where exam material was posted, correlated with higher exam scores. However, past academic performance had a stronger influence on exam scores than Blackboard usage alone.

1. Blackboard Usage and
Exam Performance
Seamus Coffey
Dept of Economics, UCC
s.coffey@ucc.ie 021-4901928

2. EC2101: Intermediate Microeconomics
Course began on 24th September 2008 and Blackboard
website was available from 22nd September.
Exam took place on Wednesday 12th November and
was for 10% of total marks available.
The one-hour exam comprised two questions each
with two parts.
Each question was worth 50 marks and each part was
worth 25 marks of the 100 available.

3. Exam Description
One week before the exam four questions were
posted to Blackboard. Two of these appeared as
the first part of the two questions. The other two
questions posted were not used.
The second part of each question was unseen.
1a – seen 1b – unseen; 2a – seen 2b – unseen
Students were told that the second parts of the
questions would relate to material posted on the
course blog on Blackboard from 25/09 to 04/11.

4. Overall Summary of Usage

5. Excluding Weekends and Bank Holiday

6. Number of people who visited the site for the first time on each day.
9 people never visited the site prior to the exam.

7. Of the 6,068 “days” (164 students by 37 days) the site was visited
on 1,358 days, or about 22%. Days exclude weekends and bank
holidays.
Of these the bottom 50% accounted for 336 “days” and the top
10% accounted for 333 “days”.
The “top” student visited on 30 of the available 37 days which was
matched by the bottom 25 students!
The average number of days the site was used was 8.3 (out of 37),
the median was 7 and mode was 5.
There were 163 students registered for the module and 140
students sat the exam.

8. 20
15
Frequency
10
5
0
Student Use by Days
0 5 10 15 20 25 30
Number of Days Used

9. 20
15
Frequency
10
5
0
Distribution of Marks
0 20 40 60 80 100
Mark

10. Distribution of Marks by Question
.15
.3.2
.1
Density
Density
.05
.1 0
0
0 5 10 15 20 25 0 5 10 15 20 25
Question 1 Part A Question 2 Part A
.2
.1
.08
.15
.04 .06
Density
Density
.05 .1
.02
0
0
0 5 10 15 20 0 5 10 15 20 25
Question 1 Part B Question 2 Part B

11. Variable | Obs Mean Std. Dev. Min Max
-------------+--------------------------------------------------------
Q1 Part A | 140 18.9 4.8 0 24
Q1 Part B | 140 9.6 5.4 0 24
Q2 Part A | 140 11.8 6.5 0 23
Q2 Part B | 140 9.6 5.4 0 24
Total mark | 140 47.9 14.7 10 90
-------------+--------------------------------------------------------
Part A | 140 30.7 9.3 6 47
Part B | 140 17.2 8.7 0 44
The average mark in the exam was just under 48%.
Students did substantially better in the questions that were seen
(Part A: 30.7 out of 50) than in the questions that were unseen
(Part B: 17.2 out of 50)

12. Variables

13. 20 40 100
Fitted values/Exam mark
0 60 80
Mark v “Oldmark”
0 20 40 60 80 100
Economics mark from Arts 1

14. 100
80
60
Mark
40
20
0
Mark v Total Blackboard Usage
0 100 200 300 400 500
Total Blackboard Usage from Start of Term

15. “Oldmark” v Total Blackboard Usage
20 40 100
Economics mark from Arts 1
0 60 80
0 100 200 300 400 500
Total Blackboard Usage for Term

16. Some Preliminary Regressions
Regression of mark on “total usage of Blackboard site” from
the beginning of term.
Regression with robust standard errors Number of obs = 140
F( 1, 138) = 2.00
Prob > F = 0.1596
R-squared = 0.0085
Root MSE = 14.733
------------------------------------------------------------------------------
| Robust
mark | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
totaltotal | .0274147 .0193897 1.41 0.160 -.0109246 .065754
_cons | 45.57679 2.137373 21.32 0.000 41.35055 49.80302
------------------------------------------------------------------------------
Relationship is positive but not significant.

17. Regression of mark on “total content usage”, i.e. section of
the site with lecture slides, readings and other handouts.
Source | SS df MS Number of obs = 140
------------+------------------------------ F( 1, 138) = 3.05
Model | 653.283045 1 653.283045 Prob > F = 0.0829
Residual | 29555.6884 138 214.171655 R-squared = 0.0216
-------------+------------------------------ Adj R-squared = 0.0145
Total | 30208.9714 139 217.33073 Root MSE = 14.635
------------------------------------------------------------------------------
mark | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
contotal | .0592053 .0338993 1.75 0.083 -.0078239 .1262345
_cons | 45.37734 1.907826 23.78 0.000 41.60499 49.14969
------------------------------------------------------------------------------
Positive relationship and significant at the 10% level.

18. Regression of mark on “total announcements (blog )
usage” from the start of term.
Regression with robust standard errors Number of obs = 140
F( 1, 138) = 5.32
Prob > F = 0.0225
R-squared = 0.0635
Root MSE = 14.318
------------------------------------------------------------------------------
| Robust
mark | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
anntotal | .1894081 .0821026 2.31 0.023 .0270664 .3517497
_cons | 42.90309 2.405475 17.84 0.000 38.14673 47.65945
------------------------------------------------------------------------------
Relationship is positive and significant at the 5% level.
Graphed on next slide.
Increased usage of the announcements section leads to a
higher mark. Not surprising given that half of the exam
was based on the 37 posts to this section.

19. Mark v Total Announcement Usage
20 40 100
Fitted values/Exam mark
0 60 80
0 50 100 150
Total Announcement Usage for Term

20. Regression of combined mark from both part As (seen questions based on lecture
content) and “total content usage”.
Regression with robust standard errors Number of obs = 140
F( 1, 138) = 0.90
Prob > F = 0.3442
R-squared = 0.0080
Root MSE = 9.281
------------------------------------------------------------------------------
| Robust
a | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
contotal | .0226657 .0238778 0.95 0.344 -.0245479 .0698793
_cons | 29.70735 1.263107 23.52 0.000 27.2098 32.20489
------------------------------------------------------------------------------
No significant findings.

21. Regression with combined marks from part Bs (unseen
questions based on blog posts to Announcements
section) on “total Announcements usage”.
Regression with robust standard errors Number of obs = 140
F( 1, 138) = 6.90
Prob > F = 0.0096
R-squared = 0.0541
Root MSE = 8.4896
------------------------------------------------------------------------------
| Robust
b | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
anntotal | .1031929 .0392844 2.63 0.010 .0255157 .1808702
_cons | 14.50552 1.176658 12.33 0.000 12.17891 16.83213
------------------------------------------------------------------------------
Significant (at the 1% level) and positive relationship.

22. Regression of combined mark from part Bs on
announcement usage from the start of term to November
4th (annall) and in the week prior to the exam (annweek) .
Source | SS df MS Number of obs = 140
-------------+------------------------------ F( 2, 137) = 4.00
Model | 579.53008 2 289.76504 Prob > F = 0.0206
Residual | 9935.69135 137 72.5232945 R-squared = 0.0551
-------------+------------------------------ Adj R-squared = 0.0413
Total | 10515.2214 139 75.649075 Root MSE = 8.5161
------------------------------------------------------------------------------
b | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
annall | .1209358 .0595487 2.03 0.044 .0031824 .2386893
annweek | .0708671 .0928607 0.76 0.447 -.1127586 .2544928
_cons | 14.77497 1.404544 10.52 0.000 11.99758 17.55236
------------------------------------------------------------------------------
Usage during term and not just prior to the exam is more
important.
But do the good students use Blackboard rather than
students being good because they use Blackboard?

23. “Oldmark” is economics mark from Arts I for students.
Reduces sample size to 109.
Source | SS df MS Number of obs = 109
-------------+------------------------------ F( 2, 106) = 34.38
Model | 8954.53176 2 4477.26588 Prob > F = 0.0000
Residual | 13803.0462 106 130.217417 R-squared = 0.3935
-------------+------------------------------ Adj R-squared = 0.3820
Total | 22757.578 108 210.718315 Root MSE = 11.411
------------------------------------------------------------------------------
mark | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
anntotal | .0502289 .0584326 0.86 0.392 -.0656194 .1660772
oldmark | .6348721 .0859628 7.39 0.000 .4644425 .8053017
_cons | 13.72065 4.2581 3.22 0.002 5.278557 22.16275
------------------------------------------------------------------------------
R-squared is now 0.39 and oldmark has a positive significant
effect on mark.
Total announcement usage which in the bivariate
regression was significant is now very much insignificant.
The dominant explanatory variable is oldmark.í