Review of test preparation offerings and their relation to test security

1. How Big is the Test Preparation Industry
and What are Employers Doing to
Counter It?
April 16, 2005
Cheryl Paullin
Personnel Decisions Research Institutes, Inc.
In Paullin, C. (Chair). Maintaining test security in a “cheating” culture. Practitioner Forum
conducted at the 20th Annual Conference of the Society for Industrial and Organizational
Psychology, Los Angeles, CA.

2. April 16, 2005 Paullin: Test Prep Industry 2
Disclaimer
• I am not saying that test preparation and cheating
are the same thing.
• I am not saying that it is unethical to provide test
preparation or test coaching services. I do believe
that there are unethical providers out there.
• Test prep is a good thing to the extent it:
– Improves relevant skills or knowledge.
– Improves general test-taking skills.
– Reduces text anxiety.

3. April 16, 2005 Paullin: Test Prep Industry 3
Method
• Searched Research literature – keywords:
– [employment] test preparation
– test coaching
• Searched the Internet, libraries, and bookstores for
information about organizations that provide test
preparation services.
• Talked with colleagues and clients about what
employers are doing to counter the test prep
industry.

4. April 16, 2005 Paullin: Test Prep Industry 4
Research Literature
• Sackett, Schmitt, Ellingson, & Kabin (2001)*
– Reviewed test coaching programs, specifically to
evaluate impact (if any) on subgroup score differences
on standardized tests.
• Conclusions -- Test Coaching Programs:
– Benefit minority & nonminority examinees slightly, but
do not reduce subgroup score differences.
– Are viewed very positively by examinees.
– Can have a large practical impact, e.g., when one or
two points is the difference between passing & failing a
certification exam.
*American Psychologist, Vol 56 (4), 302-318.

5. April 16, 2005 Paullin: Test Prep Industry 5
Type of Exams Covered
• Education Program Admission (SAT, GRE,
GMAT, LSAT, MCAT)
• Proficiency/Placement (GED, High School Exit
Exams, Advanced Placement, English & Foreign
Language Skills)
• Employment
– Selection (ASVAB, USPS Exam 470, Police Officer,
Civil Service, Fire Service).
– Certification/Licensure (Health Care Professions,
Teacher, Accountant, Engineer, Real Estate, Financial
Planning, Cosmetology, Information Technology).

6. April 16, 2005 Paullin: Test Prep Industry 6
Scope of Test Prep Industry
• Number of orgs offering test prep generally
corresponds to size of examinee population
(i.e., the market):
– Dozens for large-scale admission exams and for
nationwide or regional certification, licensure, or
employment exams.
– At least one or two for virtually all location- or
employer-specific exams administered on a regular
basis to more than a few hundred people.

7. April 16, 2005 Paullin: Test Prep Industry 7
Scope of Test Prep Industry (con’t)
• Test prep is avail for > 100 different exams.
• For anything other than printed test guides,
almost impossible to evaluate quality prior
to buying the service/materials.
• One website compares services (more later).

8. April 16, 2005 Paullin: Test Prep Industry 8
Test Prep Activities
• Test Prep Activities
– Test guide books (avail for > 100 different exams).
– Practice tests (CD-based or online avail for 75-100 exams).
– Prep courses (CD-based, online, or live avail for 20-30 exams).
– Personal tutoring (for several high-stakes exams, e.g., SAT,
GMAT, Medical Licensure, Bar Exam, CPA).
• Practice Tests
– Vary in extent to which they simulate real testing conditions.
• P&P, scoring key & explanation provided.
• Real-time administration & scoring with feedback -- access to item
bank for X number of tests or X number of hours.

9. April 16, 2005 Paullin: Test Prep Industry 9
Test Prep for a Fee
• Providers covering the widest array of
exams:
– Kaplan
– Learning Express
– Thomson Learning
– Princeton Review
– Cliff’s Test Prep
• Supplemented by an uncountable number of “Dr.
Paullin’s Best Test Prep Ever” websites.

10. April 16, 2005 Paullin: Test Prep Industry 10
Costs & Claims
• Costs
– $10-$50 for Test Guide books.
– $100-$300 for CD-based course, or limited online access to
practice tests.
– Several hundred dollars for a lengthy online course, live course, or
several hours of personal tutoring.
– Costs are higher for exams related to high-income professions,
e.g., medical licensure, dental licensure, bar exam, CPA.
• Claims
– Many test prep orgs offer a money-back guarantee unless X
happens, e.g., earning a passing score.
– Most say little or nothing about qualifications of the test prep
developer(s); a few say course is taught by or materials were
developed by someone who scored high on the exam in question.

11. April 16, 2005 Paullin: Test Prep Industry 11
Comparing Test Prep Providers
www.test-preparation.net
• For each of 23 exams:
– Which orgs offer test prep services.
– Type and cost of prep activities.
• Doesn’t evaluate quality of materials or
providers.

12. April 16, 2005 Paullin: Test Prep Industry 12
Test Prep as a Public Service
• Public Sector Jobs
– Several federal government websites
www.usajobs.opm.gov.
– Public Service Employees Network
www.pse-net.com.
• College entrance – www.number2.com
• Labor Unions – Not much on websites at national
level; may be more info at local branches.

13. April 16, 2005 Paullin: Test Prep Industry 13
Test Developers/Sponsors
• Several offer test prep, some of it free, e.g.,:
– College Board – SAT.
– ETS (Educational Testing Service) – many tests.
– ACT (American College Testing) – ACT.
– American Association of Medical Colleges – MCAT.
– National Association of Pharmacy Boards – PCAT.
– Project Management Institute – PMI Certification.
– Pearson VUE – Nursing Certification.
– Experior Online -- state-specific certification exams.

14. April 16, 2005 Paullin: Test Prep Industry 14
What are Employers Doing to
Counter Test Prep Industry?
• Relying on predictors other than multiple-choice
tests, e.g., interviews.
• Outsourcing the testing process – countering test
prep industry someone else’s problem.
• Changing test content frequently.
– Multiple forms  Adaptive testing.
– New test for each mass administration.
– Easier to do with computer-based & online testing.

15. April 16, 2005 Paullin: Test Prep Industry 15
What are Employers Doing? (con’t)
• Providing their own test prep materials.
• Warning test-takers to be cautious when
buying test prep services.
• (rare) Legally pursuing test prep
orgs/individuals that make fraudulent
claims or provide access to controlled test
content.

16. April 16, 2005 Paullin: Test Prep Industry 16
Bogus Credentials
• Issue of increasing concern.
• Search term “Diploma Mill”  10 websites.
• What’s an employer to do?
– http://ope.ed.gov/accreditation -- US Office of Post-
Secondary Education.
– Hire an organization that does education verification.
– If possible, assess KSAs directly.

17. April 16, 2005 Paullin: Test Prep Industry 17
Test Security in Litigation
• Attorneys may request, or court may order, that
test items and scoring key be shared with parties
to litigation or read into the court record.
• What should the I/O Psychologist do?
• Principles of Psychologists and Code of Conduct (American
Psychological Association, 1992) -- If psychologists’ ethical
responsibilities conflict with law, psychologists make known their
commitment to the Ethics Code and take steps to resolve the
conflict in a responsible manner. (Standard 1.02)

18. April 16, 2005 Paullin: Test Prep Industry 18
Test Security in Litigation (con’t.)
• Standards for Educational and Psychological Testing (1999) state:
When sensitive test documents are challenged, it may be appropriate to
employ a third party, using a closely supervised secure procedure to
conduct a review of the relevant materials. Such secure procedures are
generally preferable to placing tests, manuals, and an examinee’s test
responses in the public record.
• American Psychologist editorial (1999): The psychologist should ask
the court to allow delivery of the secure materials only to psychologists
or professionals who are bound by the an ethical code to maintain their
security. If this isn’t feasible or isn’t allowed, then request that the
court issue a protective order prohibiting the parties from making
copies of the materials, requiring that the materials be returned to the
psychologist at the close of litigation, and ordering that the record be
sealed if test questions or answers are admitted in court.