Traffic Tech #202: The Art of Appropriate Evaluation for Highway Safety Program Managers |
|---|
|
THE ART OF APPROPRIATE EVALUATION FOR HIGHWAY SAFETY PROGRAM MANAGERS
Highway safety program managers who want to know how to evaluate the effectiveness of their programs will want The Art of Appropriate Evaluation: A Guide for Highway Safety Program Managers. The National Highway Traffic Safety Administration (NHTSA) sponsored development of this guide to help document the good parts of our traffic safety programs and identify the things that need to be changed. The Guide de-mystifies what is meant by evaluation, describes common stumbling blocks, and discusses how to choose an appropriate evaluation method to promote the hard work that goes into traffic safety programs. The Guide is intended for state or local highway safety project directors with at least some curiosity about program evaluation. A background in experimental design or statistics is not required. The Guide tells where you can find experts to help with that part.
Contents
Introduction
The Evaluation Mentality
In Search of the Appropriate Evaluation
Evaluation Step-by-Step
Getting Help
Closing Comments
Glossary of Terms that Evaluators Use
The Evaluation Mentality
The Guide recommends that to get the full range of benefits from evaluation, you need to have an evaluation mentality. This means that you, as a project director, never consider implementing a program without first asking a set of five questions:
With the answers to these questions in hand, you are prepared to convince any funding source that you know what needs to be done and that your ideas have a high probability of success. An evaluation mentality will also ensure that at the end of the project, you can report back to these same funding sources with solid information on what you accomplished. Here are some excerpts.
Accurately identify the problem you are trying to solve -- All too often people jump into implementing a program without really understanding the underlying cause of their problem. Is underage DWI a problem because the liquor stores are selling to teenagers or because the police could better target the locations where kids are drinking? It is not enough to suspect that safety belt use is low in your community. You need to determine up front what target groups make up your non-users.
Uncover some problems you didn't know you had-You might assume that your pedestrian safety problem involves the very young and very old until you discover that a significant percentage of your pedestrian fatalities are working age adults who had been drinking at the time they were hit by a car. Each problem requires a different set of countermeasures.
Establish reasonable, practical objectives for dealing with these problems -- Global objectives are the hardest to accomplish. With good problem identification data, you can focus your objectives on the specific problem you are trying to solve. Instead of trying to "reduce unsafe driving behaviors", you might want to reduce red-light running, which is the unsafe behavior that is causing the most concern in your community.
Determine if you have accomplished your program objectives -- A major purpose of any evaluation is to determine if your program accomplished its objectives. Well thought out objectives are an important first step, but an evaluation mentality will help you zero in on what you truly need to measure. Too often, project directors waste time and money collecting data that they can never use, because they can't compare it to any baseline or because it does not relate to their program objectives.
Provide information to funding sources, the media, and the public to continue support for program -- Support for a good idea can evaporate if there is no evidence that the idea really works. With an evaluation mentality, you create consensus from the beginning on the criteria for success, and you stay focused on the data you will need to measure your performance against those criteria.
Determine if and how a program should be revised to increase its effectiveness -- With an evaluation mentality, you don't wait until the end to find out how your idea worked. Very few projects work perfectly. There are always aspects that could be tweaked to make them more effective. With an
evaluation mentality, you monitor performance throughout the project so that you can institute mid course corrections if needed, and so that you are ready to revise your concept for next year.
What Works
The Guide describes different types of evaluations a project director might consider. Certain countermeasures have been around for a while. There is no need to prove, once again, that safety belts and strong DUI laws save lives. If your program involves one of the following strategies, you can concentrate your evaluation dollars on documenting that you implemented the counter-measure, not that the countermeasure works safety belts child safety seats (always in the back seat) bicycle helmets motorcycle helmets DWI enforcement sobriety checkpoints tougher impaired driving laws.
HOW TO ORDER
For a copy of The Art of Appropriate Evaluation: A Guide for Highway Safety Program Managers, DOT HS 808 894 (64 pages), prepared by Perform Tech of Landover, Maryland, write to the Office of Research and Traffic Records, NHTSA, NTS-31, 400 Seventh Street, S.W., Washington, DC 20590, or fax (202) 366-7096. Jonathan Walker, Ph.D., was the contract manager for this project, email: jwalker@nhtsa.dot.gov
Download from NHTSA's web site at http://www.nhtsa.dot.gov/people/injury/research
U.S. Department
of Transportation
National Highway
Traffic Safety
Administration
400 Seventh Street, S.W. NTS-31
Washington, DC 20590
Traffic Tech is a publication to disseminate
information about traffic safety programs,
including evaluations, innovative programs,
and new publications. Feel free to copy it as you wish.
If you would like to receive a copy contact:
Linda Cosgrove, Ph.D., Editor, Evaluation Staff
Traffic Safety Programs
(202) 366-2759, fax (202) 366-7096
mailto:lcosgrove@nhtsa.dot.gov