Proposed Model Performance Measures for State Traffic Records Systems |
|---|
Topics: National Highway Traffic Safety Administration
|
Jeffrey Michael
Federal Register
April 12, 2011
[Federal Register: April 12, 2011 (Volume 76, Number 70)]
[Notices]
[Page 20438-20448]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr12ap11-155]
-----------------------------------------------------------------------
DEPARTMENT OF TRANSPORTATION
National Highway Traffic Safety Administration
[Docket No. NHTSA-2011-0044]
Proposed Model Performance Measures for State Traffic Records
Systems
AGENCY: National Highway Traffic Safety Administration (NHTSA),
Department of Transportation (DOT).
ACTION: Notice
-----------------------------------------------------------------------
SUMMARY: This notice announces the publication of Model Performance
Measures for State Traffic Records Systems DOT HS 811 44, which
proposes model performance measures for State traffic record systems to
monitor the development and implementation of traffic record data
[[Page 20439]]
systems, strategic plans, and data-improvement grants. These model
performance measures are voluntary and are to help States monitor and
improve the quality of the data in their traffic record systems
DATES: Written comments may be submitted to this agency and must be
received no later than June 13, 2011.
ADDRESSES: You may submit comments identified by DOT Docket ID number
NHTSA-2011-0044 by any of the following methods:
Electronic Submissions: Go to http://www.regulations.gov.
Follow the online instructions for submitting comments.
Fax: 202-366-2746.
Mail: Docket Management Facility, M-30 U.S. Department of
Transportation, West Building, Ground Floor, Room W12-140, 1200 New
Jersey Ave., SE., Washington, DC 20590.
Hand Delivery or Courier: Docket Management Facility, M-30
U.S. Department of Transportation, West Building, Ground Floor, Room
W12-140, 1200 New Jersey Ave., SE., Washington, DC 20590, between 9
a.m. and 5 p.m. Eastern time, Monday through Friday, except Federal
holidays.
Regardless of how you submit your comments, you should identify the
Docket number of this document.
Instructions: For detailed instructions on submitting comments and
additional information, see http://www.regulations.gov. Note that all
comments received will be posted without change to http://
www.regulations.gov, including any personal information provided.
Please read the ``Privacy Act'' heading below.
Privacy Act: Anyone is able to search the electronic form of all
contents received into any of our dockets by the name of the individual
submitting the comment (or signing the comment, if submitted on behalf
of an association, business, labor union, etc.). You may review the
complete User Notice and Privacy Notice for Regulations.gov at http://
www.regulations.gov/search/footer/privacyanduse.jsp.
Docket: For access to the docket to read background documents or
comments received, go to http://www.regulations.gov at any time or to
West Building Ground Floor, Room W12-140, 1200 New Jersey Avenue, SE.,
Washington, DC between 9 a.m. and 5 p.m., Eastern Time, Monday through
Friday, except Federal holidays.
FOR FURTHER INFORMATION CONTACT: For programmatic issues: Luke Johnson,
Office of Traffic Records and Analysis, NPO-423, National Highway
Traffic Safety Administration, 400 Seventh Street, SW., Washington, DC
20590. Telephone (202) 366-1722. For legal issues: Roland Baumann,
Office of Chief Counsel, NCC-113, National Highway Traffic Safety
Administration, 400 Seventh Street, SW., Washington, DC 20590.
Telephone (202) 366-5260.
SUPPLEMENTARY INFORMATION: The National Highway Traffic Safety
Administration (NHTSA) has identified 61 model performance measures for
the six core State traffic records data systems: Crash, vehicle,
driver, roadway, citation/adjudication, and EMS/injury surveillance.
These model performance measures address the six performance
attributes: Timeliness, accuracy, completeness, uniformity,
integration, and accessibility. States can use these measures to
develop and track performance goals in their Traffic Records Strategic
Plans, Traffic Records Assessments, and Highway Safety Plans; establish
data-quality improvement measures for specific traffic records
projects; and support data improvement goals in the Strategic Highway
Safety Plan. The full text of the report Model Performance Measures for
State Traffic Records Systems DOT HS 811 44, is available at http://
www.nhtsa.gov/.
Key Features of the Model Performance Measures
Use is voluntary: States should use the measures for those data
system performance attributes they wish to monitor or improve. If the
suggested measures are not deemed appropriate, States are free to
modify them or develop their own.
The measures are flexible: The measures are models. States can
modify a measure to meet a specific need as long as its overall intent
remains the same.
The measures do not set numerical performance goals: They describe
what to measure and suggest how it should be measured but are not
intended to establish a numerical performance goal. Each State should
set its own performance goals.
The measures provide a template or structure States can populate
with specific details: For example, the States must decide what data
files to use and what data elements are critical. States should take
advantage of these decision-making opportunities to focus on their most
important performance features.
The measures are not exhaustive: The measures attempt to capture
one or two key performance features of each data system performance
attribute. States may wish to use additional or alternative measures to
address specific performance issues.
The measures are not intended to be used to compare States: Their
purpose is to help each State improve its own performance. Each State
selects the measures it uses, establishes its own definitions of key
terms, and may modify the measures to fit its circumstances. Since the
measures will vary considerably from State to State, it is unlikely
that they could be used for any meaningful comparisons between States.
NHTSA has no intention of using the measure to make interstate
comparisons.
Core Traffic Records Data Systems
The model performance measures cover the six core traffic data
systems.
1. Crash: The State repository that stores law enforcement officer
crash reports.
2. Vehicle: The State repository that stores information on
registered vehicles within the State (also known as the vehicle
registration system). This database can also include records for
vehicles not registered in the State--e.g., a vehicle that crashed in
the State but registered in another State.
3. Driver: The State repository that stores information on licensed
drivers within the State and their driver histories. This is also known
as the driver license and driver history system. The driver file also
could contain a substantial number of records for drivers not licensed
within the State--e.g., an unlicensed driver involved in a crash.
4. Roadway: The State repository that stores information about the
roadways within the State. It should include information on all
roadways within the State and is typically composed of discrete sub-
files that include: Roadway centerline and geometric data, location
reference data, geographical information system data, travel and
exposure data, etc.
5. Citation/Adjudication: The component repositories, managed by
multiple State or local agencies, which store traffic citation, arrest,
and final disposition of charge data.
6. EMS/Injury Surveillance: The component repositories, managed by
multiple State or local agencies, which store data on motor vehicle-
related injuries and deaths. Typical components of an EMS/injury
surveillance system are pre-hospital EMS data, hospital emergency
department data systems, hospital discharge data systems, trauma
registries, and long term care/rehabilitation patient data systems.
[[Page 20440]]
Performance Attributes
The model performance measures are based on six core
characteristics:
1. Timeliness: Timeliness reflects the span of time between the
occurrence of an event and entry of information into the appropriate
database. Timeliness can also measure the time from when the custodial
agency receives the data to the point when the data are entered into
the database.
2. Accuracy: Accuracy reflects the degree to which the data are
error-free, satisfy internal consistency checks, and do not exist in
duplicate within a single database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. In some cases, it is possible to
determine that the values entered for a variable or data element are
not legitimate codes. In other cases, errors can be detected by
matching with external sources of information. It may also be possible
to determine that duplicate records have been entered for the same
event (e.g., title transfer).
3. Completeness: Completeness reflects both the number of records
that are missing from the database (e.g., events of interest that
occurred but were not entered into the database) and the number of
missing (blank) data elements in the records that are in a database. In
the crash database, internal completeness reflects the amount of
specified information captured in each individual crash record.
External crash completeness reflects number or percentage of crashes on
which crash reports are entered into the database. However, it is not
possible to determine precisely external crash completeness as it is
impossible to determine the number of unreported crashes. The measures
in this report only address internal completeness by measuring what is
not missing.
4. Uniformity: Uniformity reflects the consistency among the files
or records in a database and may be measured against some independent
standard, preferably a national standard. Within a State all
jurisdictions should collect and report the same data using the same
definitions and procedures. If the same data elements are used in
different State files, they should be identical or at least compatible
(e.g., names, addresses, geographic locations). Data collection
procedures and data elements should also agree with nationally accepted
guidelines and standards (such as the Model Minimum Uniform Crash
Criteria, [MMUCC]).
5. Integration: Integration reflects the ability of records in a
database to be linked to a set of records in another of the six core
databases--or components thereof--using common or unique identifiers.
Integration differs in one important respect from the first four
attributes of data quality. By definition, integration is a performance
attribute that always involves two or more traffic records subsystems
(i.e., databases or files). For integration, the model performance
measures offer a single performance measure with database-specific
applications that typically are of interest to many States. The samples
included are of course non-exhaustive. Many States will be interested
in establishing links between databases and sub-databases other than
those listed here, and therefore will be interested in measuring the
quality of those other integrations. Note that some of the specific
examples herein involve integration of files within databases rather
than the integration of entire databases.
6. Accessibility: Accessibility reflects the ability of legitimate
users to successfully obtain desired data is different. For the other
performance attributes, the owners and operators of the various
databases and sub-files, examine the data in the files and the internal
workings of the files. In contrast, accessibility is measured in terms
of customer satisfaction. Every database and file in a traffic records
system has a set of legitimate users who are entitled to request and
receive data. The accessibility of the database or sub-file is
determined by obtaining the users' perceptions of how well the system
responds to their requests. Some users' perceptions may be more
relevant to measurement of accessibility than others'. Each database
manager should decide which of the legitimate users of the database
would be classified as principal users, whose satisfaction with the
system's response to requests for data and other transactions will
provide the basis for the measurement of accessibility. Thus, the
generic approach to measurement of database accessibility in the model
performance measured by (1) identifying the principal users of the
database; (2) Querying the principal users to assess (a) their ability
to obtain the data or other services requested and (b) their
satisfaction with the timeliness of the response to their request; and
(3) documenting the method of data collection and the principal users'
responses. How the principal users are contacted and queried is up to
the database managers. Similarly, the extent to which the principal
users' responses are quantified is left to the managers to determine.
However, this measure does require supporting documentation that
provides evidentiary support to the claims of accessibility. This
measure would be best used to gauge the impact of an improvement to a
data system. Surveying the principal users before and after the rollout
of a specific upgrade would provide the most meaningful measure of
improved database accessibility.
Performance Measure Criteria
Each model performance measure was developed in accordance with the
following criteria:
Specific and well-defined: The measures are appropriate and
understandable.
Performance based: The measures are defined by data system
performance, not supporting activities or milestones: ``awarded a
contract'' or ``formed a Traffic Records Coordinating Committee'' are
not acceptable performance measures.
Practical: The measures use data that are readily available at
reasonable cost and can be duplicated.
Timely: The measures should provide an accurate and current--near
real-time--snapshot of the database's timeliness, accuracy,
completeness, uniformity, integration, and accessibility.
Accurate: The measures use data that are valid and consistent with
values that are properly calculated.
Important: The measures capture the essence of this performance
attribute for the data system; for example, an accuracy measure should
not be restricted to a single unimportant data element.
Universal: The measures are usable by all States, though not
necessarily immediately.
These criteria take a broad view of performance measures. For
example, performance on some of the model measures may not change from
year to year. Once a State has incorporated uniform data elements,
established data linkages, or provided appropriate data file access,
further improvement may not be expected. Some States cannot use all
measures. For example, States that do not currently maintain a
statewide data file cannot use measures based on this file (see in
particular the injury data files). Some measures require States to
define a set of critical data elements. Many measures require States to
define their own performance goals or standards. The model measures
should be a guide for States as they assess their data systems and work
to improve their
[[Page 20441]]
performance. Each State should select performance measures most
appropriate to the circumstance and should define and modify them to
fit their specific needs.
Performance Measures
Listed below are the 61 measures classified by data system and
performance attribute.
Crash--Timeliness
Timeliness always reflects the span of time between the occurrence
of some event and the entry of information from the event into the
appropriate database. For the crash database, the events of interest
are crashes. States must measure the time between the occurrence of a
crash and the entry of the report into the crash database. The model
performance measures offer two approaches to measuring the timeliness
of a crash database:
C-T-1: The median or mean number of days from (A) the crash date to
(B) the date the crash report is entered into the database. The median
value is the point at which 50 percent of the crash reports were
entered into the database within a period defined by the State.
Alternatively, the arithmetic mean could be calculated for this
measure.
C-T-2: The percentage of crash reports entered into the database
within XX days after the crash. The XX usually reflects a target or
goal set by the State for entry of reports into the database. The
higher percentage of reports entered within XX days, the timelier the
database. Many States set the XX for crash data entry at 30, 60, or 90
days but any other target or goal is equally acceptable.
Crash--Accuracy
Accuracy reflects the number of errors in information in the
records entered into a database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. Methods for detecting errors
include: (1) Determining that the values entered for a variable or
element are not legitimate codes, (2) matching with external sources of
information, and (3) identifying duplicate records entered for the same
event. The model performance measures offer two approaches to measuring
crash database accuracy:
C-A-1: The percentage of crash records with no errors in critical
data elements. The State selects one or more crash data elements it
considers critical and assesses the accuracy of that element or
elements in all of the crash records entered into the database within a
period defined by the State. Many States consider the following crash
elements critical:
Environmental elements: Record , Location (on/at/distance
from; lat/long, location code), Date, time (can calculate day of week
from this too), Environment contributing factors (up to 3) Location
description (roadway type, location type, roadway-contributing
factors--up to 3) Crash type, severity, involved units,
Harmful events (first harmful, most harmful).
Vehicle/Unit elements: Crash record , vehicle/unit
, VIN decoded sub-file of values for make, model, year, other
decode values, Sequence of events (multiple codes), Harmful events (1st
and most harmful for each vehicle), SafetyNet variables for reportable
vehicles/crashes (carrier name/ID, additional vehicle codes, tow away
due to damage).
Person elements: Crash record , vehicle/unit ,
person , Person type (driver, occupant, non-occupant),
Demographics (age, sex, other), Seating position, Protective device
type (occupant protection, helmet, etc.), Protective device use, Airbag
(presence, deployment: Front, side, both, none), Injury severity (if
this can be sourced through EMS/Trauma/hospital records.
C-A-2: The percentage of in-State registered vehicles on the State
crash file with Vehicle Identification Number (VIN) matched to the
State vehicle registration file.
Crash--Completeness
Completeness reflects both the number of records that are missing
from the database (e.g., events of interest that occurred but were not
entered into the database) and the number of missing (blank) data
elements in the records that are in a database. Completeness has
internal and external aspects. In the crash database, external crash
completeness reflects the number or percentage of crashes for which
crash reports are entered into the database. It is impossible, however,
to establish precisely external crash completeness as the number of
unreported crashes cannot be determined. Internal completeness can be
determined since it reflects the amount of specified information
captured in each individual crash record. The model performance
measures offer three approaches to measuring the internal completeness
of a crash database:
C-C-1: The percentage of crash records with no missing critical
data elements. The State selects one or more crash data elements it
considers critical and assesses internal completeness by dividing the
number of records not missing a critical element by the total number of
records entered into the database within a period defined by the State.
C-C-2: The percentage of crash records with no missing data
elements. The State can assess overall completeness by dividing the
number of records missing no elements by the total number of records
entered into the database within a period defined by the State.
C-C-3: The percentage of unknowns or blanks in critical data
elements for which unknown is not an acceptable value. This measure
should be used when States wish to track improvements on specific
critical data values and reduce the occurrence of illegitimate null
values.
Crash--Uniformity
Uniformity reflects the consistency among the files or records in a
database and may be measured against some independent standard,
preferably a national standard. The model performance measures offer
one approach to measure crash database uniformity:
C-U-1: The number of MMUCC-compliant data elements entered into the
crash database or obtained via linkage to other database(s). The Model
Minimum Uniform Crash Criteria (MMUCC) Guideline is the national
standard for crash records.
Crash-Integration
Integration reflects the ability of records in the crash database
to be linked to a set of records in another of the six core databases--
or components thereof--using common or unique identifiers.
C-I-1: The percentage of appropriate records in the crash database
that are linked to another system or file. Linking the crash database
with the five other core traffic records databases can provide
important information. For example, a State may wish to determine the
percentage of in-State drivers on crash records that link to the driver
file.
Crash-Accessibility
Accessibility reflects the ability of legitimate users to
successfully obtain desired data. The below process outlines one way of
measuring crash database accessibility:
C-X-1: To measure crash accessibility: (1) Identify the principal
users of the crash database; (2) Query the principal users to assess
(A) their ability to obtain the data or other services requested and
(B) their satisfaction with the timeliness of the
[[Page 20442]]
response to their request; (3) Document the method of data collection
and the principal users' responses.
Vehicle-Timeliness
Timeliness always reflects the span of time between the occurrence
of some event and the entry of information from the event into the
appropriate database. For the vehicle database, the State determines
the events of principal interest that will be used to measure
timeliness. For example, a State may determine that the transfer of the
title of the vehicle constitutes a critical status change of that
vehicle record. There are many ways to measure the timeliness of the
entry of a report on the transfer of a vehicle title or any other
critical status change. The model performance measures offer two
general approaches to measuring vehicle database timeliness:
V-T-1: The median or mean number of days from (A) the date of a
critical status change in the vehicle record to (B) the date the status
change is entered into the database. The median value is the point at
which 50 percent of the vehicle record updates were entered into the
database within a period defined by the State. Alternatively, the
arithmetic mean could be calculated for this measure.
V-T-2: The percentage of vehicle record updates entered into the
database within XX days after the critical status change. The XX
usually reflects a target or goal set by the State for entry of reports
into the database. The higher percentage of reports entered within XX
days, the timelier is the database. Many States set the XX for vehicle
data entry at one, five, or 10 days, but any target or goal is equally
acceptable.
Vehicle-Accuracy
Accuracy reflects the number of errors in information in the
records entered into a database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. Methods for detecting errors
include: (1) Determining that the values entered for a variable or
element are not legitimate codes, (2) matching with external sources of
information, and (3) identifying duplicate records have been entered
for the same event. The model performance measures offer one approach
to measuring vehicle database accuracy:
V-A-1: The percentage of vehicle records with no errors in
critical data elements. The State selects one or more vehicle data
elements it considers critical and assesses the accuracy of that
element or elements in all of the vehicles records entered into the
database within a period defined by the State. Many Stats have
identified the following critical data elements: Vehicle Identification
Number (VIN), Current registration status, Commercial or non-CMV, State
of registration, State of title, Stolen flag (as appropriate), Motor
carrier name, Motor carrier ID, and Title brands.
Vehicle-Completeness
Completeness has internal and external aspects. For the vehicle
database, external vehicle completeness reflects the portion of the
critical changes to the vehicle status reported and entered into the
database. It is not possible to determine precisely external vehicle
database completeness because one can never know how many critical
status changes occurred but went unreported. Internal completeness
reflects the amount of specified information captured by individual
vehicle records. It is possible to determine precisely internal vehicle
completeness; for example, one can calculate the percentage of vehicle
records in the database that is missing one or more critical data
elements. The model performance measures offer four approaches to
measuring the completeness of a vehicle database:
V-C-1: The percentage of vehicle records with no missing critical
data elements. The State selects one or more vehicle data elements it
considers critical and assesses internal completeness by dividing the
number of records not missing a critical element by the total number of
records entered into the database within a period defined by the State.
V-C-2: The percentage of records on the State vehicle file that
contain no missing data elements. The State can assess overall
completeness by dividing the number of records missing no elements by
the total number of records entered into the database within a period
defined by the State.
V-C-3: The percentage of unknowns or blanks in critical data
elements for which unknown is not an acceptable value. This measure
should be used when States wish to track improvements on specific
critical data values to reduce the occurrence of illegitimate null
values.
V-C-4: The percentage of vehicle records from large trucks and
buses that have all of the following data elements: Motor Carrier ID,
Gross Vehicle Weight Rating/Gross Combination Weight Rating, Vehicle
Configuration, Cargo Body Type, and Hazardous Materials (Cargo Only).
This is a measure of database completeness in specific critical fields.
Vehicle-Uniformity
Uniformity reflects the consistency among the files or records in a
database and may be measured against some independent standard,
preferably a national standard. The model performance measures offer
one general approach to measuring vehicle database uniformity.
V-U-1: The number of standards-compliant data elements entered into
a database or obtained via linkage to other database(s). These
standards include the Model Minimum Uniform Crash Criteria (MMUCC).
Vehicle-Integration
Integration reflects the ability of records in the vehicle database
to be linked to a set of records in another of the six core databases--
or components thereof--using common or unique identifiers.
V-I-1: The percentage of appropriate records in the vehicle file
that are linked to another system or file. Linking the vehicle database
with the five other core traffic record databases can provide important
information. For example, a State may wish to determine the percentage
of vehicle registration records that link to a driver record.
Vehicle-Accessibility
Accessibility reflects the ability of legitimate users to
successfully obtain desired data. The below process outlines one way of
measuring the vehicle database's accessibility.
V-X-1: To measure accessibility: (1) Identify the principal users
of the vehicle database; (2) Query the principal users to assess (A)
their ability to obtain the data or other services requested and (B)
their satisfaction with the timeliness of the response to their
request; (3) Document the method of data collection and the principal
users' responses.
Driver-Timeliness
Timeliness always reflects the span of time between the occurrence
of some event and the entry of information from the event into the
appropriate database. For the driver database, the State determines the
events of principal interest that shall be used to measure timeliness.
For example, the State may determine that an adverse action against a
driver's license constitutes a critical status change of that driver
record. There are many ways to measure the timeliness of the entry of a
report on an adverse action against a driver's license or any other
critical status change. The
[[Page 20443]]
model performance measures offer two approaches to measuring the
timeliness of the driver database. The first is a true measure of
timeliness from time of conviction to entry into the driver database,
while the second is a measure internal to the agency with custody of
the driver database.
D-T-1: The median or mean number of days from (A) the date of a
driver's adverse action to (B) the date the adverse action is entered
into the database. This measure represents the time from final
adjudication of a citation to entry into the driver database within a
period defined by the State. This process can occur in a number of
ways, from the entry of paper reports and data conversion to a seamless
electronic process. An entry of a citation disposition into the driver
database cannot occur until the adjudicating agency (usually a court)
notifies the repository that the disposition has occurred. Since the
custodial agency of the driver database in most States has no control
over the transmission of the disposition notification many States may
wish to track the portion of driver database timelines involving
citation dispositions that it can control. Measure D-T-2 is offered for
that purpose.
D-T-2: The median or mean number of days from (A) the date of
receipt of citation disposition notification by the driver repository
to (B) the date the disposition report is entered into the driver's
record in the database within a period determined by the State. This
measure represents the internal (to the driver database) time lapse
from the receipt of disposition information to entry into the driver
database within a period defined by the State.
Driver-Accuracy
Accuracy reflects the number of errors in information in the
records entered into a database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. Methods for detecting errors
include: (1) Determining that the values entered for a variable or
element are not legitimate codes, (2) matching with external sources of
information, and (3) identifying duplicate records have been entered
for the same event. The model performance measures offer two approaches
to measuring driver database accuracy:
D-A-1: The percentage of driver records with no errors in critical
data elements. The State selects one or more driver data elements it
considers critical and assesses the accuracy of that element or
elements in all of the driver records entered into the database within
a period defined by the State. Several States have identified the
following critical data elements: Name, Date of birth, Sex, Driver
license number, State of driver license issuance, Date license issued
or renewed, Social Security Number, License type, Restrictions, Crash
involvement, Conviction offenses, Violation date per event, Conviction
date per event, Driver control actions (Suspensions, Revocations,
Withdrawals), and Date of each action.
D-A-2: The percentage of records on the State driver file with
Social Security Numbers (SSN) successfully verified using Social
Security Online Verification (SSOLV) or other means.
Driver-Completeness
Completeness has internal and external aspects. For the driver
database, external completeness reflects the portion of critical driver
status changes that are reported and entered into the database. It is
not possible to determine precisely the external completeness of driver
records because one can never know how many critical driver status
change occurred but went unreported. Internal completeness reflects the
amount of specified information captured in individual driver records.
It is possible to determine precisely internal driver record
completeness. One can, for example, calculate the percentage of driver
records in the database that is missing one or more critical data
elements. The model performance measures offer three approaches to
measuring the internal completeness of the driver database:
D-C-1: The percentage of driver records with no missing critical
data elements. The State selects one or more driver elements it
considers critical and assesses internal completeness by dividing the
number of records not missing a critical element by the total number of
records entered into the database within a period defined by the State.
D-C-2: The percentage of driver records with no missing data
elements. The State can assess overall completeness by dividing the
number of records missing no elements by the total number of records
entered into the database within a period defined by the State.
D-C-3: The percentage of unknowns or blanks in critical data
elements for which unknown is not an acceptable value. This measure
should be used when States wish to track improvements on specific
critical data values and reduce the occurrence of illegitimate null
values.
Driver-Uniformity
Uniformity reflects the consistency among the files or records in a
database and may be measured against an independent standard,
preferably a national standard. The model performance measures offer
one general approach to measuring driver database uniformity:
D-U-1: The number of standards-compliant data elements entered into
the driver database or obtained via linkage to other database(s). The
relevant standards include MMUCC.
Driver-Integration
Integration reflects the ability of records in the driver database
to be linked to a set of records in another of the six core databases--
or components thereof--using common or unique identifiers.
D-I-1: The percentage of appropriate records in the driver file
that are linked to another system or file. Linking the driver database
with the five other core traffic record databases can provide important
information. For example, a State may wish to determine the percentage
of drivers in crashes linked to the adjudication file.
Driver-Accessibility
Accessibility reflects the ability of legitimate users to
successfully obtain desired data. The below process outlines one way of
measuring the driver database's accessibility.
D-X-1: To measure accessibility: (1) Identify the principal users
of the driver database; (2) Query the principal users to assess (A)
their ability to obtain the data or other services requested and (B)
their satisfaction with the timeliness of the response to their
request; (3) Document the method of data collection and the principal
users' responses
Roadway-Timeliness
Timeliness always reflects the span of time between the occurrence
of some event and the entry of information from the event into the
appropriate database. For the roadway database, the State determines
the events of principal interest that will be used to measure
timeliness. A State may determine that the completion of periodic
collection of a critical roadway data element or elements constitutes a
critical status change of that roadway record. For example, one
critical roadway data element that many States periodically collect is
annual average daily traffic (AADT). Roadway database timeliness can be
validly gauged by measuring the
[[Page 20444]]
time between the completion of data collection and the entry into the
database of AADT for roadway segments of interest. There are many ways
to do this. The model performance measures offer two general approaches
to measuring vehicle database timeliness:
R-T-1: The median or mean number of days from (A) the date a
periodic collection of a critical roadway data element is complete
(e.g., Annual Average Daily Traffic) to (B) the date the updated
critical roadway data element is entered into the database. The median
value is the duration within which 50 percent of the changes to
critical roadway elements were updated in the database. Alternatively,
the arithmetic mean is the average number of days between the
completion of the collection of critical roadway elements and when the
data are entered into the database.
R-T-2: The median or mean number of days from (A) roadway project
completion to (B) the date the updated critical data elements are
entered into the roadway inventory file. The median value is the point
at which 50 percent of the updated critical data elements from a
completed roadway project were entered into the roadway inventory file.
Alternatively, the arithmetic mean could be calculated for this
measure. Each State will determine its short list of critical data
elements, which should be a subset of MIRE. For example, it could be
some or all of the elements required for Highway Performance Monitoring
System (HPMS) sites. The database should be updated at regular
intervals or when a change is made to the inventory. For example, when
a roadway characteristic or attribute (e.g., traffic counts, speed
limits, signs, markings, lighting, etc.) that is contained in the
inventory is modified, the inventory should be updated within a
reasonable period.
Roadway-Accuracy
Accuracy reflects the number of errors in information in the
records entered into a database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. Methods for detecting errors
include: (1) Determining that the values entered for a variable or
element are not legitimate codes, (2) matching with external sources of
information, and (3) identifying duplicate records have been entered
for the same event. The model performance measures offer one approach
to measuring roadway database accuracy:
R-A-1: The percentage of all road segment records with no errors in
critical data elements. The State selects one or more roadway data
elements it considers critical and assesses the accuracy of that
element or elements in all of the roadway records within a period
defined by the State. Many States consider the HPMS standards to be
critical.
Roadway-Completeness
Completeness has internal and external aspects. For the roadway
database, external roadway completeness reflects the portion of road
segments in the State for which data are collected and entered into the
database. It is very difficult to determine precisely external roadway
completeness because many States do not know the characteristics or
even the existence of roadway segments that are non-State owned,
maintained, or reported in the HPMS. Internal completeness reflects the
amount of specified information that is captured in individual road
segment records. It is possible to determine precisely internal roadway
completeness. One can, for example, calculate the percentage of roadway
segment records in the database that is missing one or more critical
elements (e.g., number of traffic lanes. The model performance measures
offer four general approaches to measuring the roadway database's
internal completeness:
R-C-1: The percentage of road segment records with no missing
critical data elements. The State selects one or more roadway elements
it considers critical and assesses internal completeness by dividing
the number of records not missing a critical element by the total
number of roadway records in the database.
R-C-2: The percentage of public road miles or jurisdictions
identified on the State's basemap or roadway inventory file. A
jurisdiction may be defined by the limits of a State, county, parish,
township, Metropolitan Planning Organization (MPO), or municipality.
R-C-3: The percentage of unknowns or blanks in critical data
elements for which unknown is not an acceptable value. This measure
should be used when States wish to track improvements on specific
critical data elements and reduce the occurrence of illegitimate null
values.
R-C-4: The percentage of total roadway segments that include
location coordinates, using measurement frames such as a GIS basemap.
This is a measure of the database's overall completeness.
Roadway-Uniformity
Uniformity reflects the consistency among the files or records in a
database and may be measured against some independent standard,
preferably a national standard. The model performance measures offer
one general approach to measuring roadway database uniformity:
R-U-1: The number of Model Inventory of Roadway Elements (MIRE)-
compliant data elements entered into a database or obtained via linkage
to other database(s).
Roadway-Integration
Integration reflects the ability of records in the roadway database
to be linked to a set of records in another of the six core databases--
or components thereof--using common or unique identifiers.
R-I-1: The percentage of appropriate records in a specific file in
the roadway database that are linked to another system or file. For
example, a State may wish to determine the percentage of records in the
State's bridge inventory that link to the basemap file.
Roadway-Accessibility
Accessibility reflects the ability of legitimate users to
successfully obtain desired data. The below process outlines one way of
measuring roadway database accessibility:
R-X-1: To measure accessibility of a specific file in the roadway
database: (1) Identify the principal users of the file; (2) Query the
principal users to assess (A) their ability to obtain the data or other
services requested and (B) their satisfaction with the timeliness of
the response to their request; (3) Document the method of data
collection and the principal users' responses.
Citation/Adjudication-Timeliness
Timeliness always reflects the span of time between the occurrence
of some event and the entry of information from the event into the
appropriate database. For the citation and adjudication databases, the
State determines the events of principal interest that will be used to
measure timeliness. Many States will include the critical events of
citation issuance and citation disposition among those events of
principal interest used to track timeliness. There are many ways to
measure the timeliness of either citation issuance or citation
disposition. The model performance measures offer one general approach
to measuring citation and adjudication database timeliness:
C/A-T-1: The median or mean number of days from (A) the date a
citation is issued to (B) the date the
[[Page 20445]]
citation is entered into the statewide citation database, or a first
available repository. The median value is the point at which 50 percent
of the citation records were entered into the citation database within
a period defined by the State. Alternatively, the arithmetic mean could
be calculated for this measure.
C/A-T-2: The median or mean number of days from (A) the date of
charge disposition to (B) the date the charge disposition is entered
into the statewide adjudication database, or a first available
repository. The median value is the point at which 50 percent of the
charge dispositions were entered into the statewide database.
Alternatively, the arithmetic mean could be calculated for this
measure.
Note: Many States do not have statewide databases for citation
or adjudication records. Therefore, in some citation and
adjudication data systems, timeliness and other attributes of data
quality should be measured at individual first available
repositories.
Citation/Adjudication-Accuracy
Accuracy reflects the number of errors in information in the
records entered into a database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. Methods for detecting errors
include: (1) Determining that the values entered for a variable or
element are not legitimate codes, (2) matching with external sources of
information, and (3) identifying duplicate records that have been
entered for the same event. The State selects one or more citation data
elements and one or more charge disposition data elements it considers
critical and assesses the accuracy of those elements in all of the
citation and adjudication records entered into the database within a
period of interest. The model performance measures offer two approaches
to measuring citation and adjudication database accuracy:
C/A-A-1: The percentage of citation records with no errors in
critical data elements. The State selects one or more citation data
elements it considers critical and assesses the accuracy of that
element or elements in all of the citation records entered into the
database within a period defined by the State. Below is a list of
suggested critical data elements.
C/A-A-2: The percentage of charge disposition records with no
errors in critical data elements. The State selects one or more charge
disposition data elements it considers critical and assesses the
accuracy of that element or elements for the charge-disposition records
entered into the database within a period defined by the State. Many
States have identified the following as critical data elements:
Critical elements from the Issuing Agency include the offense/charge
code, date, time, officer, Agency, citation number, crash report number
(as applicable), and BAC (as applicable). Critical elements from the
Citation Data include the Offender's name, driver license number, age,
and sex. Critical data elements from the Charge Disposition/
Adjudication include the offender's name, driver license number, age,
sex, and citation number. From the charge Disposition/Adjudication:
court, date of receipt, date of disposition, disposition, and date of
transmittal to DMV (as applicable).
Citation/Adjudication-Completeness*
Completeness has internal and external aspects. For the citation/
adjudication databases, external completeness can only be assessed by
identifying citation numbers for which there are no records. Missing
citations should be monitored at the place of first repository.
Internal completeness reflects the amount of specified information that
is captured in individual citation and charge disposition records. It
is possible to determine precisely internal citation and adjudication
completeness. One can, for example, calculate the percentage of
citation records in the database that are missing one or more critical
data elements. The model performance measures offer three approaches to
measuring internal completeness:
C/A-C-1: The percentage of citation records with no missing
critical data elements. The State selects one or more citation data
elements it considers critical and assesses internal completeness by
dividing the number of records not missing a critical element by the
total number of records entered into the database within a period
defined by the State.
C/A-C-2: The percentage of citation records with no missing data
elements. The State can assess overall completeness by dividing the
number of records missing no elements by the total number of records
entered into the database.
C/A-C-3: The percentage of unknowns or blanks in critical citation
data elements for which unknown is not an acceptable value. This
measure should be used when States wish to track improvements on
specific critical data elements and reduce the occurrence of
illegitimate null values.
Note: These measures of completeness are also applicable to the
adjudication file.
Citation/Adjudication-Uniformity *
Uniformity reflects the consistency among the files or records in a
database and may be measured against some independent standard,
preferably a national standard. The model performance measures offer
two general approaches to measuring database uniformity:
C/A-U-1: The number of Model Impaired Driving Record Information
System (MIDRIS)-compliant data elements entered into the citation
database or obtained via linkage to other database(s).
C/A-U-2: The percentage of citation records entered into the
database with common uniform statewide violation codes. The State
identifies the number of citation records with common uniform violation
codes entered into the database within a period defined by the State
and assesses uniformity by dividing this number by the total number of
citation records entered into the database during the same period.
* Note: These measures of uniformity are also applicable to the
adjudication file.
Citation/Adjudication-Integration *
Integration reflects the ability of records in the citation
database to be linked to a set of records in another of the six core
databases--or components thereof--using common or unique identifiers.
C/A-I-1: The percentage of appropriate records in the citation
files that are linked to another system or file. Linking the citation
database with the five other core traffic record databases can provide
important information. For example, a State may wish to determine the
percentage of DWI citations that have been adjudicated.
* Note: This measure of integration is also applicable to the
adjudication file.
Citation/Adjudication-Accessibility *
Accessibility reflects the ability of legitimate users to
successfully obtain desired data. The below process outlines one way of
measuring the citation database's accessibility.
C/A-X-1: To measure accessibility of the citation database: (1)
Identify the principal users of the citation database; (2) Query the
principal users to assess (A) their ability to obtain the data or other
services requested and (B) their satisfaction with the timeliness of
the response to their request; (3) Document the method of data
collection and the principal users' responses. The EMS/Injury
Surveillance database is actually a set of related databases. The
principal files of interest are: Pre-hospital
[[Page 20446]]
Emergency Medical Services (EMS) data, Hospital Emergency Department
Data Systems, Hospital Discharge Data Systems, and State Trauma
Registry File, State Vital Records. States typically wish to measure
data quality separately for each of these files. These measures may be
applied to each of the EMS/Injury Surveillance databases individually.
Injury Surveillance-Timeliness *
Timeliness always reflects the span of time between the occurrence
of some event and the entry of information from the event into the
appropriate database. For the EMS/Injury Surveillance databases, the
State determines the events of principal interest that will be used to
measure timeliness. A State may, for example, determine that the
occurrence of an EMS run constitutes a critical event to measure the
timeliness of the EMS database. As another example, a State can select
the occurrence of a hospital discharge as the critical event to measure
the timeliness of the hospital discharge data system. There are many
ways to measure the timeliness of the EMS/Injury Surveillance
databases. The model performance measures offer two general approaches
to measuring timeliness:
I-T-1: The median or mean number of days from (A) the date of an
EMS run to (B) the date when the EMS patient care report is entered
into the database. The median value is the point at which 50 percent of
the EMS run reports were entered into the database within a period
defined by the State. Alternatively, the arithmetic mean could be
calculated for this measure.
I-T-2: The percentage of EMS patient care reports entered into the
State EMS discharge file within XX* days after the EMS run. The XX
usually reflects a target or goal set by the State for entry of reports
into the database. The higher percentage of reports entered within XX
days, the timelier the database. Many States set the XX for EMS data
entry at 5, 30, or 90 days, but any target or goal is equally
acceptable.
* Note: This measure of timeliness is also applicable to the
following files: State Emergency Dept. File, State Hospital
Discharge File, State Trauma Registry File, & State Vital Records.
Injury Surveillance-Accuracy *
Accuracy reflects the number of errors in information in the
records entered into a database. Error means the recorded value for
some data element of interest is incorrect. Error does not mean the
information is missing from the record. Erroneous information in a
database cannot always be detected. Methods for detecting errors
include: 1) determining that the values entered for a variable or
element are not legitimate codes, 2) matching with external sources of
information, and 3) identifying duplicate records have been entered for
the same event. The model performance measures offer one general
approach to measuring the accuracy of the injury surveillance databases
that is applicable to each of the five principal files:
I-A-1: The percentage of EMS patient care reports with no errors in
critical data elements. The State selects one or more EMS data elements
it considers critical--response times, for example--and assesses the
accuracy of that element or elements for all the records entered into
the database within a period defined by the State. Critical EMS/Injury
Surveillance Data elements used by many States include: Hospital
Emergency Department/Inpatient Data elements such as E-code, date of
birth, name, sex, admission date/time, zip code of hospital, emergency
dept. disposition, inpatient disposition, diagnosis codes, and
discharge date/time. Elements from the Trauma Registry Data (National
Trauma Data Bank [NTDB] standard) such as E-code, date of birth, name,
sex, zip code of injury, admission date, admission time, inpatient
disposition, diagnosis codes, zip code of hospital, discharge date/
time, and EMS patient report number. Data from the EMS Data (National
Emergency Medical Services Information System [NEMSIS] standard)
includes date of birth, name, sex, incident date/time, scene arrival
date/time, provider's primary impression, injury type, scene departure
date/time, destination arrival date/time, county/zip code of hospital,
and county/zip code of injury Critical data elements from the Death
Certificate (Mortality) Data (National Center for Health Statistics
[NCHS] standard) include date of birth, date of death, name, sex,
manner of death, underlying cause of death, contributory cause of
death, county/zip code of death, and location of death.
* Note: This measure of accuracy is also applicable to the
following files: State Emergency Dept. File, State Hospital
Discharge File, State Trauma Registry File, & State Vital Records.
Injury Surveillance-Completeness*
Completeness has internal and external aspects. For EMS/Injury
Surveillance databases, external completeness reflects the portion of
critical events (e.g., EMS runs, hospital admissions, etc.) that are
reported and entered into the databases. It is not possible to
determine precisely external EMS/injury surveillance completeness
because once can never know the how many critical events occurred but
went unreported. Internal completeness reflects the amount of specified
information that is captured in individual EMS run records, State
Emergency Department records, State Hospital Discharge File records,
and State Trauma Registry File records. It is possible to determine
precisely internal EMS/Injury Surveillance completeness. One can, for
example, calculate the percentage of EMS run records in the database
that are missing one or more critical data elements. The model
performance measures offer three approaches to measuring completeness
for each of the files:
I-C-1: The percentage of EMS patient care reports with no missing
critical data elements. The State selects one or more EMS data elements
it considers critical and assesses internal completeness by dividing
the number of EMS run records not missing a critical element by the
total number of EMS run records entered into the database within a
period defined by the State.
I-C-2: The percentage of EMS patient care reports with no missing
data elements. The State can assess overall completeness by dividing
the number of records missing no elements by the total number of
records entered into the database.
I-C-3: The percentage of unknowns or blanks in critical data
elements for which unknown is not an acceptable value. This measure
should be used when States wish to track improvement on specific
critical data values and reduce the occurrence of illegitimate null
values. E-code, for example, is an appropriate EMS/Injury Surveillance
data element that may be tracked with this measure.
* Note: These measures of completeness are also applicable to
the following files: State Emergency Dept. File, State Hospital
Discharge File, State Trauma Registry File, & State Vital Records.
Injury Surveillance-Uniformity
Uniformity reflects the consistency among the files or records in a
database and may be measured against an independent standard,
preferably a national standard. The model performance measures offer
one approach to measuring uniformity that can be applied to each
discrete file using the appropriate standard as enumerated below.
I-U-1: The percentage of National Emergency Medical Services
Information System (NEMSIS)-
[[Page 20447]]
compliant data elements on EMS patient care reports entered into the
database or obtained via linkage to other database(s).
I-U-2: The number of National Emergency Medical Services
Information System (NEMSIS)-compliant data elements on EMS patient care
reports entered into the database or obtained via linkage to other
database(s).
The national standards for many of the other major EMS/Injury
Surveillance database files are: The Universal Billing 04 (UB04) for
State Emergency Department Discharge File and State Hospital Discharge
File; the National Trauma Data Standards (NTDS) for State Trauma
Registry File; and the National Association for Public Health
Statistics and Information Systems (NAPHSIS) for State Vital Records.
Injury Surveillance-Integration*
Integration reflects the ability of records in the EMS database to
be linked to a set of records in another of the six core databases--or
components thereof--using common or unique identifiers.
I-I-1: The percentage of appropriate records in the EMS file that
are linked to another system or file. Linking the EMS file to other
files in the EMS/Injury Surveillance database or any of the five other
core databases can provide important information. For example, a State
may wish to determine the percentage of EMS records that link to the
trauma file that are linked to the EMS file.
* Note: This measure of integration is also applicable to the
following files: State Emergency Dept. File, State Hospital
Discharge File, State Trauma Registry File, & State Vital Records.
Injury Surveillance-Accessibility *
Accessibility reflects the ability of legitimate users to
successfully obtain desired data.
I-X-1: To measure accessibility of the EMS file: (1) Identify the
principal users of the EMS file, (2) Query the principal users to
assess (A) their ability to obtain the data or other services requested
and (B) their satisfaction with the timeliness of the response to their
request, and (3) Document the method of data collection and the
principal users' responses
Note: This measure of accessibility is also applicable to the
State Emergency Dept. File, the State Hospital Discharge File, the
State Trauma Registry File, & State Vital Records.
Recommendations
While use of the performance measures is voluntary, States will be
better able to track the success of upgrades and identify areas for
improvement in their traffic records systems if they elect to utilize
the measures appropriate to their circumstances. Adopting the measures
will also put States ahead of the curve should performance metrics be
mandated in any future legislation. The measures are not exhaustive.
They describe what to measure and suggest how to measure it, but do not
recommend numerical performance goals. The measures attempt to capture
one or two key performance features of each data system performance
attribute. States may wish to use additional or alternative measures to
address specific performance issues.
States that elect to use these measures to demonstrate progress in
a particular system should start using them immediately. States should
begin by judiciously selecting the appropriate measures and modifying
them as needed. States should use only the measures for the data system
performance attributes they wish to monitor or improve. No State is
expected to use a majority of the measures, and States may wish to
develop their own additional measures to track State-specific issues or
programs.
Once States have developed their specific performance indices, they
should be measured consistently to track changes over time. Since the
measures will vary considerably from State to State, it is unlikely
that they could be used for any meaningful comparisons between States.
In any event, NHTSA does not anticipate using the measures for
interstate comparison purposes.
Notes on Terminology Used
The following terms are used throughout the document:
Data system: One of the six component State traffic records
databases, such as crash, injury surveillance, etc.
Data file (such as ``crash file'' or ``State Hospital Discharge
file''): A data system may contain a single data file--such as a
State's driver file--or more than one, e.g., the injury system has
several data files.
Record: All the data entered in a file for a specific event (a
crash, a patient hospital discharge, etc.).
Data element: Individual fields coded within each record.
Data element code value: The allowable code values or attributes
for a data element.
Data linkages: The links established by matching at least one data
element in a record in one file with the corresponding element or
elements in one or more records in another file or files.
State: The 50 States, the District of Columbia, Puerto Rico, the
territories, and the Bureau of Indian Affairs. These are the
jurisdictions eligible to receive State data improvement grants.
Defining and Calculating Performance Measures
Specified number of days: Some measures are defined in terms of a
specified number of days (such as 30, 60, or 90). Each State can
establish its own period for these measures.
Defining periods of interest: States will need to define periods of
interest for several of the measures. These periods should be of an
appropriate length for the data being gathered. A State may wish to
calculate the timeliness of its crash database on an annual basis. The
same State may also wish to calculate the timeliness of their other
databases (e.g., driver, vehicle) on a monthly or weekly basis because
of their ability to generate revenue. These decisions are left to the
State to make per the situation and their data needs.
Critical data elements: Some measures are defined using a set of
``critical data elements.'' Unless a measure is specifically defined in
a national standard, each State can define its own set of critical data
elements. Data elements that many States use are presented as examples
for each data system.
When measures should be calculated: Many measures can be calculated
and monitored using data from some period of time such as a month, a
quarter, or a year. All measures should be calculated and monitored at
least annually. A few measures are defined explicitly for annual files.
States should calculate measures at the same time or times each year
for consistency in tracking progress.
Missing data: Some completeness measures are defined in terms of
``missing'' data, such as C-C-1--the percentage of crash records with
no missing critical data elements. ``Missing'' means that the data
element is not coded--nothing was entered. Many data elements have null
codes that indicate that information is not available for some reason.
Typical null codes are ``not available,'' ``not documented,'' ``not
known,'' or ``not recorded.'' A data element with a null value is not
counted as missing data because it does contain a valid code, even
though the data element may contain no useful information. The States
should determine under what
[[Page 20448]]
circumstances a null value is valid for a particular data element. For
accuracy measures, a data element with missing data or a null value is
not considered an error. It is up to the State--specifically, the
custodians of a database--to decided if null codes should be accepted
as legitimate entries or treated as missing values.
How to define ``entered into a database'': Some records do not have
all their data entered into a database at the same time. In general, an
event is considered to be ``entered into a database'' when a specified
set of critical data elements has been entered. In fact, many databases
will not accept a record until all data from a critical set are
available. States may define ``entered into a database'' using their
own data entry and data access processes.
How to calculate a timeliness measure: For all systems, there will
be a period of time between the event generating the record and when
the information is entered into the file (or is available for use). The
model performance measures include several methods to define a single
number that captures the entire distribution of times. Each method is
appropriate in different situations.
The median time for events to be entered into the file can be
calculated as the point at which 50 percent of events within a period
of interest are entered into the file.
The mean time for events to be entered into the file (counting all
events). The mean can be calculated as the average (the sum of the
times for all events divided by the number of events).
The percentage of events on file within some fixed time (such as 24
hours or 30 days).
Tradeoffs between timeliness and completeness: Generally speaking,
the relationship between timeliness and completeness is inversely
proportional: The more timely the data, the less complete it is and
vice versa. This is because many data files have records or data
elements added well past the date of the event producing the record, so
the files may be incomplete when the performance measure is calculated.
There are three methods of choosing data to calculate the performance
measures that offer different combinations of timeliness and
completeness. Depending on the need for greater timeliness or
completeness, users should choose accordingly.
For example, if timeliness is important when calculating the first
Crash Completeness measure C-C-1--the percentage of crash records with
no missing critical data elements--could be calculated in the following
manner: (1) Select the period: Calendar year 2007 crash file; (2)
Select the date for calculation: April 1 of the following year. So
calculate using the 2007 crash file as it exists on April 1, 2008; (3)
Calculate: Take all crashes from 2007 on file as of April 1, 2008;
calculate the percentage with missing data for one or more critical
data elements.
This method offers several advantages. It is easy to understand and
use, and can produce performance measures in a timely manner. Its
disadvantage is that performance measures calculated fairly soon after
the end of the data file's period may not be based on complete data.
For example, NHTSA's Fatality Analysis Reporting System (FARS) is not
closed and complete for a full year; the 2007 file was not closed until
Dec. 31, 2008. Timeliness measures will exclude any records that have
not yet been entered by the calculation date, so timeliness measures
may make the file appear to be timelier than it will be when the file
is closed and completed. Completeness measures will exclude any
information entered after the calculation date for records on file.
Completeness measures calculated on open files will make those files
appear less complete than measures calculated on files that are closed
and completed.
When completeness is most important the performance measure could
be calculated after a file (say an annual file) is closed and no
further information can be added to it. This method reverses the simple
method's advantages and disadvantages, providing performance measures
that are accurate but not timely. The final FARS file, for example, is
a very complete database. Its completeness, however, comes at the
expense of timeliness. In comparison, the annual FARS file is less
complete, but is more timely.
Another-preferable-method calculates a performance measure using
all records entered into a file during a specified period. The
timeliness measures produced by this method will be accurate but the
completeness and accuracy measures may not, because the records entered
during a given time period may not be complete when the measure is
calculated. For example, the Crash Timeliness measure C-T-1--the median
or mean number of days from (A) the crash date to (B) the crash report
is entered into the database--could be calculated as follows: (1)
Select the period: calendar year 2007; (2) Take all records entered
into the State crash file during the period: if the period is calendar
year 2007 the crashes could have occurred in 2007 or 2006 (or perhaps
even earlier depending on the State's reporting criteria); (3)
Calculate the measure: The median or mean time between the crash date
and the date when entered into the crash file.
States should choose methods that are accurate, valid, reliable,
and useful. They may choose different methods for different measures.
Or they may use two different methods for the same measure, for example
calculating a timeliness measure first with an incomplete file (for
example the 2007 crash file on April 1, 2008) and again with the
complete and closed file (the 2007 crash file on January 1, 2009, after
it is closed). Once methods have been selected for a measure, States
should be consistent and use the same methods to calculate that measure
using the same files in the same way each year. To accurately gauge
progress, States must compare measures calculated by the same method
using the same files for successive years.
Privacy issues in file access and linkage: Data file access and
linkage both raise broad issues of individual privacy and the use of
personal identifiers. The Driver Privacy Protection Act (DPPA), the
Health Insurance Portability and Accountability Act (HIPAA), and other
regulations restrict the release of personal information on traffic
safety data files. Information in many files may be sought for use in
legal actions. All data file linkage and all data file access actions
must consider these privacy issues.
Authority: 44 U.S.C. Section 3506(c)(2)(A).
Jeffrey Michael,
Acting Associate Administrator, National Center for Statistics and
Analysis.
[FR Doc. 2011-8738 Filed 4-11-11; 8:45 am]
BILLING CODE 4910-59-P