Monitoring and Evaluation: Bad Indicators lead to Bad Policy Making

Justin Peters over at Slate asks a very good question, “Will Arresting People Who Aren’t Very Drunk Reduce Drunk-Driving Fatalities?“.

In regulation development, if a previous restricting policy policy did little to meet the policy objective, it makes little sense to make an even stricter policy. A stricter policy will just create more costs or lower welfare with no discernible benefits.

Justin Peters notes that the NTSB reductions in the upper limit of Blood Alcohol Content (BAC) that defines drunkenness did little to stem drunk driving fatalities. Justin added a graph from the NTSB that shows that the share of drunk driving fatalities has been constant since about 1995.

The problem is that this analysis relies on a poor indicator to measure the effectiveness of previous BAC restrictions.

The share of drunk driving fatalities is equal to: # of drunk driving fatalities / # of total vehicle fatalities.

This ratio will remain constant if the actual number of drunk driving fatalities is falling in line with the total number of vehicle fatalities.

Now, the indicator the NTSB uses could be okay, but only if the safety improvement in vehicles equally improves fatality rates for both drunk and sober drivers. In fact, using this ratio only makes sense if for all possible factors that influence the number of drunk and sober fatalities are identical. For example, a defensive driving advertisement campaign could make sober drivers less likely to be in an accident, but not drunk drivers. Improvements in car safety influence both rates, but only if the improvements to vehicle safety equally reduce fatality rates in both types of collisions.

In general, using only the fatality rate obfuscates the fact that there could be more accidents but a falling number of fatalities, because cars are much safer. People could be drinking and driving more, but the number of drunk driving related fatalities could be falling.