Recap of the Veil of Darkness Hypothesis

Last week, I attended a meetup presentation titled Testing the Veil of Darkness Hypothesis with LA Open Data. Although the presentation was not recorded, a blog post on We Quant LA sums up the findings.

For the record, this work was done by a team from the NewMet Data Science bootcamp where I am a mentor. I was available to the people who did the actual work, including presenter Brian Becker, but I didn't directly contribute to it.

I'm not going to recap the work very much. In this post I wanted to comment on a bit of the discussion that came from people who attended the talk. I think an open discussion of any evidence based result is extremely important. We should debate whether or not gravity is real to better understand the theory and the evidence for it. Obviously, we have a lot more evidence for gravity than the veil of darkness hypothesis. I find the team's conclusions compelling, yet I remain open to a well formed rhetoric that convinces me otherwise.

The specific conclusion that was most controversial was the finding that based on the veil of darkness hypothesis, the group found no county-wide evidence of racial profiling in traffic stops, although a few outier districts might be worth further review. Please see their blog post for the details on their arguement.

I think some attendees found this result unpalatable and I believe the source of that reaction was their pre-existing belief that the police do act upon racial biases. This is a complicated and delicate topic.

First off, I don't think the group's result is proof that police do not act on racial bias. Their result is specific to traffic stops only, covers only LA county, and performs a hypothesis test for methodological reasons only on a specific time of day on weekdays. It's also worth mentioning that the basis of their analysis is on what is provided in the LA Open Data portal. I have no specific reason to believe this dataset is incomplete, but if it were shown to be, then that could skew their result.

It is entirely possible that police act on racial bias in ways that are unobservable in the group's analysis. I state this not to imply that it's true but to emphasize that the scope of the conclusion needs to be properly understood. In my opinion, this result is invariant to the presence or absence of bias. In fact, if one wanted to search for evidence of racial bias, pragmatic results like these simply help refine the scope to explore. Perhaps it is the case that police make traffic stops without prejudice but issue citations with prejudice. Again, I have no evidence of this but state it as a testable hypothesis.

I understand that the implication that the police do not act with racial prejudice is offensive to people who hold strong believes that they do. My personal opinion is that there certainly exist some bad apple police officers in the country who inappropriately respond to certain situations based on the race of citizens they interact with. What I am unclear about is the percentage of such bad apple officers that exist and how quickly department procedures are able to find and eliminate them. To be clear - I am not saying we do not have a problem in this country. My perspective at the time of writing this is that I am unresolved on the magnitude of the issue. I am also uncertain about whether or not the problems are indeed produced by a few bad apples or systematic in some way. Any and all opportunities for people's civil rights and due process to be violated is deeply troubling and something we should fix.

In my opinion, some attendees had a difficult time reconciling a belief in police racial profiling and the null result of this analysis. But rather than taking the analysis within its scope or seeking to dismantle it scientifically, several ad hominem or irrelevant attacks were made that I think will be interesting to discuss.

One attendee stated that the results were of no consequence until people from the statistics community could review it. I was slightly puzzled by this comment since the analysis is very statistical in it's nature and was presented that way. Of course, someone knowledgable about statistics might find a methodological or analytical error in the work. But until such a criticism is levied, I believe it's incorrect to dispute the result simply because someone might look at it and find an issue. A statistical arguement was made, and it should be upheld unless a direct flaw in it can be pointed out.

Another attendee described the presentation as "dangerous" and said it lacked peer review. It was pointed out that these results were not published in a peer reviewed publication, and therefore have not earned the associated credibility. Yet, these results were in fact peer reviewed by the NewMet mentors, including myself. Review of any kind does not constitute proof. Perhaps a serious error or methodological mistake was made that everyone involved failed to see. This is entirely possible, but in my opinion, the proper next step is to issolate and expose the flaw, rather than asserting that we have no guarentee that a flaw doesn't exist. That's true of all theories. Falsifiability is a hallmark of the scientific method.

One commentor said that these results couldn't possibly be correct because we know police racial profiling exists. I don't think it's correct to state that we know this with certainty, but if we take that for granted, it's still not a satisfying contradition to me. Knowing that profiling exists, and finding that traffic stop data does not contain profiling would only imply that the profiling manifests in other ways. After all, a corrupt police officer probably manages to do their job without prejudice some of the time.

A rather novel obsevation made by one attendee was that the analysis did not consider the make and model of the car, which they asserted is highly correlated with race. I'm not sure if I'd agree that it's highly correlated, but certainly demographic patterns exist in automobile brand ownership, and these could absolutely influence the decision making of a prejudice officer. This would be a form of leakage that would weaken the analysis. I think this is the objection from the evening that I found most interesting. If make and model of car (or other features like bumper stickers) were perfectly predictive of race, that would entirely invalidate the methodology.

The degree to which observable characteristics of the vehicle in night conditions is predictive of race weakens the methodology used in the veil of darkness hypothesis. My instincts tell me this is very weakly correlated. However, I'm very open to evidence to the contrary. Burden of proof on the claimant.

Overall I did not find any of the raised objections to have altered my interpretation of their result. I'm naturally open to new arguments, evidence, and perspectives on this, but I think it's critical that people think more about the rhetoric of their arguments if they wish to convince other people who are scientifically minded. Disagreeing with a result because you don't like it or because it just seems wrong is not sound. proper rhetoric requires an invalidation of the method, exposure of a blunder in the analysis, exposing a bias in the input data, or proposing a means of falsifying the result. Most importantly, I think it would have benefited many attendees to think about the result within it's limited, but important scope.

I was most surprised that no one attempted to dismantle the very veil of darkness hypothesis directly. The core idea is that under daylight conditions, police officers have more ability to observe the race of a driver. While I think this is true, I also believe good empirical studies could convince me that this premise is flawed. As with anything I accept, I'm open to evidence that will change my mind. But for the time being, I find these results compelling.