Blog

Not all evidence is created equally

In the policy world, not all evidence is created equally.

I’m not talking about forensic or criminal evidence (though those areas have hierarchies too). I’m referring to the evidence we need to make a good policy choice. Policy decisions in policing include if my police department should support a second responder program to prevent domestic abuse, or should dedicate officers to teach D.A.R.E. in our local school? (The answer to both questions is no). The evidence I’m talking about here is not just ‘what works’ to reduce crime significantly, but also how it works (what mechanism is taking place), and when it works (in what contexts the tactic may be effective).

We can harvest ideas about what might work from a variety of sources. Telep and Lum found that a library database was the least accessed source for officers when learning about what tactics might work[1]. That might be because for most police officers, a deep dive into the academic literature is like being asked to do foot patrol in Hades while wearing a nylon ballistic vest. But it also means that the ‘what works’, ‘how works’, and ‘when works’ of crime reduction are never fully understood. Too many cops unfortunately rely on their intuition, opinion or unreliable sources, as noted by Ken Pease and Jason Roach:

Police officers may be persuaded by the experience of other officers, but seldom by academic research, however extensive and sophisticated. Collegiality among police officers is an enduring feature of police culture. Most officers are not aware of, are not taught about and choose not to seek out relevant academic research. When launching local initiatives, their first action tends to be the arrangement of visits to similar initiatives in other forces, rather than taking to the journals.[2]

In fact, not only do officers favor information from other cops, but it also has to be from the right police officers. Just about everyone in policing at some point has been told to forget what they learned at the police academy. That was certainly my experience. And those courses were usually taught by experienced police officers! It’s too easy to end up with a very narrow and restricted field of experience on which to draw.

Fortunately, while a basic understanding of different research qualities is helpful, you do not need to have an advanced degree in research methodology to be able to implement evidence-based policing from a range of evidence sources. It’s sufficient to appreciate that there is a hierarchy of evidence in which to place your trust, and to have a rudimentary understanding of the differences between them. I’m not dismissing any forms of research, but I am saying that some research is more reliable than others and more useful for operational decision-making. For example internal department research may not be great for choosing between prevention strategy options, but it is hugely useful for identifying the range of problems.

There are a number of hierarchies of evidence available. Criminologists are familiar with the Maryland Scientific Methods Scale devised by Larry Sherman and colleagues[3]. In this scale, studies are ranked from 1 (weak) to 5 (strong) based on the study’s capacity to show causality and limit the effects of selection bias. Selection bias occurs when researchers choose participants in a study rather than allow random selection. It’s not necessarily an intent to harm the study, but it is a common problem. When we select participants or places to be part of a study, we can unconsciously choose places that will perform better than randomly selected sites. We cherry-pick the subjects, and so bias the study and get the result we want. It’s why so many supposedly great projects have been difficult to replicate.

But the Maryland scale only addresses academic research, and police officers get knowledge and information from a range of sources. Rob Briner’s work in evidence-based management also stresses this. Go into any police canteen or break room and you hear anecdote after anecdote (I wrote a few weeks ago about the challenges of ‘experience’). These examples of homespun wisdom—also known as carefully selected case studies—are often illustrative of an unusual case and not a story of the mundane and ordinary that we deal with every day.

This professional expertise can also be supplemented by the stakeholder concerns of the local community (including the public or the local enforcement community such as other police departments or prosecutors). Knowing what is important to stakeholders and innovative approaches adopted by colleagues is useful in the hunt for a solution to a crime or disorder problem.

Police also get information from experts and internal reports. In larger forces, reports from statistics or crime analysis units can be important sources of information. Organizational data are therefore useful to officers who try and replicate crime reduction that a colleague in a different district appears to have achieved. All of these sources are important to some police officers, and they deserve a place on the hierarchy. But because they are hand selected, can be abnormal, might be influenced internally, or have not been subjected to a lot of scrutiny, they get a lower place on the chart.

In the figure on this page, I’ve pulled together a hierarchy of evidence from a variety of sources and tried to combine them in such a way that you can appreciate different sources and their benefits and concerns. And hopefully this might help you appreciate a little how to interpret the evidence from sources such as The National Institute of Justice’s CrimeSolutions.gov website and The UK College of Policing’s Crime Reduction Toolkit. In a later post I’ll try and expand on each of the levels (from 0 to 5*).

 

  1. Telep, C.W. and C. Lum, The receptivity of officers to empirical research and evidence-based policing: An examination of survey data from three agencies. Police Quarterly, 2014. 17(4): p. 359-385.
  2. Pease, K. and J. Roach, How to morph experience into evidence in Advances in Evidence-Based Policing, J. Knutsson and L. Tompson, Editors. 2017, Routledge. p. 84-97.
  3. Sherman, L.W., et al., Preventing Crime: What works, what doesn’t, what’s promising. 1998, National Institute of Justice: Washington DC.

It’s time for Compstat to change

If we are to promote more thoughtful and evidence-based policing, then Compstat has to change. The Compstat-type crime management meeting has its origins in Bill Bratton’s need to extract greater accountability from NYPD precinct commanders in late 1990s New York. It was definitely innovative for policing at the time, and instigated many initiatives that are hugely beneficial to modern policing (such as the growth of crime mapping). And arguably it has been successful in promoting greater reflexivity from middle managers; however these days the flaws are increasingly apparent.

Over my years of watching Compstat-type meetings in a number of departments, I’ve observed everyone settle into their Compstat role relatively comfortably. Well almost. The mid-level local area commander who has to field questions is often a little uneasy, but these days few careers are destroyed in Compstat. A little preparation, some confidence, and a handful of quick statistics or case details to bullshit through the tough parts will all see a shrewd commander escape unscathed.

In turn, the executives know their role. They stare intently at the map, ask about a crime hot spot or two, perhaps interrogate a little on a case just to check the commander has some specifics on hand, and then volunteer thoughts on a strategy the commander should try—just to demonstrate their experience. It’s an easy role because it doesn’t require any preparation. In turn, the area commander pledges to increase patrols in the neighborhood and everyone commits to reviewing progress next month, safe in the knowledge that little review will actually take place because by then new dots will have appeared on the map to absorb everyone’s attention. It’s a one-trick pony and everyone is comfortable with the trick.

There are some glaring problems with Compstat. The first is that the analysis is weak and often just based on a map of dots or, if the department is adventurous, crime hot spots. Unfortunately, a map of crime hot spots should be the start of an analysis, not the conclusion. It’s great for telling us what is going on, but this sort of map can’t really tell us why. We need more information and intelligence to get to why. And why is vital if we are to implement a successful crime reduction strategy.

We never get beyond this basic map because of the second problem: the frequent push to make an operational decision immediately. When command staff have to magic up a response on the spot, the result is often a superficial operational choice. Nobody wants to appear indecisive, but with crime control it can be disastrous. Too few commanders ever request more time to do more analysis, or time to consider the evidence base for their operational strategies. It’s as if asking to think more about a complex problem would be seen as weak or too ‘clever’. I concede that tackling an emerging crime spike might be valuable (though they often regress to the mean, or as Sir Francis Galton called it in 1886, regression towards mediocrity). Many Compstat issues however, revolve around chronic, long-term problems where a few days isn’t going to make much difference. We should adopt the attitude that it’s better to have a thoughtfully considered successful strategy next week than a failing one this week.

Because of the pressure to miracle a working strategy out of thin air, area commanders usually default to a limited set of standard approaches, saturation patrol with uniform resources being the one that I see at least 90 percent of the time. And it’s applied to everything, regardless of whether there is any likelihood that it will impact the problem. It is suggested by executives and embraced by local area commanders because it is how we’ve always escaped from Compstat. Few question saturation patrols, there is some evidence it works in the short term, and it’s a non-threatening traditional policing approach that everyone understands. Saturation patrol is like a favorite winter coat, except that we like to wear it all year round.

Third, in the absence of a more thoughtful and evidence-based process, too many decisions and views lack any evidential support and instead are driven by personal views. There is a scene in the movie Moneyball where all the old baseball scouts are giving their thoughts on which players the team should buy, based only on the scouts’ experience, opinion and personal judgment. They ignore the nerd in the corner who has real data and figures … and some insight. They even question if he has to be in the room. In the movie, the data analyst is disparaged, even though he doesn’t bring an opinion or intuition to the table. He brings data analysis, and the data don’t care how long you have been in the business.

Too many Compstat meetings are reminiscent of this scene. The centerpiece of many Compstat meetings is a map of crime that many are viewing for the first time. A room full of people wax lyrical on the crime problem based on their intuitive interpretation of a map of crime on the wall, and then they promote solutions for our beleaguered commander, based too often on opinion and personal judgement and too little on knowledge of the supporting evidence of the tactic’s effectiveness. Because everyone knows they have to come back in a month the strategies are inevitably short-term in nature and never evaluated. And without being evaluated, they are never discredited, so they become the go-to tactical choice ad infinitum.

So the problems with Compstat are weak analysis, rushed decision-making, and opinion-driven strategies. What might the solutions be?

The U.K.’s National Intelligence Model is a good starting point for consideration. It has a strategic and a tactical cycle. The strategic meeting attendees determine the main strategic aims and goals for the district. At a recent meeting a senior commander told me “We are usually too busy putting out fires to care about who is throwing matches around.” Any process that has some strategic direction to focus the tactical day-to-day management of a district has the capacity to keep at least one eye on the match thrower. A monthly meeting, focused on chronic district problems, can generate two or three strategic priorities.

A more regular tactical meeting is then tasked with implementing these strategic priorities. This might be a weekly meeting that can both deal with the dramas of the day as well as supervise implementation of the goals set at the strategic meeting. It is important that the tactical meeting should spend some time on the implementation of the larger strategic goals. In this way, the strategic goals are not subsumed by day-to-day dramas that often comprise the tyranny of the moment. And the tactical meeting shouldn’t set strategic goals—that is the role of the strategic working group.

I’ve previously written that Compstat has become a game of “whack-a-mole” policing with no long-term value. Dots appear, and we move the troops to the dots to try and quell the problem. Next month new dots appear somewhere else, and we do the whole thing all over again. If we don’t retain a strategic eye on long-term goals, it’s not effective policing. It’s Groundhog Day policing.

What role for experience in evidence-based policing?

I recently received an illustrative lesson in the challenges of evidence-based policing. I was asked to sit in on a meeting where a number of senior managers were pitching an idea to their commander. It required the redistribution of patrols, and they were armed with evidence that the existing beats were not in the best locations and so were not as effective as they could be. The commander sat back in his chair and said “so I have to move some of those patrols?” Yes, the area managers responded, presenting a cogent yet measured response based on a thorough data analysis supported with current academic research. The commander replied “well in my experience, they are being effective so I am not going to move them”. And at that point the meeting ended. Experience trumped data and evidence, as it often does.

All the evidence available suggested that the commander made a poor decision. When I was learning to be a pilot I heard an old flying aphorism about decisions. Good decisions come from experience, and experience comes from bad decisions[1]. Unfortunately, the profession of policing cannot lurch from an endless cycle of bad decisions as new recruits enter policing and learn the business. The modern intolerance for honest mistakes and anything-but-perfection precludes this. It’s also expensive and dangerous. Therefore how do we develop and grow a culture of good decisions? And what is the role of experience?

In praise of experience

Policing is unique in the liberal level of discretion and absence of supervision given to the least experienced officers. As a teenager when I started patrol, I can testify to the steep learning curve on entering the job. Experiences come at you thick and fast. In some ways, we learn from these experiences. Most cops absorb pretty quickly how to speak with people who are drunk or having a behavioral health crisis in a way that doesn’t end up in them rolling around in the street with fists flying. My colleagues demonstrated a style and tone and I learned from the experience they had gained over time.

We should also recognize that the evidentiary foundation for much of policing is pretty thin. We simply do not yet know much about what works and what good practice looks like. It’s not as if we have an extensive knowledge bank with which to replace experience. In recognition of this, the UK College of Policing note that “Where there is little or no formal research, other evidence such as professional consensus and peer review, may be regarded as the ‘best available’”. So practitioner judgement may help fill a void until that time when we have more research across a wider variety of policing topics. In time, this research will help officers achieve better practice. In the meantime, shared experience may be of value, if (in the words of the UK College of Policing) “gathered and documented in a careful and transparent way”.

Finally, personal intuition and opinion may not be a sound basis on which to make policy, but sometimes it can offer insights in rarely studied areas. This can prompt new ways of looking at problems. By varying experience, we can learn new ways to deal with issues. These new ways could then be tested more formally. There is definitely a place for personal judgement in the craft of policing. But the current reliance on it prevents us embracing a culture of curiosity and developing that evidence base[2]. And personal experience has other limitations.

A critique of experience

Like the commander at the start of this section, unfortunately, most police leaders don’t make decisions using the best evidence available. They overwhelmingly prefer decisions that are entrenched in their personal experience. The problem is that everyone’s experience is limited (we can’t have been everywhere and dealt with every type of incident), and in policing we receive too little feedback to actually learn many lessons.

What do I mean? As a young cop, I attended countless domestic disturbance calls armed with so little personal experience in long-term relationships it was laughable. It soon because clear that the measure of ‘success’ (against which ‘experience’ was judged) was if we got a call back to that address during that shift. If we did, I had failed. If we didn’t, I had succeeded and was on my way to gaining the moniker of ‘experienced’.

But what if the husband beat his partner to within an inch of her life within hours of my going home? Or the next week? If our shift wasn’t on duty I would never learn that my mediation and resolution attempts had been unsuccessful or worse, harmful. I would never receive important feedback and would continue to deal with domestic disturbance calls in the same way. Absent supervision and feedback, not only would I continue to act in a harmful manner, worse, my colleagues and I might think I was now experienced. They might prioritize my attendance at these calls, and perhaps eventually give me a field training role. My bad practice would now become established ‘good’ practice.

As others have noted “personal judgment alone is not a very reliable source of evidence because it is highly susceptible to systematic errors – cognitive and information-processing limits make us prone to biases that have negative effects on the quality of the decisions we make”. Even experienced police officers are not great at identifying crime hot spots[3] and do not do as well as a computer algorithm[4]. This isn’t just an issue for policing. Experts, many with many years of experience, are often poor at making forecast across a range of businesses and professions. The doctors that continued to engage in blood-letting into the latter half of the 19th century weren’t being callous. They probably had good intentions. But their well-meaning embrace of personal judgement, tradition and supposed best practice (probably learned from a medical guru) killed people.

I frequently conduct training on evidence-based and intelligence-led policing. I often run a quick test. I show officers a range of crime prevention interventions and ask which are effective. It’s rare to find anyone who can get the correct answer, and most folk are wildly off target. It’s just for fun, but illustrates how training and education in policing still remains at odds with a core activity, the reduction in crime.

A role for professional experience?

As Barends and colleagues note[5], “Different from intuition, opinion or belief, professional experience is accumulated over time through reflection on the outcomes of similar actions taken in similar situations.” It differs from personal experience because professional experience aggregates the knowledge of a variety of practitioners. It also emerges from explicit reflection on the outcomes of actions.

This explicit reflection requires feedback. When I was learning to fly, a jarring sensation and the sound of the instructor wince was the immediate feedback I needed to tell me I had not landed as smoothly as I had hoped. But flying around the traffic pattern, I immediately had another chance to prove improvement and a lesson learned. This type of immediate feedback and opportunity to improve is rare in policing. The radio has already dragged us to a different call.

For many enforcement applications, a research evaluation is essential to provide the kind of feedback that you can’t get from personal observation. The research on directed patrol for gun violence is a good example of how research evidence can improve strategy and increase public safety. Science and evaluation can replicate the experiences of hundreds of practitioners and pool that wisdom. While you can walk a single foot beat and think foot patrol is waste of time, the aggregate experiences and data from 240 officers across 60 beats tells us differently.

Tapping into scientific research findings and available organizational data (such as crime hot spot maps) and temporal charts, will enhance our professional experience. Being open to the possibility that our intuition and personal opinion may be flawed is also important, though difficult to accept. And developing a culture of curiosity that embraces trying new ways of tackling crime and disorder problems might be the most important of all. The starting point is to recognize that if personal experience remains the default decision-making tool, then we inhibit the development of better evidence. And we should realize that approach is harmful to communities and colleagues alike.


[1] This quote (sometimes replacing decisions with judgement) is attributed to various sources, but the most common is Mark Twain. It should also be pointed out that my fondness for checklists stems from one of the aviation industry’s attempts to reduce the poor decision-making learning spiral.

[2] I’m grateful to a smarter friend for pointing this out to me.

[3] Ratcliffe, J.H. and M.J. McCullagh, Chasing ghosts? Police perception of high crime areas. British Journal of Criminology, 2001. 41(2): p. 330-341.

[4] Weinborn, C., et al., Hotspots vs. harmspots: Shifting the focus from counts to harm in the criminology of place. Applied Geography, 2017. Online first. 

[5] Barends, E., D.M. Rousseau, and R.B. Briner, Evidence-Based Management: The Basic Principles. 2014, Amsterdam: Center for Evidence-Based Management.

Policing explained in a few graphs

As a new semester starts and I look forward to teaching a new group of undergraduates, I am reminded that some of them have not had extensive exposure to law enforcement or policing research. How then to encapsulate the essence of what is useful to know about policing in a brief enough format? These are some of the graphs I use to help my new students understand the challenges of policing in the 21st century.

1. Policing is overwhelmingly a social service

Graph no. 1. This is from the second edition of my book “Intelligence-Led Policing“. The area of each box represents the volume of incidents in 2015 in the City of Philadelphia (about 1.5m in total). These incidents can come from verified calls for service from the public (something really took place as confirmed by a police officer), or from officer-initiated events (such as drug incidents). 

What is clear from the graphic is that violent crime plays such a small part in the day-to-day demands on police departments, even in Philadelphia, one of the more troubled cities in the U.S. While the media frets over homicide, it can be seen in the lower right as one of the least noticeable boxes in the graph. The majority of the police department’s workload is the day-to-day minutiae of life in a big city.

2. Impacting early in the crime funnel is the key to public safety

Graph no. 2 is another image from my Intelligence-Led Policing book. The crime funnel represents what happens to a random selection of 1,000 crimes that affect the public (top bar). It shows the loss of cases through the criminal justice system. These are British national data derived from public records, but the comparisons to the U.S. are very similar. If you take a random selection of 1,000 crimes actually suffered by the public (violence, robbery, vehicle theft, residential burglary, theft and criminal damage) you can see that they only report 530 to the police, who in turn record just 43 percent of the original total.

Of these 429 events, 99 are detected (solved or cleared in some way) and of these, 60 end up with a day in court. The majority of those are found guilty or plead the same, but in the end only four of the events from the original 1,000 end up in a custodial sentence for the offender. This is an incarceration rate of 0.4% based on crime suffered by the community.

The main point here is that impacting higher in the crime funnel will be more effective because it affects the numbers below and affects a larger number of actual cases. Improving the detection rate will have an impact on prosecutions, pleas and incarceration, but only to a minimal level. Being prevention focused and changing the higher numbers is much more impactful. Consider if you could have a 10% change on one level. Where would it be most effective?

3: Public perception of the police has little to do with crime

Graph no. 3. Going back nearly 20 years, it is clear from the graph that violent crime in the U.S. has continued a decline right through to at least 2013. Things have changed in the last year or two but that isn’t the point here. Public perception in terms of confidence in the police does not seem to be tied to the crime rate. Confidence in the police is pretty high, especially compared to nearly every other occupation (except the medical field). After all in June this year, it was found that only 27% of Americans say they have “a great deal” or “quite a lot” of confidence in newspapers, so police are doing pretty well. The graph shows however that there isn’t a direct and easy correlation between the crime rate and confidence in the police. It is more complicated. Public confidence is more complicated than just crime reduction. This graph is from a forthcoming book I am in the middle of writing called “Fighting Crime” and it will be a crime reduction guide for mid-level police command staff.

4: Policing is changing drastically

Graph no. 4 isn’t even about policing, but it is about external policy impacts on policing. When I joined the police service in 1984 and started policing on the old H district in East London, we had to deal with a few behavioral health patients because we had St. Clement’s hospital mental health hospital a stone’s throw from the nick. So perhaps we were immune to the gradual change in the social structure of society, a change that has affected the U.S. as well. Graph no. 4 is from the U.S. Substance Abuse and Mental Health Services Administration.

30 years ago, nearly two-thirds of U.S. mental health spending was on inpatient and residential care: in other words, professionally trained carers. Over the last 30 years this has reduced to only be about one-third of national spending. As graph number 4 shows on the right, the corresponding increase has not been in outpatient treatment, but in retail prescription drugs. This has essentially shifted the supervision of people having a behavioral health crisis onto the community – and the police.

It is indicative of just one change in society that has made demands on the police. Police officers are not really trained to be mental health professionals, and yet society has demanded they take a role in the care and control of people with behavioral health crises because society seems unwilling to pay for appropriate professional care. Behavioral health is just one factor, and students are often able to consider many other areas where the police are now the lead agency in areas that were never originally police issues.

Source: This is from SAMHSA data and screen captured from a PowerPoint I use in when consulting and training police departments on intelligence-led policing.

5. Policing is increasingly a safer occupation

This is one graphic that I confess I didn’t create myself, but it tracks with my research on this area and mimics many similar charts online. It is sourced from the Officer Down memorial page. While things have changed in the last couple of years, the good news is that policing has become increasingly safer (as has society) in the U.S. over the last 30 years.

What I find interesting is that increases in police fatalities are linked to government policy changes that alter the role of police in society. The overarching point remains that policing has become a much safer occupation over the last 30 years or more. It now does not feature in the top 10 list of dangerous occupations (This next graph from WaPo isn’t counted in my list – it’s a freebie).

6: Policing is most effective when focused on specific people and places

OK, this one takes a bit to get your head around. Graph no. 6 is the Evidence-Based Policing Matrix from Cynthia Lum and colleagues has three dimensions and four symbols. The key symbol is the black circle – this indicates an effective intervention (white is ineffective, grey is a mixed result, and the red triangle is a harmful intervention). Where the black circles are most concentrated are in the areas where interventions focus proactive activity on places that are at the neighborhood level or smaller, or on people. This (and other evidence) suggests that general interventions that are applied across whole jurisdictions or cities are less effective.

The bottom line? Police interventions that actually reduce crime are locally focused and tailored to specific problems, not citywide interventions.

Are there more graphics that depict the reality of 21st Century policing? Of course. But these ones seem to be significant. Ping me if you have others you think are relevant and important.

Year-to-date comparisons and why we should stop doing them

Year-to-date comparisons are common in both policing and the media. They involve comparing the cumulative crime count for the current year up to a certain date and comparing to the same point in the preceding year. For a Philadelphia example from April of this year, NBC reported that homicides were up 20 percent in 2017 compared to 2016. You can also find these types of comparison in the Compstat meetings of many police departments.

To gauge how reliable these mid-year estimates of doom-and-gloom are, I downloaded nine years (2007-2015) of monthly homicide counts from the Philadelphia Police Department. These are all open data available here. I calculated the overall year change as well as the cumulative change monthly from year to year. In the table below you can see a row of annual totals in grey near the bottom, below which is the target prediction as a percentage of the previous year (white text, blue background). For example, the 332 homicides in 2008 were 14.7% lower than the previous year, expressed in 2007 terms.

Let’s determine that we can tolerate our prediction to be within 5 percent plus or minus the eventual difference between this year and the preceding year. That stipulates a fairly generous 10% range as indicated by the Low and High rows in blue.

Each month you can see the percentage difference between the indicated year-to-date at the end of the month, and the calendar year-to-date (YTD) for the same period in the previous year. So for example, at the end of January 2008 we had 21.9% fewer homicides than at the end of January 2007. By the time we get to December, we obviously have all the homicides for the year, so the December percentage change exactly matches the target percentage difference.

Cells highlighted with a green background have a difference on the previous year that is within our +/- 5 percent tolerance. By the end of each January, we only had one year (2012) with a percentage difference that was within 5 percent of how the city ended the year. The 57% increase in January 2011 was considerably different that the eventual 6% increase over 2010 at the end of December. When Philadelphia Magazine dramatically posted “Philly’s Murder Rate is Skyrocketing Again in 2014” on January 14th of that year, the month did indeed end up nearly 37 percent over 2013. But by year’s end, the city had recorded just one homicide more than the preceding year – a less dramatic increase of 0.4%.

In fact, if we seek out a month where the difference is within our 10% range and later months will remain consistently accurate through to the end of the year, then we have to wait until the months shown with a border. 2009 performed well, however while 2010 was fairly accurate throughout the summer, the cumulative totals in September and October were more than 5% higher than the previous year when the year ended only 0.3% higher.

To use calendar YTD comparisons with any confidence, we have to wait until the end of October before we can be more than 50% confident that the year-to-date is indicative of how we will enter the New Year. And even then we still have to be cautious. There was a chance at the end of November 2010 that we would end the year with fewer homicides, though the eventual count crept into increase territory.

The bottom line is that with crimes such as homicide, we need not necessarily worry about crime panics at the beginning of the year. This isn’t to say we should ever get complacent and of course every homicide is one too many; however the likely trend will only become clear by the autumn.

Alternatives exist. Moving averages seem to work okay, but another alternative I like is to compare full (annual) YTDs to the prior annual (i.e. full 12 month) YTD. So instead of (for example) comparing January-April 2010 to January to April 2009, you could compare the 12-month change May 2009-April 2010 against the May 2008-April 2009 total. I’ve done that in the red graph below. The first available point is December 2008 and as we know from the previous table, the preceding 12 months had outperformed the annual year 2007 by 14.7%. But then each subsequent month measures not just the calendar YTD but the 12-month YTD.

The result is a graph that shows the trend changing over time from negative (good) territory to positive (bad for homicides because it show an increase). Not only do you get a more realistic comparison that is useful throughout the year, you can see changing trend. Anything below the horizontal axis is good news – you are doing well. Above it means that your recent 12 months (measured at any point) was worse than the preceding 12 months.

You can have overlapping comparison periods. The graph in blue below compares 24 months of accumulated counts with the 24 month totals for the previous year. For example, the first point available is December 2009. This -11.7% value represents the change in total homicides from the 24 months January 2008 to December 2009 and compares it to the 24 month total from a year previous to this (January 2007 to December 2008). For comparison purposes, I have retained the same vertical scale but note the change in horizontal axis.

You can see there is more smoothing, but the general trend over time is still visible. Lots of variations available and you might want to play with different options for your crime type and crime volume.

Classic experiments in evidence-based policing

So far, it’s been a fun semester teaching evidence-based policing for the first time. We have covered everything from evidence-based medicine to research design and the Maryland Scientific Methods Scale, and even some basic stats so that we can understand confidence intervals. It’s been particularly rewarding to see students who have spent years in policing exploring and learning about the world of research evidence that supports and helps their world, a world of which many have been until now unaware.

What I am also learning is that those of us in the police education field have done a lousy job of explaining what we do and why it is important to advancing policing and the practice of law enforcement. There is a range of classic studies that are not well known, and an absence of knowledge around these – and other important works – fuels the never-ending cycle of operational decisions that fly in the face of all we know about what works, and what doesn’t. Police still support strategies and crime reduction tactics that are known to not work.

In light of this, I started putting together a list of experiments of which that I thought my students should be aware. The original studies are described in a range of works from academic journal articles to long-winded reports. All pretty impenetrable for most folk, especially busy cops. So I have copied and pasted the key pieces of information into a single page per study, copied from the original sources directly. I cite them at the bottom of each page so you know the source.

This isn’t an exhaustive list, and I intend for it to grow, but for now the list comprises:

  1. The Kansas City Preventive Patrol Experiment
  2. The Newark Foot Patrol Experiment
  3. The Philadelphia Foot Patrol Experiment
  4. The Minneapolis Domestic Violence Experiment
  5. The Minneapolis Hot Spots Policing Experiment
  6. The Philadelphia Policing Tactics Experiment
  7. The Sacramento Hot Spots Policing Experiment
  8. The Queensland Procedural Justice Experiment

I will add to these over time, but for now if you want a copy, download a pdf of the one page summaries.

Note: If you are using these summaries to write a college paper, you should refer to the original study and cite it appropriately. All I have done is edit a copy-and-paste, but I’m 1) not writing a term paper and 2) not passing this off as my own work. If you do, that’s plagiarism. 

The Modifiable Areal Unit Problem

The Modifiable Areal Unit Problem (MAUP) is a potential source of error that can affect spatial studies which utilize aggregate data sources (Unwin, 1996). Geographical data are often aggregated in order to present the results of a study in a more useful context, and spatial objects such as census tracts or police beat boundaries are examples of the type of aggregating zones used to show results of some spatial phenomena. These zones are often arbitrary in nature and different areal units can be just as meaningful in displaying the same base level data. For example, it could be argued that census tracts containing comparable numbers of houses are better sources of aggregation than police beats (which are often based on ancient parish boundaries in the UK) when displaying burglary rates.Preview

Large amounts of source data require a careful choice of aggregating zones to display the spatial variation of the data in a comprehensible manner. It is this variation in acceptable areal solution that generates the term ‘modifiable’. Only recently (well, the last 30 years) has this problem been addressed in the area of spatial crime analysis, where ‘the areal units (zonal objects) used in many geographical studies are arbitrary, modifiable, and subject to the whims and fancies of whoever is doing, or did, the aggregating.’ (Openshaw, 1984 p.3).

As the study area for crime incident locations has effectively infinite resolution, there exists a potentially infinite number of different options for aggregating the data. Numerous administrative boundaries exists, such as enumeration districts, wards, counties, health authority areas, etc. Within modern GIS, it is an elementary task to automatically generate a huge variety of non-overlapping boundaries. Regular, often square, grids are common, though polygons have been used in other studies of crime distribution (Hirschfield et al., 1997). The number of different combinations of areal unit available to aggregate data is staggering. Openshaw (1984) calculated that if one was to attempt to aggregate 1,000 objects into 20 groups, you would be faced with 101,260 different solution combinations. Although there are a large number of different spatial objects and ways in which a large geographical area can be sub-divided, the choices of areal units tend to be dominated by what is available rather than what is best. Police crime data is often mapped to police beats, even when the information is passed to outside agencies such as neighborhood watches or local councils who might benefit from more relevant boundary structures.

The MAUP consists of both a scale and an aggregation problem, and the concept of the ecological fallacy should also be considered (Bailey and Gatrell, 1995). The scale problem is relatively well known. It is the variation which can occur when data from one scale of areal units is aggregated into more or less areal units. For example, much of the variation in census areas changes or is lost when the data are aggregated to the ward or county level.

The aggregation problem is less well known and becomes apparent when faced with the variety of different possible areal units for aggregation. Although geographical studies tend towards aggregating units which have a geographical boundary, it is possible to aggregate spatial units which are spatially distinct. Aggregating neighbors improves the problem to a small degree but does not get round the quantity of variation in possibilities which remains.

For a paper that discusses the MAUP and possible solutions, see:
Ratcliffe, J. H. and McCullagh, M. J. 1999 ‘Hotbeds of crime and the search for spatial accuracy’, Geographical Systems 1(4): 385-398. Paper available here.

Also see the Ecological Fallacy.

References:

Bailey, T. C. and Gatrell, A. C. 1995 Interactive Spatial Data Analysis, Second Edition: Longman.

Hirschfield, A., Yarwood, D. and Bowers, K. 1997 ‘Crime Pattern Analysis, Spatial Targeting and GIS: The development of new approaches for use in evaluating Community Safety initiatives.’, in N. Evans-Mudie (ed) Crime and health data analysis using GIS, Sheffield: SCGISA.

Openshaw, S. 1984 ‘The modifiable areal unit problem’, Concepts and Techniques in Modern Geography 38: 41.

Unwin, D. J. 1996 GIS, spatial analysis and spatial statistics’, Progress in Human Geography 20(4): 540-441.

The ecological fallacy

The ecological fallacy is a situation that can occur when a researcher or analyst makes an inference about an individual based on aggregate data for a group. For example, a researcher might examine the aggregate data on income for a neighborhood of a city, and discover that the average household income for the residents of that area is $30,000.

To state that the average income for residents of that area is $30,000 is true and accurate. No problem there. The ecological fallacy can occur when the researcher then states, based on this data, that people living in the area earn about $30,000. This may not be true at all, and may be an ecological fallacy.

ecological fallacyCloser examination of the neighborhood might discover that the community is actually composed of two housing estates, one of a lower socio-economic group of residents, and one of a higher socio-economic group. In the poorer part of town, residents earn on average $10,000 while the more affluent citizens can average $50,000. When the researcher stated that individuals who live in the area earn $30,000 (the mean rate) this did not account for the fact that the average in this example is constructed of two disparate groups, and it is likely that not one person earns around $30,000.

Assumptions made about individuals based on aggregate data are vulnerable to the ecological fallacy.

This does not mean that identifying associations between aggregate figures is necessarily defective, and it doesn’t necessarily mean that any inferences drawn about associations between the characteristics of an aggregate population and the characteristics of sub-units within the population are absolutely wrong either. What it does say is that the process of aggregating or disaggregating data may conceal the variations that are not visible at the larger aggregate level, and researchers, analysts and crime mappers should be careful.

 

Harm-focused policing

On 28th January, 2015 I gave the Police Foundation‘s Ideas in American Policing lecture on the topic of harm-focused policing. This brief blog provides some background details to the talk. Please note that for a number of reasons (including photograph copyright) I am not distributing copies of the PowerPoint slides.

Harm-focused policing weighs the social harms of criminality and disorder with data from beyond crime and antisocial behavior, in order to focus police priorities and resources in furtherance of both crime and harm reduction.

Example information and data sources could include drug overdose information that could help triage drug markets for interdiction, traffic fatality data to guide police patrol responses, and community impact assessments to prioritize violent street gangs. For a summary of the core of the presentation and a grey-scale version of some of the graphics, please see:

Ratcliffe, J. H. (2015). Towards an index for harm-focused policing. Policing: A Journal of Policy and Practice, 9(2), 164-182.

You can visit the journal site and access the paper here (or here) and watch my annotated video of the lecture below (you might want to make it full screen so you can read the slides).

During the presentation, I had a couple of quotes. Here are the quotes and their sources.

“to establish priorities for strategic criminal intelligence gathering and subsequent analysis based on notions of the social harm caused by different sorts of criminal activity”. The source for this is page 262 of Ratcliffe, J. H. & Sheptycki, J. (2009) Setting the strategic agenda. In J. H. Ratcliffe (ed.) Strategic Thinking in Criminal Intelligence (2nd edition) Sydney: Federation Press.

“Weighting crimes on the basis of sentencing guidelines can be justified on good democratic grounds as reflecting the will of the people. …… it remains far closer to the will of the people than any theoretical or even empirical system of weighting that academics might develop.” The source for this is Sherman, L. W. (2013). Targeting, testing and tracking police services: The rise of evidence-based policing, 1975-2025. In M. Tonry (Ed.), Crime and Justice in America, 1975-2025. Chicago: University of Chicago Press. Page 47.