Benchmarking Performance Data

Come senators, congressmen, please heed the call

Don’t stand in the doorway, don’t block up the hall...

It’ll soon shake your windows and rattle your walls

For the times they are a-changin’.

—Bob Dylan, 1964

Fifty years later, and the times they are still a-changin’. Back in the 1960s, the battle being fought was about freedom, individuality, and expression. Today’s battle is about openness, transparency, and engagement. Citizens are demanding more from their public servants than ever before—and at all levels of government. Not only do we expect our governments to provide ever-increasing levels of service to us at ever-decreasing costs, but we also want to be part of that process. The cry to rid government of waste, fraud, and abuse has been encouraged by sound-bite politics. We need to understand how our government agencies are performing, where they could be doing better, and how they can improve. We want to help influence that conversation and know that action-oriented decisions are being executed. We expect to see measurable results and want to feel the effects of those improvements in our everyday lives. We’re like the shareholders of a public corporation that want to hold our executive leadership accountable for achieving our corporate goals.

We are experiencing a radical transformation in the volume of public data available today and the velocity and means at which it is delivered to citizens. For example, the International Open Government Dataset Search web page (Tetherless World Constellation International, 2013) lists 1,028,054 datasets in forty-three countries, containing 192 catalogs and 2,460 categories of datasets. How are local government leaders responding to this phenomenon as it relates to measuring and improving operational performance? What benefits do they hope to achieve from this transparency? What does the future of local government leadership look like in terms of managing better and driving improved performance? And, how does collaboration of performance results and sharing of best practices across cities support that future vision?

This essay explores how transparency and collaboration in the municipal performance management process leads to increased engagement, better-run government, and improved outcomes for citizens. First, we’ll provide some background on what performance management in the public sector is and why we do it. Second, we’ll discuss accountability and transparency to the public and share insights from real cities doing this today. Finally, we’ll talk about how open data can be leveraged across and among cities to provide context and insight that inform the performance improvement agenda. We’ll learn how, despite our preconceived notions, cities large and small are openly collaborating to help each other improve.

Let’s start by getting a baseline understanding of what performance management is and why we do it.

What Is Performance Management?

Performance management, specifically as it relates to the public sector, is a management system to improve the delivery of services using quantified measurements that are collected and reported to focus attention and action on areas of organizational performance in need of improvement.

Performance management is, first of all, a management system. While this statement may appear obvious, another commonly used term, “performance measurement,” obscures the focus of performance management. Performance management is action-oriented, although quantitative measurement to monitor the effectiveness of the delivery of public services remains a core component. It is the periodic and routine management review of these ongoing measures that leads to new initiatives, reallocation of resources, modification of service processes and policies, and anything else that improves the service delivery of our government.

A key first step to a successful performance management process is to make sure we are measuring the right things—the issues that matter to citizens. We call these “outcomes.” Outcomes typically come in two forms—outcomes of effectiveness (doing things better) and outcomes of efficiency (being more cost-effective at doing so). One advantage of measuring outcomes is that, unlike department operations or business processes that may change over time, the outcomes that citizens desire do not. This allows us to define a long-term vision for improvement and develop a roadmap that gets us there.

There is a growing emphasis on asking citizens what is important to them, and in doing so, we often learn that our initial assumptions are off. San Francisco learned from citizen surveys that perceptions of street and sidewalk cleanliness were affected by foul odors. Bellevue, Washington, developed a set of sixteen “vital signs,” then used focus groups to validate those vital signs. Bellevue’s departments thought that the things they do most frequently were most important, but the public felt that the most severe outcomes were most important. For example, the police department focused on less severe crimes that occur more often, but the public apparently cared more about the most serious, though less frequent, crimes.

Measuring outcomes of effectiveness and efficiency is much more meaningful than simply measuring activity. Counting how many potholes we filled last month doesn’t tell us very much. It tells us how busy we are, but not whether we are particularly good at pothole repair. How long that three-foot pothole has been sitting in the middle of her street is something that might matter to Jane Q. Citizen. Two days? Two weeks? Two months? Longer? A measure of effectiveness that matters to her might be the average amount of time that elapses between the reporting of a pothole by a citizen and its repair. We’ll also want to know the average cost to repair a pothole so that we can determine whether or not we have an efficient pothole repair operation.

Once we decide what we’ll need to measure, we need to begin capturing, reviewing, and analyzing the data. Naturally, we’ll need the right technology to do so. We need a tool that can interoperate with the myriad of other systems within our operations. Furthermore, we need the ability to visualize the data in ways that tell a clear story. This tool needs to help us see patterns and relationships, correlate data across departments and agencies, and be intelligent enough to present us with the most important and relevant information, so that even our ongoing performance management process is as efficient as it can be. It should alert us when performance is moving in a negative direction so we can respond accordingly, and it should highlight positive trends so that we know that our actions are keeping us on the right course.

Why Bother?

Having an effective, ongoing performance management process in place is the key to running a successful government. Through these processes, management gains increased visibility into operational performance and results. Operational improvements can be realized through optimized resource allocations. It can even spur policy review and modifications.

Even more important is the fact that introducing these approaches as part of the way of doing business inside local government can establish a high-performance culture within the organization. This creates greater internal alignment by better communicating organizational priorities to the team.

By introducing open data into performance management processes, we can make great leaps forward in increasing accountability and data-driven communication with stakeholders. That means knowing the stakeholders of local government and monitoring how a good performance management process can properly engage them through the data.

Accountability and Communication

Government accountability is about setting expectations and reporting to stakeholders what has been accomplished relative to those expectations. It is more about the communication of actions and results, rather than the actions and results themselves. Accountability rests on an assumption of responsibility by elected and appointed officials in government to protect and serve citizens and act as stewards of the public’s resources.

The four major stakeholders of local government are:

• Department heads, who oversee the operations within each service area.
• Municipal management, who coordinate the different service areas and are responsible for the implementation of policies.
• Elected officials, who set policy, may have a management role, and are one step from the public.
• The public, either through individuals or stakeholder groups.

The benefit of a good performance management system is that objective data about the accomplishments of government can be seen by anyone, even if it is packaged in different ways and in different levels of detail. This integrated accountability promotes rational decision-making because of the commonality of the facts. Staff, management, and the public see these facts through their individual lenses:

• Staff: are we getting the job done?
• Management: is the job getting done efficiently and effectively?
• Public: are we getting the results we pay for?

To that last point, we are seeing two interesting trends emerge that support open data with respect to operational performance.

Public Participation

First, citizens, in part thanks to technological developments, are taking a more active role in wanting to understand and contribute to the performance of their cities and towns. Products like SeeClickFix, which capture input directly from citizens through their mobile devices, are informing city agencies of service needs in real time and with greater volume than ever before. They allow citizens to report everything, from graffiti to potholes to streetlights in need of repair. However, these new capabilities are also setting expectations in the minds of citizens that these incidents will be addressed on a timely basis. They expect results and feedback. City managers need to track overall performance and response times and report how well they’re doing back to the public, both individually and in the aggregate.

The use of citizens or non-government organizations to help deliver public services is known as “co-production.” Co-production can be provided by individuals or by organized volunteer groups. Originally, it was represented by low technology involvement, such as the Neighborhood Watch, an organized group of volunteers trained to reduce crime and vandalism in their neighborhoods. Technology is increasing the potential for citizens to produce services in conjunction with their local governments.

High Performance Cultures

At the same time as it is becoming easier for citizens to identify problems for their governments, forward-thinking government officials, some of whom have been influenced by private sector experience, are realizing the benefits of holding themselves and their organizations accountable to the public. A mayor or city manager instills a high performance culture within his organization when he establishes an ongoing process of tracking operational metrics across departments and agencies—and publishes those metrics to the public in conjunction with the objectives and strategies they are intended to measure. By giving each of his department heads direct responsibility for establishing those objectives, strategies, and metrics, he is driving accountability and strengthening the relationship between citizens and their government leaders.

How Cities Are Communicating Performance Results With Their Public

To date, only a small percentage of cities have gone so far as to publish their performance strategies and results publicly on their websites. These pioneers realize the value of continuously monitoring performance, striving to improve results, and sharing those results with their citizen stakeholders. Although the processes used to collect, review, and publish performance data and the content and format of the information that is presented vary widely across cities, they all have certain common characteristics:

• These initiatives are driven from the very top of the organization, often by the mayor or city manager.
• Performance data is collected, reviewed, and published on a routine basis.
• Department managers are integral to the process.
• There is a specific part of the organization dedicated to the performance management program. In smaller cities, this might be all or part of the responsibilities of a single individual. In larger cities, there is often a department or office of performance management that involves a larger team.

In reviewing websites of many of the cities that engage in performance reporting, we found a great deal of diversity in the manner in which these municipalities went about their reporting:

• The population sizes range from towns that may have less than 50,000 residents to the largest of cities throughout the nation, including New York City.
• The frequency of the data reporting varies, including monthly, bi-monthly, quarterly, semi-annually, and annually.
• Some report performance against targets and discuss the goals and objectives of their departments, while others do not.
• The most commonly used source of data is operational measures, which are collected by the departments, but some of the cities use citizen satisfaction surveys and outcomes based on inspections or other ways of determining results of the operations.
• Several of the cities have found ways to have the public determine what measures are important to them, including focus groups, citizen surveys, and individual suggestions.
• There are no apparent correlations between these diverse factors, such as a tendency for the larger municipalities to report more (or less) frequently.

We spoke to officials who manage performance programs from four cities that are widely known for their performance initiatives. The following are highlights of their programs:

San Francisco, California

San Francisco’s performance program is called “San Francisco Performs.” It is primarily focused on supporting the mayor’s proposed budget and helping to provide context to citizens. Annually, San Francisco issues the “Controller’s Annual Performance Report,” a comprehensive report with over one thousand performance metrics across the city’s forty-eight departments (e.g. Airport, Fire, and Human Services). That amounts to approximately twenty metrics per department. San Francisco also issues a smaller, bi-monthly “Performance Barometer” report on a subset of key measures. This report is approximately seven pages in length and includes a rotating “highlighted measure” and a few of the key performance indicators for various activities, such as Public Safety, Streets and Public Works, and Customer Service.

Kyle Burns, Performance Analyst & Program Lead in the San Francisco Controller’s Office stated in an interview:

The benefits of reporting performance data to the citizens are all about accountability. Why? Because it’s important. The mayor’s office uses the performance metrics to make decisions on resource allocation in the budget process… Transparency and the idea of having the data published allow citizens to understand how the city is performing and delivering services. (Burns, personal communication, 2013.)

Bellevue, Washington

The City of Bellevue, Washington, has been, “Managing for Results” since 1997, when the city manager started a performance management program based on two key goals: creating an evidenced-based government and sharing that information with the public via their website. The title they selected for their program, Managing for Results, indicates their orientation to outcomes, or results that matter to citizens. They describe their philosophy with the question “So what?” This indicates their interest in determining why a measure ultimately matters to citizens.

A cornerstone of Bellevue’s program is to investigate and utilize diverse methods that reach and engage citizens. In addition to using their website for their performance program, in City Hall, there is a board posted which displays the city’s Vital Signs—a set of sixteen key metrics. Bellevue produces an annual performance report following reporting guidelines from the Association of Government Accountants‘ (AGA) and shares this with citizens on their website. They also conduct and report on the results of citizen surveys about satisfaction with public services to complement the operational metrics they collect. Their use of community indicators, which are measures that get close to the ultimate concerns of citizens but may not be totally under the control of any single department or even the city as a whole, is evidence of their sincere belief in communicating to citizens. According to Rich Siegel, Performance & Outreach Coordinator, “We need to let citizens know if we are doing better, the same, or worse. We are likely to get support for projects when they know how well we are doing.”

Austin, Texas

Austin, Texas started “Managing for Results” in 1992. “The core focus of the performance program is to focus on the customer,” said Shannon Szymczak, Corporate Budget Manager. Szymczak continued, “It’s an old saying, but you have to measure what matters.” With over twenty years of experience measuring performance, its performance system has evolved. Austin started with over four thousand metrics, but is now down to one thousand. In 2011, they began to report on twenty-one dashboard measures that were chosen by focus groups of citizens.

Much of the focus of the performance program is aimed at the budget process. Each department is responsible for developing departmental goals to inform the budget process. Since 2005, everything reported to the budget office is made available publicly. Austin emphasizes results by distinguishing performance measures from operational measures, which assess activities. A forward-thinking approach is demonstrated in displaying the results for the past three years in the context of targets for the current and upcoming year. According to Szymczak, “goals must be measurable.”

Baltimore, Maryland

Baltimore, Maryland, is appropriately credited as a pioneer of the CitiStat model. CitiStat and all the management methods known as the Stat models share an emphasis on relentless follow-through. Officials accomplish this through periodic meetings in which they review performance and determine action plans to resolve issues. These meetings occur about once per month—or even more frequently in some implementations. The updated results are reviewed in order to evaluate actions taken based on the decisions reached in prior meetings.

The Baltimore Office of CitiStat is a small performance-based management group responsible for continually improving the quality of services provided to the citizens of the city. Staff analysts examine data and perform investigations in order to identify areas in need of improvement. The mayor or members of her cabinet attend the CitiStat meetings, which are held every four weeks, and ask the presenting agency questions about its performance. As a result of its success, local governments have adapted the CitiStat model across the US and around the world.

Through the CitiStat program, Baltimore had been tracking metrics internally for many years prior to sharing them with citizens. Initially, department managers had concerns that it might be hard for citizens to digest the information or that it would be taken out of context. Although the CitiStat process emphasized internal management and improvement in delivering services to citizens, the data was no longer being released publicly. The mayor signed an executive order in August 2012 to promote increased transparency. Chad Kenney, Director of Baltimore CitiStat says, “The key is to not overwhelm citizens with a lot of data and to put the data into context so it’s understood.” Department managers understand the detailed data because they understand the service processes. Citizens do not have the advantage of this context.

Baltimore’s public-facing reports include month-to-month and year-to-year comparisons in order to provide a baseline for citizens to evaluate performance. Some of their current initiatives include making this data more understandable by providing neighborhood-based information and working with local groups who help citizens understand the data.

Lessons Learned

What takeaway points did these four cities, as a group, suggest? How have they addressed the challenges of publicly reporting data?

Address Department Concerns of Misinterpretation

It is common for department managers to be concerned that the public will be critical and misinterpret data because they do not have knowledge of the operations of the city. In addition to providing the public with more summarized information about results and outcomes in order to reduce the misinterpretation, other responses have included:

• Emphasize performance exceeding expectations, as well as performance not meeting expectations.
• Ignore individual data points and emphasize stable trends.
• Emphasize results, rather than activities, to allow the same metrics to be reported, even when business processes change.
• Partner with non-government organizations to present the data to the public to increase their understanding.

The balancing act is ongoing: transparency to the public and other stakeholders versus the validity of the interpretation of the data. Perhaps even more important is the perception by department managers that others (including their own elected and appointed officials) do not understand their service area. Hence, some of the responses above show support for the managers’ concerns.

Our company, Revelstone, has experienced these concerns with some of its customers. Our practical experience is that leadership needs to support the importance of the data and responsibility of the service area to make its value proposition to all stakeholders. We also found evidence in the four cities we talked to that appropriate data for stakeholders will be different than that used for an internal review of operations in the service area. Baltimore, which built its CitiStat program around a model to promote frequent operational review, has the shortest program of reporting to external stakeholders. More time is needed to see if the partnership with non-governmental organizations will help public use and interpretation of the data or whether it will still need further summarization and a results orientation in the public-facing data reporting.

Public Use Is Not Always Extensive

Most of the cities felt that the use of the data by the public was not extensive. Almost all of them recognized the challenge of presenting the data at a level of detail that would encourage use by the public. To accomplish this, they used dashboards, barometers, vital signs, and community indicators that were a small set of the detailed measures that were used for internal management. Each of these mechanisms focuses on metrics that are most important to the public. These cities are increasingly concerned with presenting the appropriate data to the public—both what they think is important and the level that will encourage the public to spend time learning about how their city is performing.

One of the major challenges is to convert greater accessibility to data to greater participation and engagement by citizens. The maxim “Build it and they will come” does not apply here. The cities that we engaged with, which are among the most progressive in the arena of publicly accessible data about their municipal performance, recognize the work that needs to be done.

In addition to the efforts in these cities to increase public use of their data, there is support from a number of non-government organizations that are stakeholders in the increase in public transparency. Some of these organizations include the Governmental Accounting Standards Board (GASB), the Association of Government Accountants (AGA), and the Government Finance Officers Association (GFOA). Their work helps move best practices in public reporting more quickly to other cities and towns.

The interest in performance reporting to citizens has grown in the last decade. Several of the aforementioned organizations provide guidelines to help local governments provide effective public reporting. In June 2010, GASB issued “Suggested Guidelines for Voluntary Reporting.” These guidelines are termed SEA (Service Efforts and Accomplishments) Performance Reporting. According to GASB, the primary purpose of a government is to help maintain and improve the well being of its citizens by providing services.

Other organizations promote similar guidelines for public reporting and make awards to local governments that effectively follow those guidelines. AGA does this through their Citizen-Centric Reporting Program. The National Center for Civic Innovation recognizes local governments that engage citizens in the performance reporting process. GFOA has put an increasing emphasis on performance management in its publications and recognizes it in its Awards for Excellence.

Benefits of Public Performance Reporting

The cities we talked to cite increased transparency as a benefit of public reporting. Bellevue believes it is the fact that they are transparent and a high-performing city that accounts for the light public use of the data.

Most of the cities believe that performance data broadens the perspective of the citizen beyond anecdotes and what a citizen observed on a single occasion. Most of these cities also tie the performance data to the budget and state that doing so has enabled them to get public support for capital programs and increases in revenue generation when they were needed.

The Multiplier Effect: Inter-City Collaboration

Open data is about sharing and exposing information for the good of all. In the context of local government, this can be about more than just the relationship between cities and their citizens. There are over 39,000 counties, cities, and towns in the United States, along with another 37,000 special districts that run discrete operations, such as water and sewer, fire protection, airports, mass transit, business improvement, etc. Each of these entities working independently to establish goals and strategies, measure and review performance, and share their results with the public are certainly on the right track to managing better. Yet, imagine if they were able to harness their collective knowledge to help each other improve. What kinds of benefits could be achieved in a world where cities shared performance data and collaborated with each other to get better? We call it the “power of the network.”

A good performance management system allows us to answer three basic questions:

• How are we doing?
• What could we be doing better?
• How can we learn from our peers to improve?

Up until this point, we have been primarily concerned with the process of local governments managing performance from an internal perspective. We’ve also discussed the requirements of any technology solution that enables the capture and reporting of performance data to occur. All of this helps us answer the first question, “How are we doing?” We can also begin answering the second question, “What could we be doing better?” by examining trends in our data—are we getting better or worse? So, how do we gain even better insight into the “what” question and follow that by answering the third question, so that we don’t have to reinvent the wheel ourselves? The answer lies in the two inter-jurisdictional collaborative facets of performance management—compare and learn.

Compare

By comparing our measures, we gain actionable information. In providing a context through comparison, we can assess whether we are doing well or not as well as we would like. There are a multitude of contexts and many ways to compare any measure. If we filled ten potholes, cleared ten crimes, or confined ten fires to the room of origin, are we doing well?

Temporal Comparisons

The most common context for comparison is to what we normally do, whether normal is defined as last month, the same time last year, or a seasonally adjusted index. All of these are known as temporal comparisons or comparisons over time. These comparisons are available to any government that routinely collects the same data and uses it to monitor its performance. It provides powerful information to see if you are doing better or not.

Peer Comparisons

An enhancement to simply seeing if you are doing better is seeing if you are doing as well as others. “Benchmarking” is the comparison of our own measures to the same measures in other jurisdictions and falls under the term inter-jurisdictional comparisons. Benchmarks can tell us whether our performance is better than most others who are doing the same thing or whether it needs improvement just to get in the game.

It is important to compare to like peers, that is, other cities that share characteristics related to what you are measuring. These characteristics can be demographic, such as population, land area, or median household income. We look for like demographics because they are correlated with the workload that is encountered when delivering a particular service. In some cases, we can also find like peers by considering service characteristics, which are more directly related to the workload to deliver the service. An example is the number of collection stops for solid waste collection. Service characteristics that reflect the level of service provided also help find like peers. Providers of twice-a-week solid waste collection would not typically be compared to once-a-week collectors.

Comparisons to Targets

Proactive management does more than just comparing current performance after the fact. It motivates better performance and sets goals (through performance targets). By comparing performance to targets defined beforehand, management is acting by making a statement about the performance level that is desired. Targets, if they are reported publicly, are a commitment to your citizens.

Both temporal and inter-jurisdictional comparisons inform the setting of targets. Anticipated changes in your workload, changes in resources available for the service, and your managerial initiatives, do this as well. To complete the performance improvement cycle, the performance measures are compared to these managerial targets, and the differences are analyzed and reviewed to initiate the third step of the performance management process, which is learning what can be done.

Learn

The next step is discovering what you can do, based on your measures and comparisons, in order to achieve improvement. If you have set managerial targets, the stage is set to learn, particularly in areas (and for measures) where the targets have not been achieved. The management team for each service area learns through internal discussions of the expectations versus the actual performance. Were the expectations inappropriate? Were expected resources not available? Did conditions change unexpectedly? Is something else required that has not been accounted for?

The review by a management team is very important and can keep performance on track, accessing just the knowledge of that team, but if you are benchmarking against other jurisdictions, you may be able to learn techniques and approaches that the internal team cannot envision. Technology has started to improve the effectiveness of this external learning, but there is even greater potential. We have seen the acceleration of the ability to spread improved techniques through email listservs of a community of colleagues, but the potential offered by combining context-specific data with social networking technologies offers the promise of practical and efficient capabilities to learn from others.

Towards Performance-Based Collaboration

At Revelstone, we built an online performance platform, Compass, with these three elements of the performance management challenge in mind—measure, compare, and learn. We think tools like this, which allow cities to collaborate through performance benchmarking and peer-to-peer learning, are the future of the government-to-government movement. When we first embarked on this journey of performance-based collaboration, we expected that municipal leaders would want to see how well they were doing in comparison to others. However, we assumed they’d be reticent to expose themselves and, therefore, would want to participate in the process anonymously.

To the contrary, we discovered a specific desire for people to identify themselves on the platform, so that they could identify like peers, initiate connections, and build learning-based relationships with each other. So, we built a feature in our product that allowed each participating entity to opt out and remain anonymous. Much to our surprise, to date, not a single user of Compass has chosen to participate anonymously; they all want to be part of the collaborative community. Clearly, our skepticism about the willingness of government leaders to share their performance results—at least with each other—was largely unfounded. Is it possible we are seeing this phenomenon only among early adopters of this technology and approach (i.e. selection bias)? Sure, but we are hopeful they will set the trend for the rest of the mainstream market that is sure to follow.

Will the need for improved performance drive local government leaders to expose their data to the public, or will the open data movement influence the culture of government to be more transparent and engage with the public to its benefit? Perhaps both. Either way, change is coming. It already has been demonstrated in larger cities, in some cases for decades. Now, advancements in technology are enabling smaller cities and towns to participate in the open data movement as well. These trends are also fostering a virtual community of municipalities that are the forerunners of the gov-to-gov movement. They are collaborating with peers to gain valuable context with respect to their performance and learn from each other to improve. We should expect to see accelerated participation as the viral effect begins to surface: “Hey, you should be doing this too, so I can learn from you and you can learn from me.”

It is time for cities to embrace openness, transparency, and engagement, create a closer relationship with their citizens, and help improve their quality of life. In order to maximize impact, they should leverage each other in the process, taking advantage of the power of the network to drive learning, collaboration, and improved outcomes for all. The open data movement is here. “It’ll shake your windows and rattle your halls.”

About the Authors

As President and CEO of Revelstone, Ken Wolf is the guiding force behind Revelstone’s vision to bring a low cost and easy-to-use performance management platform to local governments. With nearly 20 years in the technology industry, Ken has a solid track record of building market-leading businesses and high performing teams. Prior to launching Revelstone, Ken conceived and established Revelwood, a performance management consultancy, more than 15 years ago. Ken is a Certified Public Accountant and received a Bachelor of Science degree in Economics with a specialization in Accounting from the Wharton School of the University of Pennsylvania.

John Fry is the Program Director for Government Solutions at Revelstone and utilizes his experience of over twenty years in local government to guide the development of content focused on the needs of government. His diverse career includes being a municipal administrator and consulting on municipal efficiency, emphasizing the techniques of performance management and shared services. His education in data, research and computing led to his work in at the Fund for the City of New York, where he implemented the first performance management system in New York City, Project Scorecard.

References

• Governmental Accounting Standards Board. (2010). Suggested Guidelines for Voluntary Reporting of SEA Performance Information. Washington, DC.
• Tetherless World Constellation International (2013). Open Government Dataset Search. Retrieved from http://logd.tw.rpi.edu/iogds_data_analytics