Hacking FOIA: Using FOIA Requests to Drive Government Innovation

Turning FOIA On Its Head

The Freedom of Information Act (FOIA) is a law that gives you the right to access information from the federal government. It is often described as the law that keeps citizens in the know about their government. (www.foia.gov)

Even according to the government’s own FOIA website, FOIA is presented as a way for citizens to gain an advantage, but there are two sides to this coin. FOIA can be used as a tool to help government agencies help themselves.

Contrary to popular belief, Google doesn’t have the answers for everything. FOIA requests, while typically presented as a way for citizens to extract information from the government, can actually provide the best pathway to help government agencies innovate, work smarter, and become more efficient.

There is an opportunity before us to use FOIA to benefit the government on a massive scale. FOIA is about more than transparency; it can be the basis for true collaboration.

The Problem: Disconnected Data Silos

We live in a world empowered by and accustomed to instant data, instant results, instant answers, and instant analysis. Unfortunately, for those in government, many agencies are forced to operate where the answers they need are either not available or difficult to track down.

Imagine traveling back to a time before smartphones, before mobile phones, before the internet. Suddenly, “Don’t Stop Believin’” takes over your universe and your brain desperately wants to know the name of the drummer for Journey. As you reach into your pocket, you find your iPhone. Then, it hits you…there is nothing for your iPhone to connect to in this year, so there is no way to get this information easily. There is no Google to help you yet. You’ll just have to wait until you can find the answer, and that may take some time and elbow grease.

When you’re used to nothing but two or three clicks between you and the data you need (in case you were wondering, Steve Smith was Journey’s drummer during their peak years), having to wait for answers feels horrifically inefficient. We’ve rapidly evolved from reactive receivers of information to proactive wielders of information.

Government agencies face much tougher questions than names of band members, and the stakes are much higher. Most government agencies do not yet live in a connected world, and they are forced to waste time, resources, and money simply due to the absence of easily accessible (and searchable) information.

This is not to suggest government agencies are devoid of technology; quite the opposite is often the case. The obstacle for most government agencies is not access to technology, but to actionable data. The government may be masters of collecting data, but it took the private sector to invent the masters of data insight and connectivity, such as Twitter, Facebook, and Google.

According to the most recent figures from the US Census Bureau (2012), there are 89,055 official governmental entities in the country. The vast majority of them have their own individualized systems and are, due to data silos, disconnected from other government agencies.

How Do Government Agencies Handle Purchasing?

An area that perfectly illustrates a stunning data disconnect is government procurement. The government, at all levels, has to buy everything; they do not make their own paper, desks, light bulbs, cables, computers, planes, tanks, or carpet. Everything the government needs, they have to purchase elsewhere from government contractors.

The purchasing process is primarily based on price thresholds, and the available pathways are verbal quotes, written quotes, bids, and RFPs (Request for Proposals). The following are examples of typical guidelines, though each jurisdiction adopts variations and exceptions (most often for construction and IT projects, which have significantly higher thresholds):

• Less than $2,500: Obtain one verbal quote from a government contractor known to be competitive, and it is within the agencies’ discretion to choose the winning contractor.
• $2,500 to $5,000: Government agencies are usually required to get three verbal quotes, and although low bid is the prevailing priority, these choices are also within the discretion of the purchasing agent.
• $5,000 to $10,000: A government agency would typically seek three written quotes. This is usually the highest price range that can still be decided by end-users.
• $10,000 to $25,000: Government agencies typically have to open a bid process, and then every registered (i.e. known) vendor gets notified.
• $25,000 and up: Government agencies usually must enter a budgetary action process and issue a call for bids and/or RFPs, which is sent to all registered vendors.

The overwhelming majority (more than 80%) of government purchases is found below the thresholds for formal bids and RFPs. Purchases for low-priced commodity items (laptops, printers, cables, paper, etc.) are usually what you’ll find below the bid/RFP threshold.

What Does a Government Agency Want to Know When Purchasing?

When a government agency has a need for a product or a service, they aspire (though currently, rarely succeed) to have answers to the following questions:

• Best price: What is a competitive price? Who has the lowest price?
• Best value: Which vendors are the most responsible and reliable? Who offers the best quality and warranty?
• Choice: Are we reaching as many vendors as we can? Are we reaching out far enough to get the right vendor? (Note: Government agencies routinely cite finding the right qualified bidders as one of their biggest challenges.)
• Piggybacking Opportunities: Are there existing contracts out there that we could piggyback on for a lower price?
• Peer Feedback: What other government agencies have purchased this before and could give me useful insights? Who has existing RFP or bid language that we could use? (Note: Lacking knowledge about the product is another top challenge cited by government agencies, especially for technology purchases.)
• References: What other government agencies have had experiences with the vendors we are considering? Is there something we don’t know?

How can a government agency answer these questions in a sector plagued with data silos? The best price may not be what is known locally or regionally, but nationally.

There are cooperative buying services that provide pricing research platforms, such as US Communities (http://www.uscommunities.org) and IPA (http://www.nationalipa.org/), but since inclusion is voluntary, these platforms only provide data from participating government agencies and vendors, and the information they provide does not answer all of the questions a purchasing agent needs to know to find the best value on all products and services they need to purchase.

The Current Challenges of Coordinated Government Purchasing Efforts

There has been no comprehensive database to find other government agencies with detailed purchasing history.

When purchasing agents are only required to get quotes from competitive vendors, without external validation, it can lead to the same vendors being used repeatedly. The nature of the process and the lack of resources typically lead to the same vendors being repeatedly called upon. What if a purchasing agent had access to every purchase made by every agency across the nation? Would they be in a better position to evaluate their purchasing options?

When a government agency must offer a contract through a bid or RFP process, the best vendors might be actively watching for RFP and bid notifications, or they might not. For the ones that do respond, they may provide references, but omit the bad references. There’s been no way for government agencies to know about the references that vendors don’t share, and there hasn’t even been a way for them to know—with certainty—where the best value can be found.

Sole source procurement is another challenge for government agencies because, by definition, a sole source cannot be the best value; it is the only choice. However, sometimes a contractor is presented as a sole source, when, in fact, they are not. While a local search may not uncover this, a national search would have a higher likelihood for success. It could be that a similar product is offered by a vendor in another state. Other times, it could be that there is another (and possibly cheaper) distributor of the exact same product that could have been unknown to the “sole source.”

The end result of these data silos is that a majority of government purchases are made at a higher price than the best available rate.

The data has always been available, but it wasn’t connected or indexed. In millions of separate data files, the answers to everything a government agency would want to know could be found. Somewhere, a lowest price was entered into a document or spreadsheet. Somewhere, a government agency found a better way, a better source, or a better product that everybody needed to know about but never got shared.

Government Purchase History as an Open Data Solution

There is, however, one place that all government purchases (at all levels) are captured: purchase orders and purchasing cards. While less than eighty percent of government purchase activity uses a bid or RFP, effectively a hundred percent use a purchase order or purchasing card.

Purchasing data contains a wealth of information: date, buyer, seller, product descriptions, line-item pricing, quantities, and more. The problem, until recently, is that this incredible repository of actionable data—the purchase transaction data—existed independently from agency to agency. Most city, county, state, and federal agencies store their purchasing data in different formats and in different systems.

The idea for SmartProcure came from witnessing the same product being purchased at wildly different prices by different departments in the same city. This was a problem that could be solved with a proper database, but no such database existed at the time.

Thus the concept of hacking FOIA began. Instead of trying to build a process from scratch, it was more efficient to use an existing and accepted government practice (FOIA) and use that to proactively obtain purchasing data. Instead of relying on the government to pay for such a database, a decision was made to create the database at no cost to the government. Revenue would be generated from selling access to the database to government contractors. In this way, the government would get the benefit of every agency’s data at no cost, and government contractors would gain access to powerful business intelligence.

FOIA requests have helped SmartProcure acquire nearly sixty million purchase orders (and counting) at the local, state, and federal level. Purchasing data is obtained from the government (voluntarily or through FOIA requests), which gets converted into a normalized and searchable format, and the data is given back to the government at no cost. SmartProcure was the first (and still the only) provider of a fully indexed and searchable database of government purchasing information.

Now, empowered by a searchable database of purchasing information from across the nation, government agencies are able get the best value. They can use the information to instantly see all data for every purchase of any product, identify who sells that product, and find the best pricing. They can get quotes from the best vendors, not simply the already-known vendors, as had been the case before searchable government purchasing data became available.

Paul Brennan, a purchasing agent for Rockland County in New York, was using purchasing history to research pricing for a current project. “I used SmartProcure to look for purchase orders for the pavement rollers we needed. I quickly found a contract in Texas at a much cheaper price than I could find here in the northeast, and I was able to piggyback on that agency’s contract and purchase two of them at a savings of $30,000,” said Brennan.

FOIA requests can help people in government agencies find the right person to talk to. For example, if a purchasing agent were tasked to purchase an unfamiliar item, it would help to talk to someone in another government agency who actually has experience with the item. The purchasing agent can search the purchase history database for the item, find a purchase order, and then look at the contact information for the purchasing agent. That goes beyond pricing data and takes it to connecting the people with the questions to the people with the answers.

Jason Phitides, a purchasing agent for the City of Jacksonville Beach in Florida, was faced with a daunting challenge. “I needed to purchase a ‘beach dune walkover,’ but I was new to the area and had no idea what it was. I used SmartProcure to search for the item, quickly found other agencies that purchased them, got the contact information for the purchasing agents, and I was able to talk directly to the people that could share relevant knowledge and experience,” said Phitides. “Access to this data not only helps me find the best prices, but it helps me do my job better.”

Access to other purchasing agents may help give content and context to RFPs that need to be written and released by another agency, thereby saving time and enhancing the quality and specifications included in the RFP. Except for highly unusual items, it is nearly certain that some agency has already done the research and created an RFP for just about any product or service.

When government agencies are separated by data silos, thousands of hours are wasted created new RFPs and conducting research, when a simple search in a purchasing history database could connect agencies in search of this content and research in just a few steps.

FOIA and Spending Analysis

The drive for best price and best value has led to an increase in government agencies needing to analyze their spending in extreme detail. However, with the barrier to entry being an expensive spending analytics platform and a custom feed of the agency’s data on the regular basis, the momentum needed to start this type of program is usually not reached. With easy access to spending analytics using a platform similar to SmartProcure or others, a government agency can easily perform a detailed analysis.

An open purchase history database includes actual purchase data at the line item level, and that means government agencies can perform highly detailed spend analytics in seconds. Want to know how much was spent on paper towels? Pencils? Chairs? Staplers? It’s easy when they’ve got the right data.

With an online purchasing database, even just one individual can instantly search all purchase history for any desired data, and easily generate reports. Government agencies—and the people who work in them—can save time, money, and resources.

A quick survey of SmartProcure’s purchase history database found more than $4 million in purchase orders where government agencies paid an outside resource to help them analyze their own spending data. This is something that can be done for free with a government purchase history database, and that’s a tangible example of how much the lack of information can cost the government.

FOIA and Crowdsourcing

We already know that crowdsourcing works in the public arena, and FOIA requests are a way to bring crowdsourcing to the government arena.

Using FOIA requests to gather, collate, and share information solves two enormous problems at once. First, government agencies want to do better, and they need the best resources to do that. Second, members of the public (e.g., commercial businesses) often have many ways they can help the government but are not connected in ways that allow them to effectively or efficiently help.

Major movements and major decisions for government agencies often have good data and oversight, but the ocean of micro-movements and micro-decisions can lead to a crippling “Latte Effect.” With more than 89,000 government agencies hobbled by disconnected data, every time a printer is bought for 10% too much, or a ream of paper for $2 over the best available price, or when a purchasing agent is forced (unnecessarily) to rely on known local vendors and overpay by $30,000… these add up to an enormous collective drain on the economy.

Imagine the possibilities if more organizations found ways to connect solutions and problems by way of FOIA requests. FOIA requests can be used not only to save the government money, time, and resources, but also to connect them to better solutions and increase innovation.

Coordinated FOIA efforts can be a springboard to innovation, helping to build other products without the need to wait for government inertia. The nature of FOIA requests is that a government response is involuntary. One doesn’t have to wait for government agencies to collectively decide to do something. You can use FOIA requests to break through the inertia that exists when developing a multi-agency solution.

A good business model using FOIA requests is to take government data silos (there are many to choose from), organize the data into a single searchable resource, then provide that information back to the government agencies in a way that connects them all to each other.

For any problem faced by any government agency at any level, rest assured that somebody, somewhere has an answer. You can use FOIA requests to connect the people with issues to the people with answers. Crowdsourcing with FOIA can provide unprecedented speed and precision.

“Momentum obstacles” plague any government agency stuck in a data silo. Coordinated FOIA requests can entirely sidestep the need for breaking through these obstacles. If no government agency has to pay, and the data brings actionable benefits, the crowd creates itself, and the benefits of crowdsourcing will flow.

Large-scale FOIA request provide a fertile ground for crowdsourcing. There are a billion separate data files that have captured details on activity and/or decisions at every level of government. If you connect them with FOIA you have instantly created a fully interconnected crowd of enormous size. With government purchasing data, for example, the crowd was there all along, but it took FOIA requests to bring everyone together.

Crowdsourcing is about creating a “master mind.” The whole is much greater than the sum of the parts. An individual may not know the answer, but crowdsourced data will provide it in seconds. Crowdsourcing also allows large-scale projects to be broken down and distributed among multiple individuals and agencies, further reducing the inertia needed to be overcome to get a necessary—but difficult—project moving forward.

Crowdsourcing with FOIA can transform the more than 89,000 government agencies into a single organism. Each individual and each entity can not only find a good answer, but the best available answer. Not just a competitive price, but the best price. Not just a solution, but the best solution.

The hunger of the incoming generations to solve big problems with data and technology is palpable. In Generation We, Eric Greenberg and Karl Weber (2008) defined millennials as those born between 1978 and 2000, which represents 95 million people, making them the largest generation in US history. It is this generation that has grown up in the era of social media and crowdsourcing, and these are the tools that they wish to change government with.

In US Politics and Generation Y: Engaging the Millennials (2013), David Rank wrote:

While the annual survey of our nation’s college freshmen revealed a three-decade trend of declining political interest, hitting a record low in 2000, by 2006 more entering freshmen had expressed interest in discussing politics than at any point in the history of the forty-year survey, including the 1960s (HERI 2007). Studies concluded not only that the emerging generation was more politically engaged, but that we needed to recognize new forms of such democratic participation (Bennett 2007a; Dalton 2008; Zukin et al. 2006). (p. 6)

Instead of complaining about the lack of young, top tech talent going into government, a better solution is to leverage FOIA to build systems that pull the best talent into the government arena.

Bringing It All Together

It is time to change everyone’s perspective on FOIA. It can be hacked to become a powerful force for positive change by the government and for the government. It’s not just about uncovering secret information, as is typically stereotyped. It can also be (and should be) about connecting disconnected information in a way that everybody can benefit.

Recently, there has been an explosion of organizations that have aggregated data to help local governments, and some have been successful in spreading their concept to multiple cities. Hacking FOIA, though, is about taking all of this to the next level. It goes beyond solving a local problem, a county problem, a state problem, or a regional problem. Hacking FOIA goes beyond a singular focus on police departments, or fire departments, or public works, or IT.

Such efforts are laudable, and they should absolutely continue. But hacking FOIA is about solving big national challenges all at once with coordinated data, and at all levels of government using existing processes. It can be done, and is being done.

About the Author

Jeffrey D. Rubenstein is the founder and CEO of SmartProcure. Jeff is an accomplished senior executive responsible for building successful technology companies, most recently building and selling Advanced Public Safety to Trimble Navigation Ltd. (TRMB). Jeff is an attorney, auxiliary law enforcement officer, and has spent his career working with government organizations. He is passionate about leveraging data to build efficiencies between the public and private sector.

References

Greenberg, Eric H., and Karl Weber (2008). Generation We: How Millennial Youth Are Taking Over America and Changing Our World Forever. Emeryville, CA: Pachatusan. Rankin, David M. (2013). US Politics and Generation Y: Engaging the Millennials. Boulder, Colo: Lynne Rienner Publishers. https://www.rienner.com/uploads/50b90f1f69e38.pdf United States Department of Justice. “FOIA.gov - Freedom of Information Act.” Accessed September 16, 2013. http://foia.gov