Search

Risk-Limiting Audits

RiskLimitingAudits.org is maintained by a community of RLA experts. Our goal is to demystify the RLA and connect you to the experts and tools you need to run RLAs in your state.

Introduction

Democracy depends upon trust in the electoral process. In the United States, nearly every jurisdiction relies on machines to tally votes, so trusting the process means trusting the machines that scan and tally our paper ballots. Given the high stakes of our elections, and our adversaries’ demonstrated interest in them, what can we do to double-check the equipment producing the tallies? The best safeguard we have against hacked or otherwise faulty voting systems is a Risk-Limiting Audit (RLA).

History

First, a little background. For almost 15 years, election administrators, statisticians, security experts, and the election integrity community have been testing innovative ways to conduct post-election audits. They now believe that RLAs are the most efficient way to gain certainty that the winner reported by the voting system is correct. Some errors are inevitable – paper gets smudged, dust particles may be read as voter marks. RLAs take all of those into account and give us confidence that the claimed outcome is, in fact, the correct outcome.

The RLA is efficient because of its insight that we do not need to manually recount every vote to gain certainty in the outcome of an election. Instead, the number of votes we need to manually count depends on the margin of victory in the contest we’re auditing. A typical state-wide RLA might require checking fewer than 1000 ballots to validate an entire state’s election results.

In 2017, Colorado became the first state to implement rigorous RLAs, auditing one race in each of 50 of its 64 counties. Colorado’s experience helped to create the requirements for software that is used to run RLAs today. RLAs are now a standard part of Colorado’s election process and Colorado’s RLAs cover the 62 of its 64 counties that use automated vote-counting equipment.

California, Colorado, Georgia, Nevada, New Jersey, New Mexico, Ohio, Oregan, Rhode Island, Texas, Virginia, and Washington have all passed legislation requiring or allowing for RLAs following elections. Other states have proposed legislation requiring RLAs.

Basic Requirements

The central requirement to running an RLA is that every participating jurisdiction uses paper ballots verified and cast by the voter (or a voter-verified paper audit trail) and organizes those paper ballots so the ballots can be located and identified after the election. If your state’s jurisdictions use hand-marked paper ballots or ballot marking devices that create a paper ballot that the voter reviews before voting, you’re most likely ready to go. If your state instead uses Directly Recorded Electronic (DRE) equipment with no paper trail, then you will need to upgrade your state’s equipment before you are able to perform any meaningful audit.

Types of RLA

There are three main flavors of RLAs: ballot comparison audits, ballot polling audits, and batch comparison audits. Ballot comparison audits require paper ballots to be maintained in scanned order to enable comparison to the digital cast-vote records, uniquely identifying the ballots is recommended. At this time, ballot comparison audits are most feasible in central-count states like Colorado, Washington, and Oregon. Jennifer Morrell, writing for the Democracy Fund, provides an excellent description of ballot comparison audits here. In most other states it is simpler to start with a ballot-polling or batch comparison audit. Fortunately, the workflow for each is roughly the same.

Demystifying the RLA

The first questions state officials have about RLAs regard their complexity and cost. So we’ll start there.

Steps to Complete an RLA

In the description below, we’ll assume the state is running the RLA, but a local jurisdiction can run an RLA independently of the state if it so chooses.

  1. The first step is to select which contest(s) to audit. This choice depends on a number of factors, including: the margin of the contest, whether all the ballots for the contest are able to be audited, which jurisdictions are participating in the audit, etc.

  2. Next, select the risk limit for the audit (usually between 5 and 10%). The risk limit is the largest possible chance that the audit will not correct an incorrect election outcome, if one exists. For example, a 5% risk limit means that in at least 95% of cases the RLA will correct the outcome of an election, if the originally reported outcome is wrong. The lower the risk limit, the more ballots need to be manually examined in order to complete the RLA.

  3. Based on the type of RLA and the details of the contest (contest margins, votes cast, etc.), RLA software is used to calculate how many ballots or ballot cards need to be audited.

  4. Each local jurisdiction participating in the audit provides its ballot manifest and, for a ballot comparison audit, its cast vote records. You can think of a ballot manifest like a shipping manifest - a list of all the containers where ballots are stored after the election and the number of ballots in each container. During the RLA, the manifest is used to randomly select ballots for the audit and locate them.

  5. The state enters a random seed into the audit software’s pseudorandom number generator to randomly select a list of ballots for audit. There is often a public ceremony to generate the random seed - and ensure it is actually random. For example, Colorado rolls twenty, ten sided dice in a public ceremony to create the random seed.

  6. Each local jurisdiction retrieves the designated ballots. The local jurisdiction’s audit board manually enters the voter’s selections into the audit software.

  7. The software determines whether the audit is complete (risk measurement falls below the risk limit) or whether additional ballots need to be examined.

  8. The state initiates any further auditing indicated by the software and then finalizes the audit report, which the audit board signs.

Cost

The costs to run a risk-limiting audit are a small percentage of the overall cost to run an election. In other words, an RLA is quite affordable and, compared to older post-election audit methods, provides a greater degree of certainty that the outcome of the election is valid.

A state running an RLA for the first time may wish to employ a consultant experienced in RLA administration to help the state run the RLA and define ongoing support processes. The software to support RLAs is free and open source. Hosting and maintenance are available from vendors. Additional expenses may include training local election administrators and staff, per diem pay for audit board members (who retrieve ballots, interpret them, and enter data), as well as any additional hardware costs related to conducting a ballot comparison audit.

Getting Started

If you are interested in learning more about conducting an RLA in your state, you should start by speaking with an expert.

Next, you’ll want to create an RLA working group in your state that should include:

  • State officials who will be responsible for running the RLA
  • County or local officials who are responsible for retrieving and inputting information for audited ballots
  • Representatives from the voting system vendor
  • Recognized statistics and auditing experts
  • Any other stakeholders who may add value or provide necessary buy-in

The goal of the advisory group is for members to become local RLA subject matter experts, make recommendations for RLA policy and rules, understand the technology requirements, outline the steps for conducting the audit, create a calendar of key dates, assist with training and pilot audits, and provide input on public communication plans.