Evaluation of Voting Systems

The recent spate of security issues and allegations of "lost votes" in the U.S. demonstrates the inadequacy of the standards used to evaluate our election systems. The current standards (the FEC Voting Systems Standards) along with the revision being developed by IEEE 1583 (see the article by Deutsch and Berger in last month’s Communications) are poor from another perspective: they establish a single pass/fail threshold for all systems, thereby eliminating incentives for existing suppliers to improve their products and rendering the market unattractive to new entrants. Moreover, they fail to precisely define the properties that should be required of a voting system. Instead, the standards rely on specific designs that are more than 15 years old. These legacy designs handicap promising new approaches, such as the various voter-verified printing schemes. New systems are unnecessarily burdened, while their substantial advantages go unrecognized.

A set of well-defined properties would encourage the development and commercialization of better voting systems, especially when combined with objective ways to measure performance with respect to those properties. The overall result would then resemble the quantitative federal ratings for automobiles, where features such as vehicle safety and fuel efficiency form a basis for Consumer Reports-style comparative tables. Similarly, specific performance rating guidelines for different aspects of voting systems would provide meaningful metrics upon which system developers could compete. Decision makers, both regulatory and purchasing, would then be free to establish their own minimums for these metrics. Such a rating system can thus cleanly disentangle the development of the technical evaluation process from the various political and regulatory processes.

The Chair of the U.S. Federal Election Assistance Commission (EAC), DeForest B. Soaries, Jr., recently asked the technical community for assistance in determining a new standard. This community is no stranger to the area of voting system properties and standards: a number of authors have tried to characterize requirements, and, in 2002, the Workshop on Election Standards and Technology addressed similar issues. The performance properties for voting systems might include the following: integrity of the votes (both voter verification, "I can check that my vote was captured correctly" and public verification, "anyone can check that all recorded votes were counted correctly"); ballot secrecy (both voter privacy and resistance to vote selling and coercion); robustness (including resistance to denial of service attacks); usability and accuracy (including access for the disabled); and transparency (both of mechanism and election data).

The inherent differences in system architectures can be characterized abstractly on two levels. Architectures are first compared by how well each can satisfy the overall properties, then are characterized by the kinds of building blocks they need and by the assumptions they need to make about those blocks. A standard should provide an objective way to measure, for a particular actual system implementation, how well its building block instances ensure the properties required of them by the architecture of that system.

Suitable performance evaluation and measurement standards already exist for several types of building blocks: FCC 47CFR shielding and emissions, FIPS rating of tamper-resistant equipment, and the Common Criteria for software. For some properties, objective and repeatable measures of overall performance can be defined. For example, the accuracy of a user interface in capturing voter intent can be experimentally tested in a practical and repeatable manner, with the result expressed as an error rate. "Tiger team" and code review security evaluation (while certainly not foolproof) should play a role along with ordinary reliability testing. Ideally, this process of developing the properties and characterizing architectures would be exceptionally transparent, such as that for Internet RFCs, and would be subject to appropriate peer review. The refinement and adaptation of the measurement techniques would proceed as an ongoing parallel activity.

The EAC’s request for assistance is a unique chance to positively affect the quality of our election systems, by tackling this new scientific and technical challenge and building a solid foundation. The aim should be to impact the 2006 elections, though the timing is already tight: the EAC is required to present technical recommendations to the House Administration Committee in April 2005. The technical community is faced with a significant need, a rare opportunity, and a growing urgency for coordinated technical effort in this area. (See www.vspr.org for further details.)