Independent qualification testing is a prerequisite in over 40 U.S. states for seeking accreditation of a direct-recording electronic (DRE) or paper-based voting system. While individual state laws vary, this is generally interpreted as being tested by National Association of State Election Directors (NASED) accredited hardware and software Independent Testing Authorities (ITAs) to the voluntary FEC Voting System Standards (VSS). Election directors require a NASED Qualification Number and Report before accepting systems for testing.
In 1975, the National Bureau of Standards issued the report Effective Use of Computing Technology in Vote-Tallying, which determined a basic problem in computerized elections was the lack of technical skills at the state/local level for development of standards and testing. This report and other studies conducted in the 1980s were the impetus for the ultimate development of federal minimum voluntary standards.
The 1990 VSS (Performance and Test Standards for Punchcard Marksense and Direct Recording Electronic Voting Systems) addressed the risks of security, privacy, accuracy, and problems plaguing jurisdictions, including: orphaned systems as vendors disappeared or went out of business; systems that didn't perform as expected or that gave no guarantee of minimum functionality or adequate instructions; and systems that broke after a few elections (for example, systems stored in Phoenix, AZ, or Hilo, HI, warehouses were adversely affected by environmental conditions).
By 2000, advances in voting technology, legislative changes, and the Americans with Disabilities Act warranted updating the 1990 VSS. Technical consultants, election officials, and testing labs representatives volunteered their time to assist the FEC in updating the 2002 VSS. In defining the scope of changes the authors noted, "Voting systems marketed today are dramatically improved. Election officials are better assured that the voting systems they procure will work accurately and reliably. Voting system failures are declining, and now primarily involve pre-Standard equipment, untested equipment configurations, or the mismanagement of tested equipment. Overall, systems integrity and the election processes have improved markedly." The VSS update focused on the areas of change along with strengthening requirements for code review and vendor documentation.
While the 1990 VSS called for independent testing, it did not explicitly define ITAs. In 1992, NASED volunteered to create and administer a program to accredit ITAs for qualification testing of voting systems. Labs participation was sought. Auditors volunteered their time to review the labs' quality systems and test methods. In onsite visits they verified conformance to the NASED Program Handbook. The labs include:
A voting system isn't just the equipment necessary to cast a vote. The VSS has two definitions, addressing the physical and functional components of a voting system. The physical aspect defines a voting system as comprising all the hardware and software, plus the procedures, manuals, and specifications that describe the physical and functional characteristics. The functional aspect states that a voting system has three functions: pre-vote, voting, and post-vote. The Election Management System (EMS) includes the pre-vote and post-vote functions, which occur outside the polls, including ballot preparation and central count. The Polling Place System includes all three functions occurring in the polling place.
ITA testing is built into the price of voting systems and maintenance contracts.
The terms hardware and software are misleading in describing the scope of the ITAs, because software is addressed by both ITAs. The hardware ITA performs the physical and functional testing of the Polling Place System, including:
The software ITA performs the physical and functional testing of the EMS and end-to-end testing of the integrated EMS and Polling Place System, including:
As part of the vendor's cost of doing business, ITA testing is built into the price of voting systems and maintenance contracts. The vendor contracts separately with the hardware and software ITA. Each lab establishes its own price and project structure. Once any deliverable is tendered to a lab, the vendor must complete qualification with that lab, with the cost of qualification depending upon the quality of the system submitted. If problems are found, regression tests and reviews increase the cost. Submission of a single voting system for all ITA testing ranges from $150,000 to $400,000 and can take two months to a year. Several vendors have single Ballot Preparation and Central Count Reporting programs that support multiple DREs and optical scanners. These may be submitted in a single ITA effort and would push the overall cost substantially higher.
ITA testing follows the principles of quality assurance: the ITAs must assure test consistency and repeatability with greatly differing systems. This is accomplished by having a defined, documented, standardized test process easily adapted to various voting systems, regardless of function and configuration. While following these principles of quality assurance, the ITA's test objective is to verify the system meets the VSS requirements and can be recommended for NASED qualification. ITAs may appreciate elegance in design, function, and execution, but an ugly voting system that meets VSS requirements merits recommendation. The VSS identifies two processes, as illustrated in this figure: the Physical Configuration Audit (PCA-blue) and the Functional Configuration Audit (FCA-green).
PCA Technical Data Package (TDP) Review: The TDP is reviewed to confirm that required documentation is present, conforms in content/format and is sufficient to install, validate, operate, maintain the voting system, and establish the system hardware baseline associated with the software baseline. Results of the review are provided to the vendor in a Pre-Qualification Report.
PCA Source Code Review: The voting system source code is reviewed for:
PCA Test Environment: The Hardware and Software ITAs document the setup of the voting system configuration to assure a consistent test environment. The ITAs observe building of the executable from reviewed source code. We work together to confirm that all testing is performed only on ITA-reviewed code built under ITA observation.
FCA Test Documentation Review: The ITA reviews and assesses prior testing performed by the vendor. Based upon the assessment of vendor testing the ITA: identifies scope; designs testing; and creates the Qualification Test Plan.
FCA Testing: Each ITA tests to its particular identified scope, using its own internal processes:
Each ITA produces a Qualification Report identifying the voting system tested, scope, approach, environment, and optional functionality supported by the system. Detailed results include documentation of the reviews (TDP and source code) and all testing performed, with description of the issues encountered and resolved. Risks that do not violate the VSS are disclosed in the conclusion along with the recommendation for NASED qualification.
The Qualification Report is released to the vendor and the NASED Technical Committee for acceptance or rejection. NASED provides a reason for rejection, and it is the vendor's option to resolve the objection. Both hardware and software ITA reports must be accepted by NASED to obtain a qualification number.
NASED advises states and jurisdictions to implement policies to guarantee validation that executables, documentation, and reports are ITA-qualified. They suggest states obtain qualified versions directly from the ITA for all accreditation testing. Jurisdictions can likewise require that the executable be obtained from the ITA for installation. If logistics make installation impractical, a validation of the version and checksum can be incorporated into the election database installation process.
With HAVA's creation of the Election Assistance Commission (EAC) to perform audits on a regular basis, the EAC and the National Institute of Standards and Technology are studying the current NASED model for ITA accreditation and qualification. Whether this program will change or stay the same will be determined by the results of their study.
©2004 ACM 0001-0782/04/1000 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2004 ACM, Inc.
No entries found