Wednesday, October 29, 2008

BREAKING: Deja Vu (all over again) -- Feds Pull NY's Testing Lab's Accreditation

The ink has barely begun to dry on a petition to save New York's lever voting system signed by over 1,100 courageous New Yorkers (so far) and -- what do you know?:

The National Institute of Standards and Technology (NIST) and the US Election Assistance Commission (EAC) have, for the second time in two years, suspended the accreditation of New York's electronic voting system testing lab, SysTest, Inc.

The first such suspension was carried out against Ciber back in 2006, after NY State tech consultants, as well as citizens who were paying attention, couldn't help but notice that Ciber had not been conducting tests required to meet federal and NY State standards and election laws.

That was then and this is now

As NY Yankees and Mets catcher, coach and manager Yogi Berra would say, this time "it's deja vu all over again." Yogi brought the 1973 Mets from last place in the final month of the season to win the National League pennant the year after the death of their beloved manager, Gil Hodges. But unlike Hodges' untimely passing, the death of New York's lever voting system has been greatly exaggerated by everyone from election officials, to election integrity advocates, to rabid anti-lever zealots who, unfortunately, just haven't thought the lever replacement issue through.

Fortunately, cooler heads have prevailed

You don't walk in the winning run with the bases loaded, and you don't dismantle a properly functioning, testable, transparent voting system and replace it with the kind of junk that can't even be adequately tested in the first place (because it runs on software), by labs such as Ciber and SysTest, who can't even meet the standards that allow them to attempt to do the testing.

And besides, New York's voting system is now fully HAVA-compliant, thanks to the addition of at least one ballot marking device for voters with special needs at each polling place. The lever system meets all other HAVA requirements.

Here's an explanation of why some of these problems have been occurring (NIST's computer security experts have already stated that testing software to high levels of security and reliability is impossible) -- not that it excuses SysTest's performance or Ciber's.

And here's the memo from NIST withdrawing SysTest's accreditation, as transcribed by NY State Board of Elections Co-Chair, Doug Kellner (Note -- we took the liberty of correcting some errors that occurred during scanning of the original document by a software-based optical character recognition system. Hope we got 'em all.):

National Institute of Standards and Technology
Gaithersburg, Maryland 20888

October 28, 2008

Mark Phillips
Vice President of Compliance Services
SysTest Labs, Incorporated
216 16th Street, Suite 700
Denver, CO 80202-5115

NVLAP Lab Code 200733-0

Dear Mr. Phillips,

On behalf of the National Voluntary Laboratory Accreditation Program (NVLAP), I write to notify of you of NVLAP's decision to suspend its accreditation of SysTest's electronic voting testing program pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10. This letter provides an explanation of NVLAP's decision and describes the steps SysTest can take to reinstate its accreditation. This action pertains to voting systems under review by SysTest to be recommended for certification by the Election Assistance Commission for future elections and is not pertinent to systems already deployed for the 2008 election which were certified under alternate systems.

Background Discussion

SysTest Labs, Incorporated is currently accredited by the National Voluntary Laboratory Accreditation Program (NVLAP), a program within the National Institute of Standards and Technology (NIST), to perform testing to federal standards in accordance with the Help America Vote Act of 2002 (HAVA). These standards are the 2002 Voting System Standards (VSS-2002) and the 2005 Voluntary Voting System Guidelines (VVSG-2005). On August 8, 2008, NVLAP sent SysTest Labs a letter outlining specific concerns with respect to SysTest's NVLAP-accredited testing of voting systems, including voting system test campaigns submitted to the Election Assistance Commission (EAC) under their voting system certification process. These specific concerns are documented in the March 2008 NVLAP on-site assessment checklist, produced as part of the normal reassessment process, and in communications between the EAC and NIST regarding issues that EAC staffidentified with test reports submitted by SysTest Labs (enclosed).

The August 8th letter (also enclosed) outlined three specific concerns. In short they were:
1) SysTest's lack of properly documented and validated test methods.
2) Testing conducted by unqualified or untrained personnel.
3) Improper assurances made to manufacturers regarding testing outcomes.

NVLAP directed SysTest to submit information to NVLAP, including a schedule of all accredited voting systems testing planned, within 14 days of receipt of the August 8th letter. NVLAP informed SysTest ofits intention to conduct on-site monitoring of the testing of electronic voting machines. SysTest was notified by email on October 6,2008 of NVLAP's intention to visit their lab on October 14th through 16th to observe testing that had been scheduled during that period.

NVLAP assembled a team consisting of the NVLAP voting system technical assessor, the NIST/NVLAP program manager for voting system testing and four members of the NIST Information Technology Laboratory (ITL) involved in writing the federal voting system standards. In addition, two EAC staff members were invited to provide their observations. During the on-site visit this eight-member team witnessed several tests, interviewed testers , and examined documents related to the areas of concern.

Site Visit Observations

As a result of this on-site monitoring visit, NVLAP has serious concerns about SysTest's performance of voting system testing. These concerns were supported by observations of testing where the test methods being used were not fully developed, validated, mapped to the requirements of the applicable standards, and controlled under SysTest's document control policy.

From the team's observations it was unclear who at SysTest had the ultimate responsibility for test method development. During the observed tests, it appeared that the testers were running the tests for the first time. Changes were made to the test procedures to address items that should have been caught during an initial run-through of the test. Basic tests, such as the system readiness test, were not conducted successfully. Three test methods failed due to problems with the procedure, tester error, or unfamiliarity with the test set-up. Some anomalies or potential problems during testing were not reported by the testers but were pointed out by members of the on-site team.

During the team's visit SysTest personnel stated that their policy was to validate test methods during the actual testing of voting equipment. This approach is unacceptable. The lab must validate all test methods separate from actual testing so that equipment nonconformance can be isolated from test method problems. This validation must follow set documented procedures and show a clear chain of responsibility for the process. SysTest has undergone numerous changes in personnel since its original accreditation and, in fact, since the March 2008 NVLAP on-site assessment. SysTest staff conducting testing during the monitoring visit demonstrated a lack of familiarity with the test equipment and procedures. Some personnel who participated in past on-site assessments were no longer associated with the NVLAP-accredited testing; they had been reassigned to work in support of state certification of voting systems. SysTest management's stated goal was to transfer the expertise and testing approach from their New York testing campaign to the NVLAP/EAC accredited testing campaign. SysTest must improve the level of training of personnel involved in NVLAP/EAC accredited testing given that SysTest has reassigned experienced testers to other work. SysTest should consider bringing in outside instructors to train laboratory personnel.

SysTest was advised that an appearance of impropriety had occurred in a case where personnel had given a client an indication that their equipment would successfully pass testing. SysTest's response was that this was an isolated incident and the person involved had not intended to give this impression. SysTest further stated that their employees were given a quiz which they feit covered training in this situation. It is NVLAP's position that this quiz is insufficient and SysTest must provide specific training to their employees on professional ethics and document the employees' intent to adhere to SysTest's stated policy.

NVLAP's Decision

Pursuant to NIST Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section 3.10, NVLAP hereby suspends SysTest's accreditation effective as of the date of this letter. SysTest Labs, Incorporated is prohibited from using the NVLAP symbol on its test reports , correspondences, and advertising during the suspension period for all voting system testing. Accreditation may be reinstated only after such time that SysTest can demonstrate voting system testing in accordance with the requirements of the applicable voting system standards and NIST Handbook 150. This demonstration must be achieved through an on-site visit to SysTest to witness testing , review documentation, interview personne1, and any other means necessary to gather objective evidence in support of a decision regarding reinstatement.

This on-site visit will occur only after NVLAP is convinced, through the submission of documentation, that SysTest has taken the necessary steps to correct the areas of nonconformance herein addressed. This documentation will include, but is not limited to:
procedures for test method deve1opment; procedures for test method validation; revised document control procedures that specifically address technical procedures; fully developed test methods showing validation, document control , and mapping to the federal voting system standards; and, procedures or policies that address methods by which SysTest will control statements or assurances to their clients regarding the outcome of voting system testing.

SysTest was accredited by NVLAP based on its ability to develop and perform competent testing within the framework of an effective management system. SysTest now needs to revise its management system to correct the nonconformances found during this visit and implement these system changes. NVLAP believes that the current SysTest management team is committed to accomplishing this goal and will work with them to that end.


Jon Criekenberger
NIST/NVLAP Program Manager
Cc: Brian Hancoek, Election Assistance Commission


Anonymous said...

So I read the latest comment by the lab that this does not relate to current elections.

Well how is that true?

Who did testing for the voting systems in use for 2008?

Howard Stanislevic said...

That comment was from the NY State Board of Elections. The reason why it doesn't affect 2008 elections in NY is because NY is NOT using any of the new systems to count votes this year. The only votes counted by computers in NY are a relatively small number of absentee ballots. Some of these are hand counted, and in total, they amount to only about 6% of all ballots cast in the state.

New York continues to have a secure election system, but this may not be the case if we replace the lever machines.

As far as the rest of the country, nearly ALL voting systems in use today were certified by either SysTest or CIBER, both of which have had problems getting and keeping federal (EAC) certification.