60 Day Comment Period Comment Summary

Submitter	Date	Subject	Comment	EAC Response	EAC Reason for decision	Status/EAC action items
Frank Padilla	1/24/2011	Conformity Assessment (Section 1.6.2.1)	NOCs and RFIs are not referenced, but also address the issue.	Accepted	Document changed to reflect the suggestion.	complete
Frank Padilla	1/24/2011	Certification Application (Section 4.3)	Implementation Statement reference to the VVSG 2005 Version 1.0, Section 1.6.4 should be deleted.	Accepted	Document changed to reflect the suggestion.	complete
Frank Padilla	1/24/2011	Certification Application (Section 4.3)	Consider adding definition of Test Readiness Review to Section 1.16 (Definitions).	Not accepted	No change necessary.	complete
Frank Padilla	1/24/2011	Coding Convention Database (Section 4.3.1.6.4)	In addition, specified coding conventions should also be addressed since these are also used.	Accepted	Document changed to reflect the suggestion.	complete
Frank Padilla	1/24/2011	Implementation Statement (Section 4.3.2.2)	Same comment as 4.3 above; reference to specific VVSG location should be deleted.	Accepted	Document changed to reflect the suggestion.	complete
Frank Padilla	1/24/2011	Test Readiness Review (Section 4.4)	The statement regarding system TDP is vague; more specific requirements for content should be added.	Not accepted	No change necessary.	complete
Frank Padilla	1/24/2011	Test Readiness Review (Section 4.4)	System Components information should match the Manufacturer's application to the EAC.	Accepted	Document changed to reflect the suggestion.	complete
Frank Padilla	1/24/2011	Test Readiness Notification (Section 4.4.1)	To ensure consistency, the EAC should include a format for this statement for the VSTLs to use.	Accepted	Format will be created and added to Manual.	create form and add to manual
Frank Padilla	1/24/2011	Modification Test Plans (Section 4.5.2.3.1)	Since an outline has been included for a full test plan and outline for a modification test plan should be included for all VSTLs to follow.	Accepted	Format will be created and added to Manual.	complete
SOS Debra Bowen (CA)	1/31/2011	Table of Contents	Fix title for Section 3.	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Table of Contents	Add titles to each appendix listing.	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 1.3 (Page 1)	The Voting System Testing and Certification Manual ("Manual") should not supersede the VVSG 2005 or any subsequent version of the VVSG. The VVSG is a document that, as required by HAVA, was developed in conjunction with the members of the TGDC, who are appointed by congressional leaders and organization with appropriate subject matter expertise, then formally adopted by the commissioners. The Manual and this proposed Version 2.0 have been developed solely by EAC staff.	Not accepted	No change necessary.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 1.16	Section 1.16 should be included on Page 7, underneath Section 1.15	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 1.16	The definition for "sub-assembly" is out of alignment with the rest of the items on the page.	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 1.16	The spacing between the definition for "sub-assembly" and "system identification tools" is different from the spacing for the rest of the items on the page.	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 1.6.1.5 (Page 4)	I would like to acknowledge and express my appreciation to the EAC for including post-election auditing as part of its Testing and Certification Program in proposed Version 2.0 of the Manual. This is especially important to me as I am pilot-testing more stringent, accurate and efficient post-election auditing mechanisms with the aim of implementing them throughout California.	Not applicable	Thank you for your comment.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 3.2.2.2 (Page 19)	The first sentence can be read as giving the EAC the option to apply standards that predate VVSG 2005, such as the 2002 VSS, to new certification applications. This is prohibited, as beginning in 2008, every application for certification had required testing under the VVSG 2005 standards. This sentence needs clarification.	Not accepted	No change necessary.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 3.2.3 (Page 20)	State who is to make the judgment. (e.g., "tested by a VSTL against, and judged by that VSTL to be in conformance with…"). This clarification is necessary for a clear assignment of responsibility and accountability.	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 3.5 (Page 22)	The opening paragraph gives a stringent and specific description of "De Minimis Changes." However Section 3.5.1, entitled "De Minimis Change - Defined," gives a less stringent and very generic definition of the term. The definition in Section 3.5.1 should use the language of the opening paragraph of Section 3.5, page 22.	Accepted	EAC rewrote this section based on comments and updates to the Program.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 3.5 (Page 22)	In the first paragraph it states "A de minimis change is a change to a voting system hardware which is so minor in nature and effect that it requires no additional testing and certification…Any proposed change not accepted as a de minimis change is a modification and shall be submitted for testing and review consistent with the requirements of this manual." However, VSTLs often determine that additional hardware testing, especially environmental testing, is required, even for de minimis changes, to determine if they meet the requirements described in VVSG 2005 - Version 1.0, Volume I, Section 4, but such changes are not resubmitted as modifications. This conflict should be resolved.	Accepted	EAC rewrote this section based on comments and updates to the Program.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.3 (Page 30)	When referencing a specific section of the VVSG, this section states "(see VVSG 2005 - Version 1.0, Vol. I, Section 1.6.4)." However, in Section 4.3.2.2, the word "see" is omitted, which makes the citation appear to be a specification rather than a reference. Therefore, each time a new version of the VVSG is created; this document will need to be updated to correctly cross-reference. All quoted sections shall be listed as an example or location reference.	Accepted	EAC rewrote this section based on comments and updates to the Program.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.4 (Page 31)	Once a system has completed the Test Readiness Review ("TRR"), received Test Readiness Acknowledgement, and creation of a Test Plan is under way, how will changes to the TDP or system components affect the test plan and the Testing and Certification process? This section needs clarification.	Not accepted	TRR should be done on all components before they are allowed in to the campaign. Testing should not be required to stop on components that have already completed TRR.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.4 (Page 31)	What is the definition of the term "function" as it is used here? Is an added feature, a fix to software bug, etc., considered equivalent function? This term needs to be defined.	Not accepted	No change necessary.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.4 (Page 31)	If a component is added, halting the testing, and the VSTL ultimately determines that the system is not ready for testing, will repeating the TRR portion of the program be required?	Not accepted	TRR should be done on all components before they are allowed in to the campaign. Testing should not be required to stop on components that have already completed TRR.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.5.1 (Page 33)	What changes are considered to be allowable and acceptable changes to a manufacturer's application that will allow the testing effort to proceed? The Manual does not establish criteria for acceptable and allowable changes to an application. This section needs to be described in further detail.	Not accepted	No change necessary.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.5.1 (Page 33)	This section states that an Engineering Change that alters the function of the voting system would likely require an update to the test plan. However, Section 4.4 entitled Test Readiness Review, states that the final production model must be equivalent in function to the one submitted for testing. These two statements contradict one another and the conflict should be resolved.	Not accepted	No change necessary.	complete
SOS Debra Bowen (CA)	1/31/2011	Section 4.5.2.3.1 (Page 35)	Three additional items should be added to list general topics that must be included in all modification test plans: *Detailed description of the standard (VSS/VVSG) to which the original system was certified. *Detailed description of the standard (VSS/VVSG) to which the modified system is to be tested. *When the original system and modified system or components thereof are tested to a different standard, a detailed description of which specific components, including version, are tested to which standard.	Accepted	Document changed to reflect the suggestion.	complete
SOS Debra Bowen (CA)	1/31/2011	Appendix D	The title and documentation are not on the same page.	Not accepted	No change necessary.	complete
SOS Debra Bowen (CA)	1/31/2011	Appendix E	The title and documentation are not on the same page.	Not accepted	No change necessary.	complete
Joseph Lorenzo Hall	1/31/2011	Section 1.16	The list of definitions in this section must define COTS or include a reference to the COTS definition in the Glossary of the 2005 VVSG.	Accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	Section 2.3.1.6	The three paragraphs following the initial paragraph in this section are not justified to align with the 2.3.1.* list.	Accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	Section 2.3.2.7	The term "malfunction" should be defined and the conditions for triggering manufacturer reporting to EAC specified along with a more detailed set of reporting requirements.	Partially accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	Section 4.4	The requirement for source code review of 1% of Lines of Code (LOC) during the TRR needs to be better specified to be effective; we propose a few ways this could be improved. We would like to suggest a more effective procedure where a VSTL randomly chooses coherent pieces of source code until the aggregate amount reaches 1% of the total LOC. In addition to random selection of source code to ensure that all the relevant code has similar probability of selection, it is critical to have context from the surrounding code to understand a given line of source code. The VSTL should randomly choose a subset of functions (i.e., methods, procedures, etc.), chosen so that the aggregate count of LOC in the bodies of those functions is at least 1% of the source code. The VSTL would then review all of the bodies of those functions. That is, the VSTL could repeatedly select a random function, mark it "to be reviewed", count the number of lines in that function, add it to a running total, and continue random selection until the total LOC "to be reviewed" exceeds 1% of the lines of code. The VSTL should document what process they used to randomly select 1% of the code in their TRR notice to the EAC and then EAC experts that review this document can evaluate this scheme as part of the TRR notice and acknowledgment process.	Not accepted	The suggestion offered is more complicated and cumbersome and would add a lot of time to the TRR without much benefit. In our experience a cursory glance at the code provides sufficient information about how the code will fare during testing.	complete
Joseph Lorenzo Hall	1/31/2011	Section 4.4.1	Clarify that the VSTL must affirmatively state in the Test Readiness Notification that they've performed the TRR and that the system has passed and appears to be ready for further testing.	Accepted	Document changed to reflect the suggestion.	create form and add to manual
Joseph Lorenzo Hall	1/31/2011	Section 4.5.4.3	The forth bullet in this subsection includes an incomplete sentence.	Accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	Section 5.5.3	The phrase "The source of each COTS product…" is clearly not intended to refer to source code, as obtaining source code availability of COTS software has been a difficult proposition in past certification and evaluation efforts. An alternative working for this phrase that made it clear that the TDP should specify how the VSTL obtained COTS software would be, "The vendor or source from which each COTS product was procured must be included in the TDP."	Accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	Section 5.6.4.3	This subsection includes the statement: "Further source code review may be required of unmodified files to validate that they are unmodified from their originally certified versions." This should be clarified somewhat; there are three cases involved here: *If the original build environment is unavailable, but file signature are available, unmodified files that pass signature verification should be acceptable. *If the original build environment is available and unmodified files do not pass signature verification, the unmodified files will have to be compared against their counterparts in the original build environment, using tools like Diff, to ensure they are exactly the same. In fact, it should be possible to simply copy the original build files that are claimed to be unmodified to the new build environment such that the new (copied) files passes signature verification. *Finally, if the original build environment is unavailable and either unmodified files do not pass signature verification or file signatures are unavailable, these supposedly unmodified files will have to be treated as modified files as there is no basis to verify that they have not been modified.	Not accepted	EAC will create an SOP for Trusted Build process.	complete
Joseph Lorenzo Hall	1/31/2011	Section 5.7.7	The abbreviation "HDD," presumably meaning "hard disk drive" is not defined elsewhere in the manual. This abbreviation should be defined when it is first used.	Accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	Section 8.6 & 8.8.7	Both these sections use the term "correlation" to compare the use and configuration of fielded voting systems to the state of those systems during certification testing. Technically, "correlation" has a very specific meaning referring to quantitatively comparing the interdependence of two variables. To avoid any confusion--e.g., that the EAC will be comparing two lists of variable between fielded voting systems and those systems as tested at the VSTL--we suggest a more qualitative word be use, such as "correspondence."	Accepted	Document changed to reflect the suggestion.	complete
Joseph Lorenzo Hall	1/31/2011	"Reproducibility of Testing"	There should be explicit recognition that an important goal of the test plan and test report is to facilitate reproducibility of certification testing.			more discussion needed
Joseph Lorenzo Hall	1/31/2011	"Closing the loop"	It is important that VSTLs use information about voting system performance in the field during certification testing. First, VSTLs should use this information to better understand the conditions in which voting systems are actually used, and modify their test suites and plans to make testing more realistic.			more discussion needed
Joseph Lorenzo Hall	1/31/2011	"Closing the loop"	VSTLs can assess whether or not critical assumptions and configuration choices made by the vendor during certification testing about the reliability, usability and security of their systems hold when they are used in the field.			more discussion needed
Joseph Lorenzo Hall	1/31/2011	"Closing the loop"	For technical glitches that are seen in the field but not seen during certification testing, the VSTL can ass regression tests and test cases specifically meant to exercise such glitches and ensure that no future version of the voting system is certified that will display the observed anomalous behavior.			more discussion needed