NSV Justification Part A

TABLE OF CONTENTS

PAGE

introduction 1

Pilot Study for the National Survey of Veterans 1

Background 1

A. JUSTIFICATION 2

1. Need and Legal Basis 2

2. Information Users 3

3. Use of Information Technology 3

4. Duplication of Efforts 4

5. Small Businesses 4

6. Less Frequent Collection 4

7. Special Circumstances 5

8. Federal Register/Outside Consultation 5

9. Payments/Gifts to Respondents 6

10. Confidentiality 6

11. Sensitive Questions 7

12. Burden Estimates (Hours & Wages) 7

13. Capital Costs 8

14. Cost to Federal Government 9

15. Changes to Burden 9

16. Publication/Tabulation Dates 10

17. Expiration Date 15

18. Certification Statement 15

Tables

Table 1. Time and Cost Burden 8

Table 2. Survey Operations Data 11

Table 3. NSV Survey Metrics 14

Supporting Statement A - Justification

Pilot Study for the National Survey of Veterans

INTRODUCTION

Pilot Study for the National Survey of Veterans

The National Survey of Veterans (NSV) will be used to collect information that is not available in VA administrative files. The NSV is planned for administration in August 2009. Before the NSV is administered to a large national sample, the VA is sponsoring a design study that will determine the nature, scope, content, and sampling design for the National Survey of Veterans. In particular, this design study will:

• Develop and design survey instruments for the main NSV;

• Develop and design a sampling methodology plan with cost effective approaches for maintaining national-level sample representation of subpopulations of interest;

• Develop a cost effective data collection plan; and

• Develop and provide a project plan to conduct the NSV.

• An important task within the design study will be to conduct a Pilot Study which will be used to test the assumptions in the sample design and also to test the data collection methodology and operations.

Background

The NSV to be administered in 2009 will be designed to help VA improve services for beneficiaries and their families. For this most recent requirement the scope of the survey has been expanded to address the requirements of P.L. 108-454, Section 805, to assess awareness of VA benefits and services among four populations: veterans, active duty service members, National Guard and Reserve members activated under Title 10, and spouses and survivors of veterans. The NSV provides VA, Congress, stakeholders, and the public more accurate descriptions and assessments of the characteristics of the veteran population to evaluate existing programs and policies, to establish baseline measures before planning and implementing new programs and policies, and to monitor progress of programs and policies and their impacts on the population. The NSV will provide information to support VA policy, planning, and quality improvement decisions.

The NSV is of great importance to the VA because it is often the only source of information on veterans who are not currently using VA benefits and services. The NSV provides the VA with an opportunity to gauge future use so that it can plan more effectively for resource allocation. Since the data are so important to the VA it will be conducting a Pilot Study to determine the best sampling scheme and data collection methodology for the main study.

Among the goals of the Pilot Study are to assess the following:

• The response rates for the two-stage sampling approach.

• The coverage of veterans of the two-stage sampling approach

• The best methodologies to implement the two-stage sampling approach

A. JUSTIFICATION

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

Under Section 527, Title 38 U.S. Code, the Secretary of VA is authorized to gather data for the purposes of planning and evaluating programs. The VA is responsible for providing an array of assistance to more than 20 million veterans. As such, one of the VA’s many goals is to monitor and improve veteran health and well-being. The NSV will address the requirements of P.L. 108-454, section 805, to assess awareness of veterans’ benefits and services.

Because of the infrequent administration of the NSV, the data from any single administration are the only source of information on veterans who are not currently receiving benefits or services from the VA. The NSV is currently the only tool for reaching this population, and the NSV is the only existing tool that will allow for direct comparisons between veterans using VA services and those who are not. The NSV is also the only existing tool that can evaluate the VA outreach efforts and assess benefit awareness among veterans. Since the data from the NSV are so important, and because the NSV is conducted infrequently, it is important that we minimize the total survey error associated with the study design. It is for these reasons that the VA has planned to conduct a Pilot Study to inform the design of the NSV.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The VA will be the primary user of the results from the Pilot Study. The VA will use the results to inform the design of the NSV. At a minimum, the findings will be used to inform:

• Response rates. How will the response rate vary by households that are in the different sampling strata (e.g., matched to VA files; matched to DoD files; do not match to any files)?

• The coverage and response by veterans to the screening survey. How many veterans will respond to the survey? Does this allow for adequate coverage of the veteran population?

• Estimates for important subgroups? How many surveys are returned for important subgroups of interest for the survey (e.g., different eras; non-users of VA services)?

• The effect of allowing respondents to fill out the screening survey on the Web. How many households will respond using the Web, when given the option to do so?

• The effect of requesting contact information on the screener. If we request a veteran’s name in the screener, does this have a negative impact on the return rate of the screener instrument?

• The effect of different questionnaire items that ask about the status of veterans? What is the best approach to enumerate veterans within the household?

• The performance of selected questions on the extended interview? The abbreviated extended instrument will include a small selection of items that may be included in the main study.

• Survey completion procedures. Based on a telephone debriefing of a sample of respondents, we will assess how the selection procedures worked, as well as understand the process that respondents used to fill out the screeners and extended surveys.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

Since the sampling frame for this residential survey will be address-based, the NSV Pilot Study will utilize mail as the initial mode of communication with the household. In the design of the NSV Pilot Study, the VA will include a split ballot experiment on web-based response. Half of the sample will receive an initial letter asking an adult household member to complete a screener questionnaire via the web. Non-responding households will receive a paper questionnaire as followup (these households will retain the ability to complete the questionnaire on the web). In contrast, the second half of the sample will first receive a paper screener with no mention of the web. Non-respondents in this group (paper) will receive a second paper screener and cover letter in this wave 2 mailing.

The former method is certainly more cost effective and (we theorize) reduces paper by encouraging more respondents to respond electronically. However, there is concern that some respondents may get frustrated if they are unable or unwilling to fill out the survey on the web. If this is true, then offering the paper screener initially, then offering non-respondents the web as an option may yield a higher response rate. The Pilot Study will provide important feedback on these questions about response efficiency and response rate.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

Currently, there are no national surveys that address the full array of benefits and services that the VA makes available to the veteran population. In addition, there are no data collection efforts that respond to Public Law 108-454, section 805; the NSV will be used to respond to this Congressional legislation. Finally, the NSV Pilot Study is the only data collection effort that the VA has planned to help inform the design of the 2009 NSV.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize burden.

The sample for the NSV Pilot Survey will be U.S. residential addresses. No small businesses will be surveyed.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The NSV is conducted periodically. The data from the last NSV (completed in 2001) are greatly out of date. Not only are the 2001 data out of date, they miss entirely an important subpopulation: recent veterans of Operation Iraqi Freedom (OIF) and Operation Enduring Freedom (OEF). Finally, the Public Law (P.L. 108-454, section 805) requires that the VA conduct a national survey of veterans.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

• Requiring respondents to report information to the agency more often than quarterly;

• Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• Requiring respondents to submit more than an original and two copies of any document;

• Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;

• In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

• Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• Requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

There are no special circumstances.

8a. Part A: If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

The Department notice was published in the Federal Register on October 10, 2008, Volume 73, Number 198, Page 60406. There were no comments received in response to this notice.

8b. Part B: Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported. Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years - even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

Outside consultation From Westat:

• David Cantor, Senior Methodologist, 301-294-2080

• Bob Fay, Senior Statistician, 240-314-2318

• Pamela Giambo, Senior Study Director, 240-453-2981

• Michele Harmon, Senior Study Director, 301-294-3814

• John Helmick, NSV Project Director, 301-294-2010

• Wayne Hintze, NSV Associate Project Director, 301-517-4022

VA staff who participated in the design include:

• Maribel Aponte, NSV COTR/Project Manager, Office of Policy and Planning, 202-461-5790

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

The VA will not offer payment or gifts to respondents as incentives to complete the screener or extended questionnaires.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

The National Survey for Veterans is being conducted for the Department of Veterans Affairs (VA) to assess future program and benefits.  Participation in this effort is strictly voluntary and all responses will be kept confidential and used only for the purpose of this study.

All collected information is protected by the Privacy Act of 1974 (5 U.S.C. 522a) and the VA Claims Confidentiality Statute (38 U.S.C. § 5701) as implemented by 38 CFR § 1.526(a), 38 CFR § 1.576(b).  Additionally, 38 U.S.C. 7332 places restrictions on the release of information relating to drug abuse, alcoholism or alcohol abuse, sickle cell anemia, or infection with the human immunodeficiency virus.  System of records notice “Veterans, Service Members, Family Members, and VA Beneficiary Survey Records-VA” (43VA008) outlines VA’s data security measures for this effort.

Any disclosure of information involves the release of statistical data and other non-identifying data for the improvement of VA benefits processing system and for associated administrative purposes.  If you have comments regarding this burden estimate or any aspects of this collection of information, please contact the VA Clearance Officer (005R1B), address: 810 Vermont Ave. NW. Washington, DC. 20420.

All identifying information will be protected and masked with a pre-coded identification number. Only the survey vendor will have access to the identities associated with each number. The survey vendor protects (and will continue to protect) the Web survey application with a password and identification number. Sampled households can access the Web survey ONLY with the password and ID assigned to them.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

The screening survey asks households to indicate whether a veteran resides at the address. The extended Pilot Study survey will ask standard demographic questions (e.g., age. gender, race, income, etc.) along with a sample of questions planned for the main NSV, such as awareness of benefits and services, health care usage, etc. The Pilot Study survey does not contain questions considered personally sensitive or threatening.

12. Provide estimates of the hour burden of the collection of information. The statement should:

• Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

• If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

• Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 13.

The screener survey (Attachment 1 [Screening Survey (Version 1)], and Attachment 2 [Screening Survey (Version 2)]) includes a very limited set of questions. We estimate that the screening survey will take no more than three minutes to complete. We estimate that there will be 3,627 responses to the screener, for a total burden of 181 hours [(3,627*3)/60].

Of the 3,627 screener responses, we estimate that 43.1%, or 1,566 of the responding households, will include veterans. We expect that there will be 1.042 veterans per responding veteran household, for a total of 1,632 (1,566 x 1.042) screened veterans. Based on an assumption of a 70% response rate to the extended survey, the total number of extended-level respondents (Veterans) would be 1,142 (1,632 x .70). We estimate that the Pilot extended survey will take no more than 10 minutes, for a total burden of 190 hours [(1,142 x 10) / 60].

Once the Pilot data collection is complete, we will select a sample of respondents and non-respondents and conduct a debriefing interview to help assess the procedures and the instruments, particularly the screener. We estimate that we will debrief 100 respondents and 20 no responding households; the debriefing interview will take about 10 minutes to complete.

Table 1 shows how many respondents are estimated to submit each of the survey instruments, as well as corresponding minutes and cost burdens. The total cost per respondent ($1.85) is calculated by dividing the total cost of all interviews ($9,024) by the total number of respondents (4,889).

Table 1. Time and Cost Burden

Survey Type	Respondents	Estimated Minutes/ Respondent	Estimated Cost/ Respondent	Total Cost of all Interviews	Total Cost/Hour	Total Burden Hrs
Screener	3,627	3	$1.15	$4,177	$23.08	181
Extended	1,142	10	$3.84	$4,385	$23.08	190
Debriefing	120	10	$3.85	$462	$23.08	20
Total	4,889	4.8	$1.85	$9,024	$23.08	391

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).

• The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services component. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life of capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling and testing equipment; and record storage facilities.

• If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10), utilize the 60-day pre-OMB submission public comment process and use existing economic or regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate.

• Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices.

There is no capital cost to respondents.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

The overall contract with the survey vendor (for design and implementation of the NSV) is a Firm Fixed Price contract. The survey vendor estimates that its cost for conducting the Pilot Study will be $128,874.

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I.

This is a request for a new data collection.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

The substantive results from the NSV Pilot Study will not be published or tabulated for public consumption. The results will be used to inform the design of the main NSV to be conducted during summer 2009. Any publication of results would only involve the survey methodology or experimental treatments, and would not include responses provided by the respondents. These results will be described in an Executive Summary to be included in the OMB clearance materials for the main NSV survey and will also appear in final reports covering the main data collection efforts.

See the attached MS Project schedule for a timeline of the pilot study activities. Task 43 of the attached schedule is “Analyze Pilot Data”. The remainder of this section presents our data analysis plan for the NSV pilot survey.

Data Analysis Plan

Objectives

• Estimate sample-design parameters

• Assess data quality

• Analyze screener-data-collection experiments

Data sources

To evaluate the NSV pilot survey, we will analyze the following data:

• Survey-operations data: This includes the number of letters mailed, number of screener letters returned as undeliverable, the number of completed screener questionnaires, and the number of completed extended questionnaires. The survey-operations data are disaggregated by stratum and treatment combination. Table 2 specifies the recording frequency and some mathematical notation for these data.

• Questionnaire data: Veteran-status information and demographic data on the returned questionnaires will be used to disaggregate the survey-operations data. Also, the returned questionnaires will be reviewed to check for missing information and unusable information.

Table 2. Survey-operations data

Symbol	Description	Updating frequency
mi	Number of letters mailed/emailed i=1 Screeners, 1st i=2 Screeners, 2nd i=3 Extended, 1st i=4 Extended, 2nd	Weekly Weekly
u	Number of undeliverable screeners
n(s)	Number of screeners returned	Weekly
	Number of screeners returned by veteran households	Weekly
v (s)	Number of veterans in screened households	Weekly
v (e)	Number of extended questionnaires completed by veterans	Weekly

Analysis

Estimation of sample-design parameters

The following is a prediction equation for the number of completed extended questionnaires by veterans in a particular stratum:

,

where

m₁ = the stratum’s number of mailed screener questionnaires,

d = the stratum’s delivery rate for screener questionnaires,

p_v = the stratum’s prevalence rate for veteran households,

= the stratum’s screener response rate for veteran households,

= the average number of veterans per screened household containing at least one veteran, and

= the stratum’s extended response rate for veterans.

The corresponding yield prediction equations for spouses of veterans and spouses of deceased veterans are similar, except that the subscript v for veterans is replace with subscript s for spouses or subscript w for widows or widowers.

For the pilot survey, we assumed that d=0.90, =0.50, =0.70, =1.04. and p_v=0.80, 0.80, and 0.158 for strata 1, 2, and 3, respectively. Data from the pilot survey will be used to revise our estimates of the sample design parameters by stratum in the yield prediction equations. Because whether or not a household contains a veteran is known only for screener respondents, the pilot data does not permit us to separately estimate the veteran prevalence rate, p_v, and the veteran-household response rate, , by stratum. This is not necessary, however, because by letting

denote the screener yield rate for veterans in a particular stratum, the prediction equation for veterans becomes

.

This form of the prediction equation contains design parameters that can all be estimated from the pilot-survey data, via the following estimation equations:

The symbols on the right-hand sides of these equations are defined in Table 2. The pilot-survey data also allows us to estimate screener yield rates by stratum for spouses of veterans and for spouses of deceased veterans. The pilot-survey data does not permit us to estimate the corresponding extended-questionnaire response rates for spouses of veterans and deceased veterans, however, because for the pilot survey extended questionnaires will be mailed only to veterans.

Assessment of data quality

In addition to reviewing the returned questionnaire to check for missing data and unusable information, we will also use the plot data to calculate a data quality measure that assesses both frame coverage and the completeness and accuracy of responses about veteran status. We will calculate effective coverage rates, which are relative weighted counts of the number of veterans that are enumerated at each stage of the survey process. The analysis will compute the weighted estimate of the number of veterans that are enumerated on the screener and divide this by the number of veterans in the population according to an external source, for the entire veteran population and for subpopulations defined by demographic categories. A similar calculation will be completed for the returned extended questionnaires. These calculations will also be done for specific population groups. Decomposing the estimates by the screener and the extended returns will inform how the process is working for each of the survey stages. Examining these rates by population groups will assess the extent there is underenumeration for particular populations.

The denominators of the calculated effective coverage rates will be based on projections in the Veteran Administration’s VetPop2007 database. These projections are for September 30 of each year through 2036. We will calculate the needed denominators by linearly interpolating between the projections for September 30, 2008, and September 30, 2009, to the midpoint of the pilot survey’s 85-day data collection period. The extended questionnaire’s demographic questions and the disaggregations available in VetPop2007 permit us to calculate effective coverage rates by race and ethnicity, gender, age categories, and military-service eras.

These rates will be assessed against the initial assumptions of the design, which included a 50% response rate for veterans at the screener and a 70% response rate at the extended interview. The standard error of this rate will be approximately 1.55%, yielding a confidence interval of approximately +/- 3.1%.¹ The extent there is significant under-representation of important population subgroups will also be examined using a similar calculation. This calculation assumes that data will initially be aggregated over all of the experimental treatments. There will be larger standard errors if it is necessary to compute this rate within a particular experimental factor. For example, if it is important to concentrate on just those that were sent a paper questionnaire at the screener, the standard error of the effective coverage rate for veterans at the screener will be approximately 2.1%.

As describe at the end of Question 3, Part B, qualitative data on the process used to fill out the questionnaire will be collected by interviewing approximately 100 households (80 respondents, 20 non-respondents). The interviews with the respondents will ask how the survey came to their attention, how they determined who should fill out the survey, whether the correct individuals were selected for the survey and actually filled out the second questionnaire. The non-respondents will be asked on the reasons why they did not respond.

Analysis of screener-data-collection experiments

The primary research questions that will be addressed in this analysis are described in Part B, Question 4. The experimental measures for each of the screener data-collection experiments will be the following two weighted rates:

, and

.

To the test main-effects of each experiment, we will use SAS’s PROC SURVEY MEANS to calculate the standard errors for each experimental measure, and then we will use two-sample t-tests to determine if the main-effect differences are significantly different from zero. These tests for main-effect differences will be performed at the population level and within domains defined by race and ethnicity, gender, age categories, and military-service eras. We will test for interactions between the experimental treatments by using SAS’s PROC SURVEYREG to determine if the appropriate contrasts of means by treatment-combination are significantly different from zero. The response to question 3, in Part B, provides the power of the tests for the two sample t-tests.

Usage of Analysis Results

The results from the analysis described above will be used to plan the main NSV survey. The findings from the screener-data-collection experiments will determine if a web-data-collection option will be offered, if there will be an insert to encourage returning the screener, and if there will be screener questions about active duty to determine veteran status. The calculated sample-design parameters will be compared to the values we assumed for these parameters in the budgeting for the NSV data collection. Table 3 lists the NSV sample-design parameters and data quality measures, indicating the assumed value that we used in the initial planning of the NSV and whether or not an estimated value will be calculated from NSV pilot data.

Table 3. NSV survey metrics

Usage	Metric	Assumed value	Pilot to estimate?	Comparison value
Develop data collection budget	Deliverable rate by stratum	0.90	Yes	Revised cost projection not greater than available budget
	Veteran-household prevalence rate by stratum	0.80 Stratum 1 0.80 Stratum 2 0.158 Stratum 3
	Veteran-household screener response rate by stratum	0.50
	Veteran-household screener yield rate by stratum	0.40 Stratum 1 0.40 Stratum 2 0.08 Stratum 3	Yes Yes Yes
	Average number of veterans in screened households containing at least one veteran, by stratum	1.04	Yes
	Veteran extended response rate by stratum	0.70	Yes
Assess data quality	Veteran effective coverage rate: Screener Extended	0.472 0.332	Yes Yes	Not significantly less than: 0.20 0.12

The pilot survey’s tasks can be organized into the four steps of plan, do, check, and act. Deming refers to these four steps as the Shewart cycle for process improvement. Though some process-improvement practitioners argue that the Shewart cycle should be carried out only if a process is in a state of statistical control; the Shewart-cycle terminology is useful in discussing the usage of the analysis results from the pilot survey. This OMB clearance package presents the results of the plan step. Our fielding of the NSV pilot and tabulating the resulting data will be the do step. In the check step, one measures the process and compares the results against expectations. If the results are not what was expected, then in the act step one decides to change the process and the plan-do-check-act cycle repeats.

From the viewpoint of the check step of the Shewart cycle, we have the following expectations for the NSV pilot:

(1) All pilot study tasks will be completed by the due dates shown in the MS Project schedule sent on March 23rd.

(2) The projected data collection costs based on the pilot survey’s estimated sample-design parameters and results from the screener data collection experiments for the two-stage address-sample approach satisfying NSV precision requirements will be less than the amount budgeted.

(3) The data quality resulting from the two stage address-sample approach will approximate a landline RDD survey.

To check results against Expectation (1), the timeliness of completed tasks, we will use the capability of Microsoft Project to track “variances” between planned and actual completion dates. If there are variances indicating missed due dates, then the act step will be to determine if tasks need to be redesigned or if changes are needed to the timeline for the full-scale NSV

To check results against Expectation (2), the control of data collection costs, we will first make use of the sample-design parameters estimated from the pilot survey and the results from the screener data collection experiments to optimize the sample design for the main survey. This will involve determining which treatment options to use in the full-scale survey and recalculating optimum stratum allocations and second-phase sample rates. The projection of the data collection costs for the full-scale survey will then be revised. If the revised projection exceeds the amount budgeted for data collection, then the act step will be to explore changes to the budget, procedures, and /or precision requirements.

To check results against Expectation (3), the quality of collected data relative to landline RDD, we will compare calculated effective coverage rates against the values shown in rightmost column of Table 3. These comparison values assume that the coverage of landline RDD for the veteran population is 80%, the screener household response rate for veteran household is 25%, and the extended response rate for veterans is 60%. In addition, we will examine differentials in the coverage rate by important analytic groups, such as those in younger age groups, Veterans in different eras and other groups of interest. If the effective coverage rates are significantly less than the values in the rightmost column of Table 3, then the act step will be to explore changes in data collection procedures.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

We propose including an expiration date of July 2009 for the NSV Pilot Study.

18. Explain each exception to the certification statement identified in Item 19, "Certification for Paperwork Reduction Act Submissions," of OMB Form 83-I.

There are no exceptions to the certification statement.

1 This was calculated based on the assumption that there are approximately 2000 veterans in households for the Pilot (10,000 x 20%). A 50% response rate would then have a standard error of 1.55%, which takes into account the disproportionate sampling rates across strata.

2 Assumed veteran effective coverage rates based on assumed coverage rate of 0.95 for the address sampling frame, assumed veteran-household response rate of 0.50, and assumed veteran extended response rate of 0.70.

- 2 -