NCSES' Response to OMB's Questions

NCSES’ Response to Brian A. Harris-Kojetin, OMB on Generic Clearance Request

1. Question on the expanding power analysis: What is the minimum detectable effect to justify the sample size?

The minimum detectable effect is 5% at 80% power, an alpha level of 0.05, and an overall response rate between 75% and 80%.  As seen in the table and graph below, we considered other configurations of institutions and ECDs per institution, but several factors moved us toward the current design, of which the 5% discernible effect is just one, albeit an important one. First, as the goal of the methodological pilot is to test our frame creation and contact strategies, it is imperative that we sample a wide variety of institutions. Second, for experimental validity, it is important that the response burden for each institution in the pilot reflect the likely burden in the future full-scale data collection. Third, for the experiment to be meaningful, we need enough ECDs to have a reasonable chance at detecting a difference in response rate within each experimental group. Sampling 80 institutions rather than 60 will ensure that we get sufficient variation across the institutional respondents, and sampling 40 ECDs per institution will ensure experimental validity. The overall sample size of 3,280 yields a minimum detectible difference close to 5% starting at an overall response rate of 75% and provides the best opportunity to achieve a meaningful and reasonable result, namely that if the difference in response rates in less than 5% the ECD contact strategies will be deemed equivalent.

Possible sampling designs	Selected	Alternate 1	Alternate 2	Alternate 3
Number of GSS + FFRDC Institutions	80	80	60	60
Number of ECDs per GSS + FFRDC Institution	40	30	40	30
Number of ECDs from the NIH IRP	80	60	80	60
Total Institutions	81	81	61	61
Total ECDs	3,280	2,460	2,480	1,860
Detectable Difference in Response Rates	5.0%	5.5%	5.5%	6.0%

2. Attachments A.8 – Initial and A.9: Whose email (HA or RTI) will be shown to respondents in the initial and final appeal e-mails from the institutional HA?

Both the initial and final appeal e-mails to the respondents (ECDs) will come from the institutional HA. So the ECDs will see the e-mail address from the institution.

3. Check on how non-respondents will be contacted in the final appeal from the institution HA. Will NSF be providing a list of the non-respondents to the institutions?

We considered providing the list of non-respondents to the institutions, but decided that due to confidentiality concerns, it would be better to ask each institution to send the final appeal message to each respondent. The final appeal e-mail will be revised such that the second sentence in the first paragraph (Attachment B.6 ECD Contact Final Appeal E-mail – HA Group) will be replaced with: “If you have already responded to this survey, thank you. If you have not yet responded, I urge you to do so now.”

The estimated stage 2 burden of one hour per HA or designee reflects half-an-hour each to send 40 emails for the initial notification and for the final appeal.  We believe this is a good estimate of the burden as we have developed a tool that will automatically generate each component of the email.

4. Check to make sure that all burden estimates are included in the total burden estimates.

All burden estimates have been included in the calculation of the total burden estimates of 1,754 as shown on Table 2 of the OMB cover memo.

5. We need to include the boilerplate language from the NCSES’ generic clearance authorization (OMB control number and expiration date) and its appropriate burden hour estimate. This must be included on the (1) participation agreement form (Attachment A.1), (2) list of data elements and description (Attachment A.4); and (3) web questionnaire (questionnaire specs).

Boilerplate language:

Pursuant to 5 CFR 1320.5(b), an agency may not conduct or sponsor, and a person is not required to respond to an information collection unless it displays a valid OMB control number.  The OMB control number for this collection is 3145-0174 (Exp. 5/31/2013).  Public reporting burden for this collection of information is estimated to average xx minutes/hours [NOTE: see attachments and below for specifics] per response, including the time for reviewing instructions.  Send comments regarding this burden estimate and any other aspect of this collection of information, including suggestions for reducing this burden, to:  Suzanne Plimpton, Reports Clearance Officer, Facilities and Operations Branch, Division of Administrative Services, National Science Foundation, Arlington, VA  22230.

Note for each of the attachments requiring this boilerplate language we indicated the estimate time required in the respective attachment and provided a brief detail as follow:

1. Attachment A.1 – Study Participation Form (page 5 and 7 of attachment A) with an estimate time of 1 hour.

2. Attachment A.4 – Description of Data Field (page 17 of attachment A) with an estimate time of 6 hours.

3. Attachment C – ECD Questionnaire Specs (page 2 of attachment C) with an estimate time of 30 minutes.

6. Race question and response option in Q6.o is not acceptable category and should be deleted.

The following were deleted and the corrections are reflected on page 63 of Attachment C – ECD Questionnaire Specs:

• Deleted: Item Q6.o (Item wordings: Some other race)

• Deleted: Item Q6.o_OTHER (Item wordings: Please specify:)