NCSES Responses to OMB Questions for 3145-0019

OMB questions Concerning the SED Clearance Package (3145-0019)

and NCSES Responses

OMB asked clarification on 3 items in its 5/10/2011 email to NSF. The NCSES response appears in italics following each item [NOTE: “NCSES” is substituted for “SRS” wherever SRS originally appeared in OMB’s Terms of Clearance statement.]

1. “I’ve now gone through all of the materials in ROCIS and am disappointed that none of the Terms of Clearance (TOC) are clearly addressed.  For your convenience, these TOC are copied here:”

“Previous terms of clearance remain in effect, as follows. OMB appreciates NCSES’s efforts to revisit its confidentiality language to bring more fully it into alignment with recent OMB guidance. In addition, OMB appreciates that NCSES is interested in approving its disability measures for this survey. NCSES should plan to submit the results of its findings from the revised disability question to OMB at its earliest convenience; as NCSES knows, OMB continues to encourage interagency efforts to bring disability questions into alignment across major surveys, and wishes to continue those discussions with NCSES.”

Response: From late 2009 to early 2010, NCSES held a series of internal meetings for the purposes of improving its disability measures and making the measures consistent across NCSES surveys and aligned with surveys fielded by other federal agencies. Attendees included survey managers in the Human Resources Statistics Program, analysts from the Science and Engineering Indicators Program, NCSES mathematical statisticians, and the NCSES Deputy Division Director. Disability items from the Washington Group (WG), American Community Survey (ACS), and Current Population Survey (CPS) were compared to disability items on NCSES surveys, and the strengths/weaknesses of different items were debated. NCSES staff was favorably impressed by the WG disability items but expressed concern that, as the WG items had not yet been fully tested, aligning NCSES items to WG items would be “trying to hit a moving target” and might require additional NCSES survey revisions in subsequent years. The consensus decision that emerged from the meetings was to make a slight revision to the SESTAT surveys (we added a response category without changing the format of the item), but to postpone revisions to the SED. A copy of the disability items used in the 2010 Survey of Doctorate Recipients (p.14) is attached.

NCSES will now use the SESTAT disability question on the 2012 SED to make it completely consistent with the SESTAT surveys. We have already discussed with our contractor the necessity of revising the format of the SED questionnaire to accommodate this change. NCSES will revise the relevant documents in the SED 2012-2013 OMB Clearance Package, including a revised version of the proposed 2012 SED questionnaire, and submit them to OMB as soon as possible.

Finally, NCSES is cautioned not to submit questionnaire or other changes in final documentation that have not been fully discussed with OMB in advance. This is not conducive to an efficient and timely conclusion of the review process. In addition, OMB looks forward to NCSES releasing results of its evaluation of the quality of data resulting from the addition of the two newly-added questions related to funding sources related to the American Recovery and Reinvestment Act of 2009.

Response: NCSES added two ARRA-related items to the 2010 SED instrument, one (item A7) concerning whether ARRA funds were used as financial support during graduate school, the other (item B6) addressing ARRA funding of the doctorate recipients’ postgraduation plans (i.e., postdoc or employment). A small NCSES-sponsored study (20 cognitive interviews) completed in March 2010 examined whether individuals are able to respond to ARRA support questions accurately (a copy of the technical report from this study is attached). The findings indicate that while most respondents answered the questions correctly, many were unsure of their responses, and both the survey item wording and the concept of “ARRA support” were noted as sources of uncertainty. The study’s authors also pointed out that as more time elapses between the ARRA and when respondents complete the survey, measurement error from respondent recall problems is likely to grow.

With this last point in mind, NCSES will delete ARRA items A7 and B6 from the 2012 SED instrument, and will revise the relevant OMB Clearance package documents accordingly. NCSES has not yet examined the quality of data collected by the ARRA items during the 2010 and 2011 SED survey cycles. Subject to resource constraints, in 2011-2012 NCSES will conduct an evaluation of the data quality of the ARRA items in accordance with the 4/28/2009 nonsubstantial change request memo to OMB.

2. “Is NSF proposing to conduct methodological work such as focus groups under THIS OMB control number?  That’s what it looks like it is proposing in about three places, including in the burden discussion.  If so, please be advised that these will likely require a 30 day notice since they are not specified herein.  It would seem preferential to exclude these plans from this package and to plan to conduct them under the Generic Clearance package.” 

Response: NCSES agrees, and has deleted mention of these plans from the revised Supporting Statement. Future methodological work on the SED will be conducted under the Generic Clearance package.

3. “Please clarify what is intended with question B4b which is highlighted in the questionnaire.  It looks like a pretty meaningless set of response categories as written.”

Response: To provide context for the requested revision to B4b, items B4, B4a, and B4b (from the 2011 SED) appear below.

B4. What best describes your post-graduation plans (within the next year)?

Mark (X) one

1 “Postdoc” or further training  GO TO B4a

2 Employment  GO TO B4b

B4a. What best describes the nature of your further training or study?

Mark (X) one

1 “Postdoc” fellowship

2 “Postdoc” research associateship

3

GO TO B5

Traineeship

4 Internship, clinical residency

5 Other Training - Specify

B4b. What best describes the nature of your employment?

Mark (X) one

1

SKIP TO B6

Non-“Postdoc” employment

2 Military service

3 Other Employment - Specify

Analysis of 2010 and 2011 responses to this item showed that a large proportion of respondents chose category 3, “Other Employment – Specify.” However, the verbatim information many respondents provided in the “Specify” box indicated that category 1 (Non-“Postdoc” employment) would have been a more appropriate response. It appeared the description for option 1 in SED 2011 – the “Non-Postdoc” prefix – may have confused many respondents.

Consequently, in the 2012 SED the “Non-Postdoc” prefix is deleted in the first response option in B4b; the item now appears as follows:

B4b. What best describes the nature of your employment?

Mark (X) one

1

SKIP TO B6

Employment (other than military service)

2 Military service

3 Other Employment - Specify

It is analytically useful to distinguish “Military service” from “Employment (other than military service).” A subsequent item on the SED collects data on the economic sector of the doctorate recipient’s principal employer in the coming year, and includes response categories for foreign government and federal government. The data from item B4b enable analysts to distinguish doctorate recipients entering military service from those taking government positions. The “Other Employment – Specify” response category in B4b is useful for collecting data from doctorate recipients pursuing atypical employment opportunities (e.g., self-employment). The verbatims from this response option are also useful for back-coding responses into the “Employment” option, if warranted. For example, 499 respondents in 2010 selected “Other Employment – Specify” in B4b, and wrote in some variation of “academic employment” as a verbatim; these responses were back-coded into the “Employment (other than military service)” response option.