Response to OMB questions

Generic clearance request for SED confidentiality pledge (3145-0019):

OMB questions and NCSES Responses

In an 8/20/2013 email to NSF/NCSES, OMB requested the following revisions to the research plan and protocols described in the generic clearance request.

“Please rethink the proposed incentives.  First, offering the same amount for focus groups versus individual interviews is not typical (our ‘going rate’ for interviews is lower than for focus groups and $55 exceeds that rate).  Second, the need to incentivize at all, and at what level, seems that it would vary by position.  For example, wouldn’t a university executive or staff person have a different expectation about this project being work-related than a graduate?  Please justify your revised proposal, particularly if any amount above the ‘going rate’ is requested.”

Response: The section on Incentive Payments (p.4) has been revised. Graduate deans and institution contacts will not be offered incentive payments to participate in the study; these individuals help administer the SED at their institutions, and will likely view participation in this research as part of the normal work-related activity. Different levels of payments will be used to incentivize doctorate recipients to participate in telephone interviews ($30) and institutional researchers to participate in electronic focus groups ($40). The $30 incentive payment for doctorate recipients is similar to what is used to recruit doctoral participants to the Survey of Doctorate Recipients.

From the protocols:

 

1. “The following question, ‘Do you think the extent of those confidentiality assurances is reasonable?  That is, do you think the confidentiality assurances cover too much ground or, conversely, they promise too little regarding the protection and usage of your data.’ could confound the strength of the confidentiality pledge with the level of detail provided in the lengthy paragraphs.  Please probe this issue.”

Response: In the second sentence above, the phrase “cover too much ground” is replaced by “ensure adequate protection,” which we believe will shift the respondent’s focus from the length and detail of the confidentiality statement to the purpose of the statement (see item 1a on pp. 6 and 9, and item 1b on p.12). A separate probe was added to elicit respondents’ perceptions of the length of the statement (see item 1b on pp. 6 and 9, and item 1c on p.12).

1a. “Related, the sentence in the pledge that describes the purpose of the SSN digits being about program evaluation sounds like it could be describing uses potentially unrelated to the SED, unlike the explanation offered in the brochure, which actually provides a different purpose (quality assurance) in addition to program evaluation, and also emphasizes that the data stay ‘inside the project.’  So we would like NCSES to probe on whether respondents see these two statements as conveying different meanings.” 

Response: The following probing question was added to the protocol section “Understanding of SED confidentiality assurances” (see item 1d on pp. 6 and 9, and item 1e on p.12): “When you compare the part of the confidentiality assurance that relates to the partial SSN to the part of the confidentiality brochure that relates to the collection of partial SSN, do you interpret those passages as saying the same or different things about the usage of the SSN data?”

1b. “Finally, it would be useful to probe which parts seem less essential since the literature suggests it is unlikely that respondents read that much detail and since the original pledge was already long prior to adding the AAU information.” 

Response: The following probing question was added to the protocol section “Understanding of SED confidentiality assurances” (see item 1c on pp. 6 and 9, and item 1d on p.12): “Do you think that any part of the current confidentiality pledge is not essential, and could be removed without reducing the overall level of protection provided? Which part or parts of the pledge do you think are not essential?”

2. “Related, OMB would prefer that a much shorter version of the pledge be tested as well, either in this round or in a subsequent round.  What are plans in this regard?”

Response: A shortened version of the pledge now appears as Attachment 5. The reactions of study participants to this shorter pledge are probed in the protocol section “Understanding of SED confidentiality assurances” (#4 on pp. 6-7 and 10; #5 on p. 13).

 

3. “The sentence: ‘Please note:  no direct identifiers that could personally identify any individual doctorate recipient would be shared by the institutions.’ seems to suggest that removing direct identifiers precludes re-identification, which of course is not true and is the whole reason that you are exploring how to maintain confidentiality.  Please suggest improved wording.”

Response: The sentence is re-worded as follows (on pp. 8, 10, and 14): “Please note: all direct personal identifiers would be removed before the data are shared by the institutions. Because re-identification may still be possible using the remaining information, data-sharing institutions will be constrained to use the information for statistical purposes only and to report findings in an aggregate manner that does not lead to personal identification.”

By deleting the phrase “that could personally identify any individual,” from the first sentence, we believe we have removed the suggestion that re-identification is not possible. We added the second sentence to address the risk of re-identification explicitly, and to place it within the context of other confidentiality protection procedures.