ACS Messaging -- Key Informant Interviews Justification Part B

SUPPORTING STATEMENT

U.S. Department of Commerce

U.S. Census Bureau

ACS Messaging Key Informant Interviews

OMB Control No. 0607-0760

This Supporting Statement provides additional information regarding the Census Bureau’s request for processing of the proposed information collection, ACS Messaging Key Informant Interviews. The numbered questions correspond to the order shown on the Office of Management and Budget Form 83-I, “Instructions for Completing OMB Form 83-I.”

B. Collections of Information Employing Statistical Methods

1. Universe and Respondent Selection

This qualitative study uses a series of quotas and a non-probabilistic sample design to understand the experience and perception of a variety of key informants and stakeholders.

All interviewed individuals must meet two basic criteria: 1) all stakeholders interviewed will hold a leadership position in their organization, department, division, or group, 2) stakeholders must either use data professionally or work for an organization that conducts outreach to a low-income, minority, or immigrant population. Previous Census analysis of decennial and ACS data has found that these factors are associated with lower self-response rates (See Bates 2008).

All stakeholders interviewed will be from one of five segments. These segments are designed to provide a broad range of perspectives into the survey, and serve as a framework for data analysis. The five segments are academic/research, state/local government, Tribal governments/ organizations, advocacy/community associations, and private sector business. Each of the five segments has additional sampling requirements to account for unique characteristics of that segment.

The n=100 interviews are distributed between the five segments to provide the broadest range of findings with a reasonable degree of confidence that additional interviews would not yield significantly different qualitative findings. The goal is to approach saturation, or the point at which no new information or themes are observed in the data with an additional interview (see Morse, 1995 and Guest, 2006).

These sample sizes are justified to identify themes among the perceptions and experiences of stakeholders. In relatively homogenous groups – in terms of their research and outreach efforts – six to twelve interviews are sufficient to identify these themes (Guest et al., 2006). Among the Academic/Research and State/Local Government, fifteen interviews will be conducted to provide sufficient coverage for those segments. For the Advocacy/Community Associations and Private Sector Business segments, where experience with demographic data and community outreach varies more significantly, additional interviews will be conducted to account for heterogeneity and to drill down into particular themes about outreach and data use.

The following table provides an overview of the distribution of interviews between the five segments:

Segment	Sample Size (n=)
Academic / Research	15
State / Local Government	15
Tribal Governments / Organizations	10
Advocacy / Community Associations	30
Private Sector Business	30

Like other qualitative stakeholder surveys, these interviews are not subjected to the same scrutiny as scientifically drawn samples where estimates are externally published. All reports will include a clear advisory that the data represent only the interviewed individuals, and findings will not be used to make nationally representative statements. These interviews, however, will be useful to incorporate perspectives on how to improve ACS data dissemination and outreach to key communities. Subsequent research, such as the ACS Messaging Refinement study, may iteratively build off these insights.

The following section discusses how each segment is operationalized individually.

Academic/Research (n=15 completed interviews)

These interviews will include college and university professors, researchers at think tanks and other research organizations, and experts at other national academic organizations (i.e., American Political Science Association, Association of American Geographers, and National Social Science Association).

The sample frame will focus on academic disciplines that use ACS data and research, particularly:

• Anthropology

• Area/cultural studies

• Demography

• Geography

• History

• Political science

• Public administration

• Sociology

• Statistics

• Survey research

• Market research

From this target sampling frame, approximately 375 Academic/Research sample records will be identified for initial contacts in order to ensure that n=15 interviews are completed. Interviews will be identified from commercially available leadership directories and may be supplemented by contractor-generated lists, where necessary. This sample ratio incorporates assumptions about this particular segment concerning the percentage of telephone completions that will be bad numbers, individuals who decline to be interviewed, and participants who do not pass the screening questions. For quality control, each contact will be manually inspected before fielding to ensure that the sample record is likely to result in a high-quality interview.

State/Local Government (n=15 completed interviews)

The sample frame will be developed in two stages. First, the 3,144 counties (including parishes and city-regions that are independent of counties) of the United States will be ranked by 2011 ACS self-response rates, which is the most current annual data available. The sample frame will be developed from state and local government agencies located in the lowest one-third of counties.

Once the target counties have been identified, the contractors will develop a sample list with specific government agencies. The sample will include employees from a variety of state-level and local-level governments, primarily focusing on municipal and county governments. Interviews will include government workers from a mix of jurisdiction sizes, including small (fewer than 25,000 residents), medium (25,000-500,000 residents), and large (more than 500,000 residents).

The sample will include a variety of job functions, especially:

• Planning/development

• Demographics/statistics

• Law enforcement

• Executive (i.e. elected officials)

From this target sampling frame, approximately 375 State/Local Government sample records will be identified for initial contacts in order to ensure that n=15 interviews are completed. These interviews will be sampled from commercially available leadership directories. If necessary, the contractor will supplement existing leadership directories with Internet-generated sample listings for local government officials in low ACS self- response counties. This sample ratio incorporates assumptions about this particular segment concerning the percentage of telephone completions that will be bad numbers, individuals who decline to be interviewed, and participants that do not pass the screening questions. For quality control, each contact will be manually inspected before fielding to ensure that the sample record is likely to result in a high-quality interview.

Tribal Governments/Organizations (n=10 completed interviews)

The sample for these interviews will include administrators or elected officials from national Tribal organizations or individual reservations, as well as officials from some of the 12 Bureau of Indian Affairs regions. Interviews will be sampled from approximately 310 Federally recognized reservations and tribal governments in the United States, based on the list of recognized tribes produced by the Bureau of Indian Affairs. In addition, state-recognized Tribal governments and organizations will be eligible as well. The sample will include tribes from across the country, ensuring regional diversity.

From this target sampling frame, approximately 300 Tribal Governments/ Organizations sample records will be identified for initial contacts in order to ensure that n=10 interviews are completed. This sample ratio incorporates assumptions about this particular segment concerning the percentage of telephone completions that will be bad numbers, individuals who decline to be interviewed, and participants that do not pass the screening questions. For quality control, each contact will be manually inspected before fielding to ensure that the sample record is likely to result in a high-quality interview.

Advocacy/Community Associations (n=30 completed interviews)

Similar to State/Local Governments, the sample frame will be developed in two stages. First, the 3,144 counties (and equivalents like parishes) of the United States will be ranked by ACS self-response rates from 2011, which is the most current annual data available. The sample frame will be developed from not-for-profit organizations located in the lowest one-third of counties.

Once the target counties have been identified, the contractors will develop a sample list with specific community / advocacy organizations. These organizations will be required to have a physical location or provide services in a low-response county. The contractor will then manually inspect the non-profit listings to remove organizations that are unlikely to address community development, poverty or low-income families, particular ethnic or racial groups, or immigrant/non-English speakers. Individuals who are interviewed will have a variety of roles, including communications, outreach, management, partnerships, programming and strategy.

It will include community organizations focused on at least one of the following areas, which will be verified during the interview:

• Community development

• Poverty or low-income families

• Ethnic or racial groups

• Immigrant or non-English speakers

From this target sampling frame, approximately 750 Advocacy/Community Association sample records will be identified for initial contacts in order to ensure that n=30 interviews are completed. This sample ratio incorporates assumptions about this particular segment concerning the percentage of telephone completions that will be bad numbers, individuals who decline to be interviewed, and participants who do not pass the screening questions. For quality control, each contact will be manually inspected before fielding to ensure that the sample record is likely to result in a high-quality interview.

Private Sector Business (n=30 completed interviews)

The goal of this segment is to interview individuals with a wide range of experiences at private sector businesses. These interviews will include a mix of business sizes: small (fewer than 100 employees), medium (100-500 employees), and large (more than 500 employees). Given the wide variety of job functions within the private sector, the sample will include employees in a variety of roles, including business and market development, marketing, management, and sales.

These interviews will focus on data use by data-intensive and consumer focused industries and sectors, particularly:

Industry	NAICS Code
Accommodations / Food service	72
Construction	23
Educational Services	61
Finance and Insurance	52
Health Care and Social Assistance	62
Information	51
Professional, Scientific, and Technical Services	54
Real Estate	53
Retail Trade	44, 45
Transportation /and Warehousing	48, 49
Utilities	22
Waste Management, and Remediation Services	56
Wholesale trade	42

Interviews will be conducted with individuals that have a range of job functions including business/market development, marketing, management, and sales.

From this target sampling frame, approximately 840 Private-Sector Business sample records will be identified for initial contacts in order to ensure that n=30 interviews are completed. Contact information will be identified from commercially-available leadership directories, supplemented by contractor-generated lists, where necessary.

This sample ratio incorporates assumptions about this particular segment concerning the percentage of telephone completions that will be bad numbers, individuals who decline to be interviewed, and participants that do not pass the screening questions. For quality control, each contact will be manually inspected before fielding to ensure that the sample record is likely to result in a high-quality interview.

2. Procedures for Collecting Information

Data will be collected using in-depth telephone interviews in order to identify overarching themes based on experiences and perceptions. The semi-structured, open-ended interviews are designed to capture the wide range of experiences that participants may have with data collection and high-interest communities. The interview will begin with 6 general screening questions and 1-3 segment-specific screening questions to ensure sample validity. Then the interview will have a series of open- and close-ended questions to prompt exploration of relevant experiences and perceptions. Close-ended questions are designed to classify key informants into different segments based on their occupation and the types of people they interact with on a daily basis. Open-ended questions, on the other hand, are typically most useful in elite interviews because key informants will want to provide contextual richness to their responses, allowing them to explore the subtleties and nuances of community outreach and data usage that close-ended questions do not allow (Aberbach, Chesney, and Rockman, 1975; Bernard Brown, 1969; Deutsch, Macridis, Edinger, and Merritt, 1967; Lane, 1962). The discussion guide for the interviews has been reviewed by Census Bureau researchers who routinely design telephone studies and has been tested internally by the contractor.

Interviewers will be instructed to probe further on particular questions. The interviews will be recorded and transcribed for analysis. All participants will be notified the interviews are being recorded.

All interviewers are trained in qualitative interviewing and have experience with in-depth telephone interviews of senior level audiences like doctors and C-suite executives. Prior to the interviews, the research team will review relevant information regarding the ACS and the Census Bureau, including material on the ACS, an overview of the data products that ACS offers, and a description of the project design and goals.

3. Methods to Maximize Response

The key informant interviews have incorporated several design features to maximize response. We will invite stakeholders to participate and then schedule interviews at a convenient time. Interviewers have experience with scheduling interviews with senior-level individuals. Up to six contact attempts will be made on weekdays during business hours. To maximize response, we will identify the research as being on behalf of the US Census Bureau.

4. Test of Procedures or Methods

As qualitative research, no formal statistical testing is planned besides counts, basic marginal percentages, and cross-tabulations. Transcripts from the interviews will be coded to identify major themes, specifically regarding data usage and community outreach. We will also systematically monitor data collection procedures in order to identify ways to reduce burden, streamline processing, and assure quality data.

Transcripts from the interviews will be coded to identify major themes, specifically regarding data usage and community outreach (the open-ended questions toward the end of the questionnaire). Since the demographics and occupations of our sample vary greatly, our coding strategy will employ three broad categories typically used to categorize elite interviews: manifest coding, latent coding, and global coding (Aberbach, Chesney and Rockman, 1975).

The first category is “manifest” coding, which measures direct responses to questions, as well as the number of direct responses. For example, we ask key informants about the best ways to educate people about ACS, and each direct response given (for example: email, regular mail, face-to-face communication, etc.) will fall under “manifest coding” (Aberbach, Chesney and Rockman, 1975). These codes will be the most diverse in our coding scheme, but they will be critical in categorizing the main points of our key informants.

We will also employ “latent” coding, which measures the structure of an individual’s response style (Aberbach, Chesney and Rockman, 1975). In other words, latent coding identifies responses that are not explicitly called for by the questions themselves. Responses can be coded as “negative” or “positive” references toward a particular question, indicating the respondent’s tone. With latent coding, we will be able to distinguish which outreach strategies, for example, are talked about in a negative or positive tone.

Finally, we will also employ a basic “global” code, which allows coders to form a broad contextual judgment from the entire interview transcript (Aberbach, Chesney and Rockman, 1975). We need to make sure key informants use a coherent framework in responding to the questions – we do not want an interview from someone who passes the screening questions but then can barely give substantive responses about outreach and data usage. In other words, coders will briefly summarize, with a specific code, the level of engagement given by the key informant during the interview.

Finally, codes for individual interviews will be combined, by segment, to enable the research team to draw comparisons across the segment as a whole.

To make sure our coding scheme is consistent across coders, we will be doing an inter-coder reliability test, where three or more members of the research team code one entire interview after being taught very specific coding instructions. If the inter-coder reliability is sufficiently high among the coders, then our coding structure will be considered reliable. The inter-coder reliability will be measured with a Kendall tau correlation coefficient, which is a standard measure of inter-coder reliability (Vavreck, 2009; Dorussen, Lenz, and Blavoukos, 2006; Aberbach, Chesney and Rockman, 1975). A reliable coding structure, via inter-coder reliability, means that our results can be replicated by any coder who is taught our instructions, and therefore, coding all our interviews can be completed by as little as one coder or multiple coders as long as they have been properly trained (Vavreck, 2009; Aberbach, Chesney and Rockman, 1975). In other words, the goal of the inter-coder reliability is to make sure our codes can be replicated with our instructions.

5. Contacts for Statistical Aspects and Data Collection

Consultants outside of the Census Bureau are listed below.

Kiera McCaffrey Reingold 202.559.4436 kmccaffrey@reingold.com	Sam Hagedorn Penn Schoen Berland 202.962.3056 rgreen@ps-b.com
Jack Benson Reingold 202.559.XXX jbenson@reingold.com	Robert Green Penn Schoen Berland 202.962.3049 rgreen@ps-b.com

9