0596-0236-ssb_030524

Supporting Statement - Part B for OMB 0596-0236 EXTENSION

Interagency Generic Clearance for Federal Land Management Agencies Collaborative Visitor Feedback Surveys on Recreation and Transportation Related Programs and Systems

B. STATISTICAL METHODS

Data collection methods and procedures, while varying across projects, be limited to the list of known methods approved as part of this generic information collection clearance.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The potential respondent universe will consist of FLMA customers, who include visitors or users, potential visitors or users, recreationists, and members of stakeholder communities, potentially including business owners, in or near FLMA units. All study proposals submitted for approval under this generic clearance will include a description of the information collection’s respondent universe as well as actual response rate achieved in cases where the collection is a continuation of prior work.

Sampling methods will be based on the needs of each study. If gathering data that is statistically representative of a study population is a study goal, random selection methods, including the use of the last-birthday method to select a random individual from a group of individuals traveling together, will be employed to obtain data that are representative of the study population. In addition, information collections may involve samples of self-selected customers, convenience samples, and quota samples, with respondents selected either to cover a broad range of customers or to include specific characteristics related to certain products or services. Convenience samples will primarily be confined to studies using a comment card. Data collected via convenience sample will not be used for statistical inference or generalized to a broader population. The specific sampling design for each individual information collection and the method for soliciting participation will be described fully in each individual collection request made under this generic clearance.

Based on past applications and discussions with FLMAs who may use this generic collection in the coming years, we estimate that for the three-year period there will be approximately 54,000 survey respondents. Based on results of previous information collections at various FLMAs nationwide, it is anticipated that response rates will be at or above levels needed to obtain statistically viable results. Collections will incorporate best practices to maximize response rates. In all surveys, participation will be voluntary and time commitments will generally not exceed 21 minutes.

In addition to surveys, over three years we estimate 2,160 respondents participating by other means such as focus groups, interviews, or comment cards. In this data collection mode, participation may require up to 60 for interviews and 90 minutes for focus groups. Only qualitative methods are appropriate for assessing information derived from these modes, and while questions with numeric response scales may be used, the results are not statistically valid or generalizable to a population.

2. Describe the procedures for the collection of information including:

* Statistical methodology for stratification and sample selection,

* Estimation procedure,

* Degree of accuracy needed for the purpose described in the justification,

* Unusual problems requiring specialized sampling procedures, and

* Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The FLMAs will use a variety of methods for these information collections, as described in Part A. For example, survey staff may intercept customers at FLMA sites and distribute or administer survey instruments for completion on-site. Alternatively, they may intercept customers as they are leaving the FLMA units and ask respondents to complete a survey online at a later time, if that is a more sensible approach for those collections. The approach will depend on the objectives of the individual study, resources available, and other constraints.

In addition to physical copies, information collection instruments may be electronically disseminated and/or posted on target pages of the respective FLMA’s web sites. In such cases, the FLMAs or their contractors may use commercial survey-specific software to automate collection of feedback. Electronic surveys may be accessed through the internet or a smartphone application. This method is likely to be combined with an in-person intercept, whereby visitors are intercepted at an FLMA site, their email information is obtained, and then they are sent a link to the online survey. Electronic surveys may also be combined with an address-based sampling approach, whereby potential respondents are sampled via their address and then invited to participate online.

Mail surveys, telephone scripts, personal interviews, focus groups with professional guidance and moderation, and social media platforms may also be used, and are described in more detail below:

Mail surveys: This method may be used in combination with an on-site intercept or with an address-based sampling approach that seeks to identify customers of one or more FLMAs. Address-based sampling approaches are being used with greater frequency—relative to telephone-based sampling frames—due to the better coverage of these sampling frames.

Telephone surveys: This method is most likely to be used for national or regional population surveys or for populations that are geographically concentrated. Telephone numbers (including cell-phone numbers) may be purchased for sampling purposes, and the survey may be conducted using Computer-Assisted Telephone Interviewing (CATI). It is possible that telephone numbers may also be acquired during an on-site intercept of customers. Depending on the target population, it is also possible that telephone lists may exist (e.g., a recreation club’s database of members).

Interviews: This method will be used if qualitative, in-depth information needs to be collected. Customers may be intercepted on-site or may be contacted by mail, email, or phone call and asked to participate in a brief interview (either then or later, either in-person or virtually). The target population for the interviews will depend on the objectives of the study, and the intercept method will be tailored accordingly. For example, if qualitative interviews of hikers are being conducted, the hikers might be intercepted and interviewed at the end of their hike. It is possible that interviews may be combined with other methods. For example, interviewees could be identified via on-line or on-site the survey that includes a question asking respondents if they would be willing to participate in a brief interview to obtain more-in-depth information on their responses.

Focus Groups: Focus group participants may be initially contacted on-site (the location of the intercept will depend on who the target population is) or via mail, email, or phone call and asked to participate in an on-site focus group later. In addition, it is possible that focus groups may be conducted in locations across the U.S., if the objectives are to obtain in-depth, qualitative information from a broadly dispersed customer group. Individual information collections will specify the methods for how focus group participants will be identified.

Social Media: Under certain circumstances, it may be appropriate to engage with potential respondents via social media platforms (e.g., Facebook). When social media is used, the information collection request will specify how the study will ensure collection of a statistically valid and reliable sample or, if the sample will effectively represent a convenience sample, state that the information will not be used for statistical inference or generalizing to a broader population. Existing guidance, such as “Social Media, Web-Based Interactive Technologies, and the Paperwork Reduction Act (April 2010),” will inform the study methods used in individual ICs.

All submissions under this generic clearance will fully describe the methods and will be evaluated to ensure consistency with the intent, requirements, and boundaries of the anticipated generic clearance and to ensure that information-collection procedures are appropriate for the intended uses of the data. All information collection instruments will be designed and deployed based upon accepted statistical practices and sampling methodologies, where appropriate, to obtain reliable and statistically valid data that are representative of the target populations, account for non-response bias, and achieve response rates at or above levels needed to obtain statistically useful results. Proposed collection instruments and procedures must comply with OMB guidance as described in the OMB publication Guidance on Agency Survey and Statistical Information Collections (January 2006).

Data collection methods and procedures will vary by information collection and the specifics will be provided with each collection request, including:

• respondent universe

• the sampling plan and sampling procedure (including stratification and selection methods for individual respondents)

• how the instrument will be administered to respondents

• expected response rate and confidence intervals

• strategies for maximizing response rate and dealing with potential non-response bias

• planned analyses

A description of any pre-testing and peer review of the methods and instrument will be provided with each information collection. In addition, all submissions under this generic clearance process will describe how data will be presented to managers and any others that will use the results of the information collections, particularly in cases where response rates were lower than anticipated. In these cases, program managers will take steps to ensure that the results will not be used inappropriately, for example, generalized outside the population of interest. Explanations will be provided with data presentations and reports so that users of the data understand any possible biases associated with the data.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Individual collections submitted under this generic collection will identify the best practices they are using to maximize response rates. For example, the scripts used to recruit participants should be designed to maximize response rates by emphasizing the importance of each respondent’s participation and by explaining how the information collection informs the improved management of their public lands. Respondents contacted in similar studies have often expressed gratitude and appreciation for the opportunity to participate in studies of this nature. FLMAs will also be sensitive to survey length to minimize respondent burden, and will use follow-up contacts as appropriate, to increase response rates.

Individual decisions to participate, or not, in information collections may affect the accuracy of the data. Best available practices to reduce, mitigate, and correct for self-selection and non-response biases will be used. All strategies for detecting and analyzing non-response bias will be included in the individual information collection submission package. For example, the survey administrators may record observable information about every customer contacted (e.g., group size, presence or absence of children in the group, time of contact) to be compared later against those agreeing to participate and/or general observations about the sample day conditions, such as day of week or weather. In addition, each individual declining to participate in the study may be asked a few brief questions (e.g., their state or country of residence, is this their first visit to the location, and did they have trouble finding parking at the location). FLMAs can then make comparisons between respondents and non-respondents to consider non-response bias.

If there is indication of potential non-response bias in the data, the potential effects of potential non-response bias on results and their interpretation will be discussed in reporting materials and additional statistical tests may be performed to assess whether there are non-response bias effects for key questions in the information collection (e.g., questions about visitor experience quality, mode choice, travel and parking issues). In cases where such bias exists, standard practices for weighting the data may be used to align important sample statistics with known population parameters. All procedures and results for the non-response bias analyses will be included in technical reports, and the likely effects of this bias, if any, on the interpretation of the data will be made clear to managers.

Measures will be taken to address item non-response for those who agree to participate but are hesitant to provide information for some specific items. For example, experienced survey administrators will highlight to respondents why the item is of importance (e.g., data on household income helps to assess how different FLMA customer groups perceive recreation offerings) and remind them that answers will not be associated with the participant. Ultimately, participation will be voluntary and a participant may refuse to address any specific item.

Measures to test and correct for non-response bias issues, coupled with the accuracy and statistical power associated with the projected sample size for each information collection instrument, are expected to result in levels of accuracy and reliability that are generally accepted as sufficient in peer-reviewed social science quantitative study findings. Interpretation of data will be made clear to all data users.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Pre-testing of new information collection instruments (or new applications) and procedures will be strongly encouraged to reduce respondent burden and maximize the quality of the information collected. If the number of pre-test respondents exceeds nine members of the public, the affected FLMAs will submit the pretest instruments for review under separate cover, as part of this generic clearance or other appropriate OMB clearance procedure. While most of the methods and questions submitted under this generic clearance have a long history of successful application and may not require pre-testing, FLMAs will be encouraged to pre-test all survey instruments and methods. This enables testing for respondent comprehension, the identification of sources of measurement error, and the estimation of burden hours. Ideally, participants in pre-tests should be drawn from the same respondent universe as the full sample. However, if this is not feasible, a similar respondent universe should be used.

5. Provide the names and telephone numbers of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The names and contact information of the responsible FLMAs and the principal investigator(s) who will collect and analyze the data will be included on all submission forms provided under this generic clearance. Each information collection project will be reviewed by social scientists and statisticians to ensure that the survey instruments and methods are designed to provide scientifically valid data. The FLMAs will include the names and contact information of persons consulted in the specific information collection requests submitted under this generic clearance.

This generic clearance, including the compendium of survey questions and collection methods, is the product of a collaborative effort among the FLMAs. Each of the partner agencies has actively participated in development of these materials since the initial approval. We have also coordinated with subject matter experts in FLMAs in preparing this extension. The USDA National Agricultural Statistics Service and four academic researchers (from Auburn University, North Carolina State University, Oregon State University, and Virginia Tech) who are experts in social science, have reviewed supplemental forms A and B, the compendium of questions, and the burden estimates submitted in this extension.

For FLMAs involved in any particular Information Collection using this generic, the lead agency will assign their Paperwork Reduction Act (PRA) officer to conduct an administrative review of the request and send the submission through its normal peer review channels on behalf of the participating agencies of that Information Collection. This review will ensure that all documentation is complete and sufficiently detailed to meet the requirements of the PRA and scientific validity. The package will then be sent to the Forest Service, where it will be reviewed by the relevant experts prior to submission to USDA and OMB.

Page 1 of 3