SSA_Generic Pretesting and Customer_5-13-2011

SUPPORTING STATEMENT A For:

A Generic Submission for Formative Research, Pre-testing, Stakeholder Measures and Advocate Forms at NCI

May 2011

Shannon K. Bell

Office of Advocacy Relations

Office of the Director

National Cancer Institute

31 Center Drive, Bldg. 31

Room 10A30, MSC 2580

Bethesda, Maryland 20892

Telephone: 301-451-3393

Fax: 301-480-7558

E-mail: bells@mail.nih.gov

Table of Contents

A. Justification 1

A.1 Circumstances Making the Collection of Information Necessary 1

A.2. Purpose and Use of the Information 5

A.3 Use of Improved Information Technology and Burden Reduction 7

A.4 Efforts to Identify Duplication and Use of Similar Information 9

A.5 Impact on Small Businesses or Other Small Entities 10

A.6 Consequences of Collecting the Information Less Frequently 10

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 10

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 12

A.9 Explanation of Any Payment or Gift to Respondents 13

A.10 Assurance of Confidentiality Provided to Respondents 15

A.11 Justification for Sensitive Questions 17

A.12 Estimates of Annualized Burden Hours And Costs 17

A.13 Estimates of Other Total Annual Cost Burden to Respondents and

Record Keepers 19

A.14 Annualized Cost to the Federal Government 19

A.15 Explanation for Program Changes or Adjustments 20

A.16 Plans for Tabulation and Publication and Project Time Schedule 20

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 22

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 22

List of Attachments

1. NCI Office of Advocacy Relations, Post-Activity Survey for Advocates

2. NCI Office of Advocacy Relations, Post-Activity Survey for Staff

3. Pre-Test for Advocates, CARRA Peer Review Workshop

4. Post-Test for Advocates, CARRA Peer Review Workshop

5. Reflections and Directions: Interviews and Focus Groups with CARRA Members and NCI Staff

6. Benefits of Advocacy Involvement in the Research Setting

7. Advocates in Research Working Group Membership

8. Memo from NIH Privacy Act Officer

9. Privacy Impact Assessment (PIA)

1. JUSTIFICATION

A.1. Circumstances Making the Collection of Information Necessary

The National Cancer Institute (NCI) is the Federal Government's principal agency for research, training, health information dissemination, and other efforts with respect to the cause, diagnosis, prevention, and treatment of cancer, rehabilitation from cancer, and the continuing care of cancer patients and the families of cancer patients [Section 410 of the Public Health Service Act (42 USC § 285)].

The NCI Office of Advocacy Relations (formerly the Office of Liaison Activities) was established in 1996 in order to promote the Institute’s mission and support its programs. The Office of Advocacy Relations (OAR) is NCI’s liaison to patient advocacy organizations, individual patient advocates, and professional societies concerned about cancer. The OAR disseminates cancer-related information to these stakeholders, seeks their input and feedback, and facilitates collaboration between the Institute and these external partners to advance NCI’s authorized programs [Section 412 of the Public Health Service Act (42 USC § 285a-1)]: “The Director of the Institute shall establish and support demonstration, education, and other programs for the detection, diagnosis, prevention, and treatment of cancer and for rehabilitation and counseling respecting cancer.”

The OAR works with the internal NCI and NIH communities to identify opportunities for patient advocates to participate in NCI and NIH activities. For example, patient advocates participate as volunteers in peer review of grant applications and as members of advisory boards and committees. Once an opportunity is identified, the OAR then works with the external advocacy community to identify advocates who have the appropriate experience for that particular activity. Although most patient advocates participate in many different kinds of advocacy (patient support, fundraising, lobbying, etc.), OAR seeks to identify advocates who work specifically in research advocacy. Given NCI’s research mission, advocates who do not have experience in research advocacy may not be appropriate participants in NCI activities.

The process of engaging advocates in NCI activities has several distinct steps that benefit from program evaluation research and standardized information collection. The steps in the process are:

• Recruitment of advocates

• Application process for advocates

• Training advocates and providing information

• Matching advocates to NCI activities

• Tracking and evaluating advocate engagement at NCI

• Promotion of advocate engagement at NCI

In the past the OAR has conducted research with both internal (NCI staff) and external (advocates) stakeholders and also collected information to enable the advocate engagement process. Past research has enabled OAR to monitor stakeholder trends, design and develop materials based on user feedback, assess the impact of activities, and improve service delivery. With assistance from the NCI Office of Science Planning and Assessment (OSPA) and an external contractor (WESTAT), OAR has focused on staff and external stakeholder KAP (knowledge, attitudes, and practices) and other characteristics to examine expectations, willingness to participate, post-activity satisfaction, individual performance, and changes in knowledge and behavior around engaging advocates in scientific activities.

The OAR has utilized questionnaires (written surveys), pre-post tests, interviews, and focus groups to gather information in the past. For example, post-activity surveys have been provided to research advocates¹ and NCI Staff to capture information on consumer involvement in NCI activities (Attachments 1 and 2). This information has helped OAR to adjust the matching process to better meet the needs of NCI staff and advocates. Pre- and post- training tests were utilized to measure changes in attitudes, beliefs, and knowledge as a result of participating in a peer review workshop (Attachments 3 and 4). This information helped OAR to fine-tune the training workshop to ensure that the information was presented in a way that was effective for the advocates. Through a series of interviews and focus groups², OAR gathered feedback on the perceptions of research advocates, and the perceived value of partnership with the advocacy community (Attachment 5). The information collected through these mechanisms has guided OAR’s strategic planning and helped improve its training and customer service.

In addition to collecting information for research, OAR has also requested information from advocates in order to match them appropriately to NCI activities. Due to the diversity of NCI activities that advocates participate in and the diversity of advocates’ experience and preferences, there are many variables to consider when matching advocates to NCI activities. Since no administrative form was available in the past, individual advocates have submitted resumes or biosketches describing their experiences in research advocacy. In a tedious and time-consuming process, OAR staff have extracted the necessary information from the resume or biosketch to input into the online database of individual advocates³. The resumes and biosketches submitted are not standardized or consistent in any way and often the advocate has to be contacted by OAR staff to provide additional information or clarification.

The lack of administrative forms to collect information from advocates in the past has also resulted in OAR needing to re-contact individual advocates to determine their interest and experience with new scientific topic areas that did not exist when the original resume or biosketch was submitted. New areas such as nanotechnology, proteomics, and genomics generate new NCI granting opportunities and other activities that involve advocates. It is imperative that OAR have information about advocates’ experiences with these new areas in order to appropriately match them to NCI activities and to provide appropriate training opportunities. OAR’s database of individual research advocates is often changed to accommodate this new information.

Past research conducted by OAR were sub-studies under an OMB generic clearance held by the Office of Communications and Education (OCE) under OMB #: 0925-0046. Past information collection activities provided general guidance to advocates but no online form or mechanism was provided for information collection. The OAR is now seeking its own OMB generic clearance to expand the capacity of NCI to submit generic sub-studies in order to conduct formative research on program initiatives, pre-test of educational materials, and gather information about stakeholder KAP and satisfaction. OAR is also seeking its own OMB generic clearance to allow OAR to create and update in real-time administrative forms used in the advocate involvement process.

A.2. Purpose and Use of the Information

The Office of Advocacy Relations intends to collect data from different stakeholder constituencies: NIH and NCI staff, NCI-funded scientists and staff from universities and cancer centers, research advocates and their advocacy organizations, and members of scientific and professional societies and their organizations concerned about cancer. The data anticipated to be collected can be organized into several categories. Respondents may provide information about:

• Contribution/Value Advocates and Advocacy Organizations Bring to the Research Process. The NCI Advocates in Research Working Group has identified several possible benefits to involving advocates and advocacy organizations in cancer research (Attachment 6). It is believed that these stakeholders help expedite clinical trial recruitment, improve the informed consent process, and strengthen human subject protections. The OAR intends to measure outcomes of stakeholder involvement in research activities.

• Expectations – In an effort to manage expectations, OAR intends to collect data on whether anticipated beliefs about advocacy performance, extent of participation, and program support were met. For example, “Did your overall contribution to the activity or project meet your expectations?”

• Facilitators and Barriers – The OAR strives to facilitate advocate involvement and reduce barriers to it. Items to be measured include program awareness, availability of adequate travel funds, ease of advocate request process, appropriate orientation and activity preparation, and timely follow-up.

• Performance – The OAR matches advocates to NCI activities based on a number of factors, including past performance in similar activities. Aspects of performance to be assessed include verbal contributions, written submissions, adhering to confidentiality requirements, and representing the patient perspective, etc.

• Program Recommendations – The OAR has greatly benefited from incorporating stakeholder feedback into efforts to improve program structure and processes. For example, the OAR is redesigning its advocate involvement program based on input from research advocates and other stakeholders who were members of the Advocates in Research Working Group (ARWG). Questions to elicit recommendations would include, “What would you like to see improved at OAR?”

• Recruitment – A diverse pool of qualified advocates must be recruited and matched to NCI-activities based on scientific advances and the subsequent needs of researchers and other staff. The OAR would support its recruitment efforts by identifying “What experiences and skill sets are required of advocates for this activity?” and also by asking advocates if they have these experiences and skills in the administrative forms. Due to natural attrition and changing scientific needs, OAR will need to continually recruit new advocates to participate in NCI activities.

• Application – The ARWG has recommended an open application process to bring new advocates into the engagement process at NCI. Therefore OAR anticipates using an administrative form to capture applicant information such as basic contact information, age, gender, race and ethnicity, work focus, employment status, health experiences, and research advocacy experience. This form will be continually updated and adjusted to meet NCI’s changing scientific needs in emerging scientific areas. Additional forms may be used for application to various advocate programs such as the NCI’s only all consumer advisory board – the NCI Director’s Consumer Liaison Group (DCLG) – and the NCI Translates annual conference.

• Satisfaction – Partly based on recommendations from the ARWG, the OAR is working to help foster an organizational atmosphere that values the contributions of research advocates. To help determine if this goal is being met, it’s important to measure staff satisfaction with the process of requesting advocates, the extent to which advocates abilities and experiences matched the activity, and the overall contribution of the advocate. Advocates and organizations will also provide feedback on their satisfaction working with the NCI and the information they receive from NCI.

• Training Needs – Advocate training is a necessity because science is rapidly advancing and research advocates need new knowledge and skills to fully contribute as equal partners in the research process. Training needs are identified with questions such as, “Are there any specific areas in which you believe advocates should be trained?”

• Knowledge and Behavioral Assessment – Based on the recommendations of the ARWG, while applying to participate in NCI activities, advocates may be asked about their knowledge of NCI, research advocacy, and their behavior in various scientific activity scenarios.

To date, information and data have been collected by OAR, or its evaluation contractor, and utilized by the OAR Director and Advocacy Program Managers to recruit and assess new advocates, manage expectations, successfully match advocate skills and experiences to specific NCI activities, develop effective training workshops, market the OAR, and demonstrate the value advocates bring to the research process. Not having this type of information in the future would delay or impair OAR’s ability to fulfill its identified mission and support the programs of the NCI.

A.3. Use of Information Technology and Burden Reduction

As computer technology has continued to improve and become more widespread, opportunities to pretest messages on the Internet using either Web site questionnaires or on-line focus groups with Internet users have increased. Improved technology in the collection and processing of data has the potential to reduce the time burden for respondents and data collectors. For example, respondents can access and respond to data collection requests at a time and place that is convenient to them, eliminating the need to travel for in-person or group interviews. Also, individual NCI advocates can update their resume information online at a time and place that is convenient to them, and as often as their experiences and interests change. This eliminates the need for Federal Government staff and contractors to contact advocates individually to determine their interests and experiences with new scientific topics. Wherever possible, NCI will make use of Web- or computer-based data collection methods. Transmission of data collection instruments and responses by electronic mail or facsimile will be utilized as appropriate. NCI anticipates that of the majority of data will be collected electronically. Privacy safeguards will be undertaken with assistance from the NCI Privacy Act Coordinator and the Information Security Office during the data collection process to mitigate any risks.

Computer-assisted telephone interviewing (CATI) will be utilized when geographic diversity is important and participants come from hard-to-recruit populations, such as Native Americans or physicians. Computer-assisted personal interviewing (CAPI) technology allows interviewers to ask questions of a respondent using a computer to enter data. ACASI software technology offers many advantages of CAPI technology, but removes the need to have a person administer an interview.

On-line surveys represent an especially convenient option for eliciting feedback from consumers of Web-based products. With online surveys developed by the NCI or approved contractors, respondents can easily submit feedback during or immediately after using a Web-based product. They also allow participation from international audiences with virtually no additional costs.

Technology now enables the conduct of focus groups to occur in a range of locations and with great variation and flexibility. Depending on factors such as geographic distribution and schedules of NCI’s customers and on the nature of the products and services under investigation, focus groups, when appropriate, may be implemented using a variety of technology-based formats including videoconferencing, internet/online conferencing (a sort of “chat room” in which a moderator intercepts and distributes e-mail transmissions), teleconferencing, and Advanced Strategies Lab (ASL).

Advanced Strategies Lab (ASL) is a qualitative research process that uses an online interactive discussion. In the 90-minute, online ASL sessions, participants receive instructions and are asked questions from the moderator over the phone on a conference call. Participants’ answers are typed into their designated dialogue box (which hides their identity from the other participants to ensure security). The facilitator then guides the group through online brainstorming, discussion, and assessment exercises.  The ASL’s software automatically formats and tabulates data as the session proceeds. Summaries of survey results are available seconds after respondents complete a question, and full verbatim reports are available within hours after a session.

A Privacy Impact Assessment (PIA) has been completed and published by HHS on February 22, 2011 (Attachment 9). The IT system name is “NIH NCI Office of Liaison Activities Database (OLA).”

A.4. Efforts to Identify Duplication and Use of Similar Information

The general areas in which information needs to be gathered (as described in A.2. above) are similar to questions asked previously of NCI stakeholders. However, because advocates are continually paired to new activities with different researchers, the measurement of these experiences for individual performance, met expectations, facilitators/barriers, and satisfaction do not impose unnecessary duplication. Currently, there is no similar information that would serve the agency’s need and purpose.

Literature searches, professional-to-professional discussions, use of data collections in the private sector, as well as other government surveys or pilot studies will be employed whenever possible to meet the needs of NCI. Additionally, NCI has an internal review process for surveys that will be used by this generic clearance to assess the quality of each survey prior to its use.  The NCI will provide direct oversight for any and all surveys conducted under this generic clearance to avoid duplication of effort and information collected.

The administrative forms will be used to collect demographic information and to assess research advocates’ experience and skills. This information is only available from the research advocates themselves, and cannot be found anywhere else. Having an administrative form to collect this information will lessen the burden hours required for advocates to submit information to become involved in NCI activities.

A.5. Impact on Small Businesses or Other Small Entities

Small businesses that are non-profits and independently-owned may be participants in this generic submission. The small businesses we may include are physicians, other health care providers, and highly specialized individuals for evaluation of NCI’s communication information and customer satisfaction materials. When small businesses are asked to complete an information collection, all efforts will be made to reduce their burden by using a short survey and interviewing more small businesses than larger ones.

A.6. Consequence of Collecting the Information Less Frequently

For the most part, formative research, pre-testing, and stakeholder satisfaction information will be collected only one time for each material tested or activity completed. Administrative forms about the experience of research advocates will be completed once initially for each advocate and then updated by the advocate when they believe it is appropriate. However, there may be occasion where a pre- and post-test to assess differences in knowledge, attitudes, or practices may be useful for a particular sub-study. Additionally, previous respondents may be contacted to participate in follow-up studies if they have originally granted consent for such and if the subsequent study uses that population.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

NCI recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, NCI anticipates that information collections under this clearance package will comply with 5 CFR 1320.5(d)(2) requirements with only two anticipated exceptions.

• When NCI asks for responses to surveys or administrative forms in less than 30 days, receipt of the survey or form is generally preceded by advance notification to respondents explaining the purpose of the survey or form, the approximate length of time that completion will take, and the voluntary nature of participation. All efforts are made to keep such surveys or forms short and focused.

• Because NCI’s pre-testing activities are often qualitative in nature, the results are not generalizable to the population at large or to the particular target audience under study. However, the nature of pre-testing is such that generalizability is not a critical feature; the emphasis is on obtaining timely, useful information that can be fed back into the development of new messages or materials or the revision of existing ones.

There are no other special circumstances.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult

Outside Agency

The 60-day Federal Register notice soliciting comments on OAR’s efforts prior to initial submission to OMB was published on March 15, 2011 (76 FR 14034). No public comments were received.

The questionnaires previously used by OAR were developed with consultation from a number of scientists and research advocates. External stakeholders, including some members of the Advocates in Research Working Group (e.g., Jane Perlmutter, PhD, Founder – Gemini Group, (734) 604-5263) will help craft future research instruments for the OAR (Attachment 7). Also, the OAR will also consult with several internal experts across NCI (e.g., Rick Moser, PhD, Psychologist, 301-496-0273 and Gordon Willis, PhD, Cognitive Psychologist, 301-534-6652- both from the Division of Cancer Control and Population Sciences). Additionally, a number of outside experts will be contracted to review the plans for program development research and evaluation of OAR activities.

A.9. Explanation of Any Payment or Gift to Respondents

It is possible that some information collection activities will entail incentives or gifts to respondents as a thank you for completing the information collection. Small amounts of money, a free meal or snack scheduled around the time of the pretest, for parking and/or transportation are most often used, particularly when recruiting hard-to-reach and minority respondents. Incentives may also be necessary in some sub-studies where highly specialized individuals are invited to participate in a survey, and will cover the cost of transportation and other types of local expenses.

Research has shown the advantages of providing a small incentive for improving response rates and decreasing item nonresponse, especially in mail and telephone surveys.⁴ Studies of participants in the original National Health and Nutrition Examination Surveys (NHANES) found that response rates for those told they would receive remuneration versus not were 82% and 70%, respectively.⁵ The National Survey of Family Growth conducted an experiment with incentives and found increased response rates, reduced interviewer labor (broken appointments and callbacks), and improved data quality.⁶ In the 1999 Observing Protein and Energy Nutrition study, remuneration was credited as contributing to high response and retention rates (OMB No. 0925-0465).⁷

Instances for offering a small incentive will be determined on a case-by-case basis (depending on the particular information collection design). The following are the kinds of situations for which respondents may receive a monetary incentive or given a gift as a thank you:

• Individuals who participate in in-person focus groups may receive an incentive (perhaps $25-75) to cover their time, transportation costs, and childcare expenses.

• Individuals who participate in website usability testing may receive an incentive ($25-75) to cover their time, transportation costs, and childcare expenses.

• Advocacy organizations who support the acquisition of data related to educational programs and products may be able to request and receive certain quantities of materials that exceed the limits usually established for those materials.

Circumstances, however, do not always require that an incentive be given; many audiences including the public, patients, cancer survivors, and other health professionals often participate gratis because of their interest or involvement in the topic, or as a professional courtesy. For example, in situations when the general public is completing an online survey, no remuneration will be involved unless influenced by other factors.

A.10. Assurance of Confidentiality Provided to Respondents

Information provided by respondents will be kept secure to the extent provided by law. This will be communicated to respondents by means of introductory letters, explanatory texts on the cover pages of question­naires, scripts read prior to focus groups or telephone interviews, and consent forms. There will be a separate consent form for each generic sub-study, and the consent form will be submitted to OMB for review with each sub-study submission. Respondents will also be advised of the following: the nature of the activity; the purpose and use of the data collected; NCI sponsorship; and the fact that participation is voluntary at all times. Because responses are voluntary, respondents will be assured that there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.

As a further guarantee of security, all presentation of data in reports will be in aggregate form, with no links to individuals preserved. Reports will be used only for research purposes and for the development of communica­tion messages and educational materi­als. Only NCI staff and contractor personnel conducting the information collection will have access to individual-level survey, interview, or focus group data. All project/contractor staff conducting the information collection will sign a confidentiality agreement, and all electronic and hard-copy data will be maintained securely throughout the information collection and data processing phases. While under review, electronic data will be stored in locked files on secured computers; hard-copy data will be maintained in secure building facilities in locked filing cabinets. Before any data are released for public use data sets, any identifying information will be stripped from each respondent’s record and the identifying information will be destroyed.

The NIH Privacy Act Officer has reviewed the work scope of this proposal and has determined that the Privacy Act is applicable to this data collection and is covered by NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” (Attachment 8). The NIH Privacy Act Officer will be asked to review the protocols of each sub-study under this generic clearance to ensure that NCI adheres to privacy requirements.

Personally identifiable information (PII) will be collected (see Section A.11 for further details). Although some PII will be collected, data will not be retrieved by personal identifiers unless the respondent voluntarily agrees to provide the information, so he/she can be contacted for follow-up. Instances could arise for activities that, for example, gather and retain respondent names and contact information. These data would be used to measure KAP information at one time and then at some point later (for example, to learn whether research advocates are using new skills or knowledge gained in training).

The Office of Human Subjects Research (OHSR) considers pre-testing efforts described in this proposal exempt from the “Regulations for the Protection of Human Subjects,” in accordance with paragraph (b)(3) of 45 CFR Sec. 46.101 (http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm), and as deemed by the OHSR. The OAR will work with OHSR in determining which sub-studies should be submitted for review/exemption.

A.11. Justification for Sensitive Questions

As mentioned in sections A.2. and A.10. above, some studies require the inclusion of people who match selected characteristics of the target audience that NCI is trying to reach. Therefore, PII such as gender, age, race/ethnicity, address, telephone number, email address, education, medical/health status, occupation, and ability to travel may be required to be asked on the initial screening question­naire used for recruiting. Potential participants are informed that this is being done to make sure that NCI speaks with the kinds of people for whom its messages are intended. Again, respondents are assured that the informa­tion is voluntary and will be kept secure to the extent provided by law. All information on race/ethnicity will comply fully with the standards of OMB Statistical Policy Directive No. 15, October 1997.

Raw data from data collections that include sensitive information (for example, screening questionnaires and audio tapes) are not retained once the data have been extracted and aggregated; nor does the information become part of a system of record containing permanent personally identifiable information that can be used for retrieval.

A.12. Estimates of Annualized Burden Hours and Costs

The number of respondents will vary depending on the nature of the NCI activity involving advocates or the topics being addressed by interviews or focus groups. Table A.12-1 below provides an example of a distribution of respondents and hours by type of data collec­tion over the three-year data collection period. It is estimated that there will be 1,975 respondents, the same number of responses, for a total annual burden 1225 hours. The NCI anticipates that over the three-year life of the project, there will be a total of 5,925 respondents (and responses), amounting to a burden of 3,675 hours.

A.12-1: Estimate of Burden Hours Over Three Years (For Generic Submissions)
Survey/Instrument	Number of Respondents	Frequency of Response	Average Time Per Response (Minutes/Hour)	Annual Burden Hours
Self-Administered Post-Activity Questionnaires	3,600	1	20/60 (.33)	1,200
Other Self-Administered Questionnaires and Forms	1,800	1	60/60 (1.0)	1,800
Individual In-Depth Interviews	225	1	60/60 (1.0)	225
Focus Group Interviews	300	1	90/60 (1.5)	450
Totals	5,925			3,675

*On occasion, respondents may give permission for a follow-up survey or interview. As this happens fairly infrequently, we are not adding line for interview or survey respondents who give two responses. Sub-study submissions will indicate when NCI anticipates follow-up surveys/interviews with a given number of respondents.

It is estimated that the collective annualized cost to the respondents will be $74,875. The NCI anticipates that over the three-year life of the project, the total cost to respondents will amount to $224,625 (Table A.12-2). This is calculated based on the cost to individual respondents who are members of the general public is based on the estimate of $17.00/hour and an average respondent burden of 75 minutes per respondent. While physicians and researchers sometimes­ participate gratis in telephone or self-adminis­tered surveys (time permitting), it is customary to reimburse them at the average rate of $67.50 per hour for taking part in focus groups (http://www.bls.gov/oes/current/oes291062.htm). The cost for participating is approximate­ly $67.50 for physicians and researchers based on average respondent burden of 55 minutes per respondent. Table A.12-2 represents the best estimate that can be given based on estimates of who the respondents are for upcoming generic sub-studies. The number and types of respondents may vary from this estimate.

A.12-2: Cost to Respondents Over Three Years (For Generic Submissions)
Type of Respondents	Number of Respondents	Frequency of Response	Average Time Per Response (Minutes/Hour)	Hourly Respondent Wage Rate	Cost
General Public (Consumer Advocates)	3,495	1	75/60 (1.25 hours)	$17.00	$74,268.75
Physicians/ Researchers	2,430	1	55/60 (.92)	$67.50	$150,356.23
	5,925				$224,624.98

­­

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers

There are no capital or start-up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.

A.14. Annualized Cost to the Federal Government

The estimated annual cost to the government for the services of the evaluation contractor is $85,000 for interviews and focus groups. NCI staff time required to participate in planning and design activities, collecting data, and conducting analysis is estimated to average .5 FTE over the 12-month study period. These figures correspond to a total of $45,000 over 12 months. Finally, there are costs associated with data analysis, which total $1,500. The total annualized cost to the government is $131,500, which amount to an estimated $394,500 over the course of three years.

The overall government distribution is summarized in the following table:

Table A.14-1 Government Cost Distribution
	ANNUAL AVERAGE	COST FOR THREE-YEAR PROJECT PERIOD
Contractor Costs	$85,000	$255,000
NCI Personnel Subtotal	$45,00	$135,000
Analysis	$1,500	$4,500
Grand Total	$131,500	$394,500

A.15. Explanation for Program Changes or Adjustments

This is a new, generic collection of information.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Evaluation/research staff will discuss objectives with the individuals responsible for development; determine the analytic questions to be addressed; and then prepare the research procedures, instruments, and data analysis plan. The analysis conducted for each study will be determined by the objectives of the research and the target stakeholder audience. Research techniques may include qualitative analysis, satisfaction analysis, descriptive statistics, statistical analysis and parametric statistical tests. Analysis may be conducted using cross-tabulation procedures, with categorical variables; or between-group procedures, with continuous variables.

While the primary purpose of all OAR studies is to provide information for the purposes of improving programs and activities, NCI shares information internally and also makes results available to a variety of health program planners at Government agencies, voluntary organizations, health professional organiza­tions, and medical institutions. Information collected will be compiled and presented in reports and briefings for staff from OAR and other divisions within NCI and NIH. Reports will include information regarding respondent demographics, basic descriptive data, comparisons across demographic and stakeholder subgroups, recommendations for improving programs and products, and analyses of longitudinal changes. In addition, although OAR has not published articles in the past, NCI may wish to publish new findings in journals and present the results of its research at meetings of profes­sional associations, for example, the American Public Health Association and the Society for Public Health Education. Formative research conducted by OAR may also be summarized in news-related publications such as the NIH Record.

The specific materials and activities that will be evaluated and the timing of these studies are not known at this time. While the research period varies somewhat depending on the complexity of the testing and number of respondents required, the typical study will require approxi­mately 12 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical pretest is shown below:

A.16-1 Project and Publication Timeline
Activity	Time Schedule After OMB approval
Initial review of research questions	1-2 weeks after OMB approval
Write data collection instrument	3-4 weeks after OMB approval
Preparation of design and plan	4-5 weeks after OMB approval
Review of design	6-7 weeks after OMB approval
Collection of data	7-8 weeks after OMB approval
Analysis of data	9-10 weeks after OMB approval
Write report of findings	11-12 weeks after OMB approval
Develop manuscript (when seeking publication)	4-6 months after OMB approval
Submit for publication	7-8 months after OMB approval

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

The OMB control number and expiration date will be displayed in the upper right-hand corner of all data collection instruments.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

NCI is in full compliance with the provisions contained within the Certification for Paperwork Reduction Act.

1 This was OMB approved research under OMB: # 0925-0046-08, expiry date 10/31/2006.

2 This was OMB approved research under OMB #: 0925-0046-17, expiry date 10/31/2006.

3 PIA SORN 09-25-0106

4 E. Singer, J. Van Hoewyk, and M. P. Maher, “Experiments with Incentives in Telephone Surveys,” Public Opinion Quarterly, Vol. 64, No. 2, Summer 2000, pp. 171-188; A. H. Church, “Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-Analysis,” Public Opinion Quarterly, Vol. 57, No. 1, Spring 1993, pp. 62-79.

5 U.S. Department of Health, Education, and Welfare, “A Study of the Effect of Remuneration Upon Response in the Health and Nutrition Examination Survey, United States,” Vital and Health Statistics, Series 2-No.6, 1975.

6 W. D. Mosher, W. F. Pratt, and A. P. Duffer, “CAPI, Event Histories and Incentives in the NSFG Cycle 5 Pretest,” American Statistical Association, 1994 Proceedings of the Section on Survey Research Methods, Vol. 1, 1995, pp. 59-63.

7 A. F. Subar, V. Kipnis, R. P. Troiano, et al., “Using Intake Biomarkers to Evaluate the Extent of Dietary Misreporting in a Large Sample of Adults: the OPEN Study.” American Journal of Epidemiology, Vol. 158, 2003, pp. 1–13.

14