SUPPORTING STATEMENT
Part B
Connecting Primary Care Practices with Hard-to-Reach Adolescent Populations
February 8th, 2011
Agency of Healthcare Research and Quality (AHRQ)
B. Collections of Information Employing Statistical Methods 3
1. Respondent Universe and Sampling Methods 3
2. Information Collection Procedures and Analysis Plan 5
3. Methods to Maximize Response Rates and Deal with Non-Response 10
Four primary care practice sites have agreed to participate in this project (Table B-1). These sites are members of the SNOCAP-USA practice-based research network in Colorado. They were purposively selected for this mixed-method research to reflect a variety of practice types (including youth from underserved and Hispanic communities) and from a mixture of urban and rural practices. Youth younger than 13 or older than 18 years of age at the time of enrollment in the study will be excluded as the focus of the study is on adolescents and not children or adults.
Name of site |
Location |
Type of practice |
Approximate number of adolescents patients (ages 13-18) in practice |
Projected number of adolescent clinic visits in 12 months (40% of adolescent population with an average of 1.25 visits each) |
Urban |
School-based clinic |
176 |
88 |
|
Urban |
Federally qualified health center |
150 |
75 |
|
Rural |
Family practice |
275 |
138 |
|
Rural |
Family practice |
175 |
87 |
|
TOTAL |
|
|
776 |
388 |
As described in Supporting Statement A, a variety of data collections, both quantitative and qualitative, and involving both practice staff and patients, will be conducted as part of this project. These include the RAAPS (Rapid Assessment of Adolescent Preventive Services) questionnaire, which is collected at every patient visit and is used for process improvement. These also include two data collections involving statistical surveys (Table B-2).
The response rate for the adolescent behavior and communication survey deserves special attention. The response rate for participants (that is, those who receive permission from parents to participate) is acceptably high at 80%. It is recognized that if the denominator of the response rate is considered to be the respondent universe, the response rate is only 24%. Nonetheless, this response rate is common in surveys of sensitive behaviors, particularly in adolescents, and still provides useful information:
Stakeholders considering adoption of these technologies will be interested in some analysis of the effects on behaviors and behavioral mediators.
Self-report is the only feasible method of assessing these behaviors and behavioral mediators.
Surveys conducted as part of the Youth Risk Behavior Surveillance System are considered to be valid and useful, and have received OMB approval. It is notable that our projected overall response rate of 24% is similar to the overall response rate of 29% observed for the 2007 Colorado YRBS survey .
Alternatives to the use of a survey, such analysis of behaviors in reported to the in-office screener (RAAPS), would not allow assessment of pre-post changes and would not include outcomes that could be compared to previous studies .
Findings from this project can also be used to assess whether larger scale studies with more resource-intensive data collection methods are worth pursuing.
Outcome |
Respondent universe |
Description of sample |
Sampling methods |
Projected response rate |
Projected sample size |
Adolescent behavior and communication survey |
776 patients (of which 30%, 233, will receive parental consent) |
Adolescent patients in clinic panel (those who have had an appointment 18 months prior to intervention start date |
Universe survey (census) |
30% participation rate 80% response rate (completion of baseline and six month surveys) among participants. |
186 patients (80% of 233) |
Post-visit satisfaction survey |
388 visits |
Adolescent patients having clinic visit in 12 month period |
Universe survey (census) |
80% |
310 visits |
For other quantitative data that are to be collected on the practice level (Table B-3), three practices will provide data for the assessment of the proportion of visits for adolescents, and all four will provide data on documentation of health behaviors. (The Sheridan practice sees only adolescent patients, so the proportion of adolescent visits is not a relevant measure.)
Outcome |
Sample universe |
Description of sample |
Sampling methods |
Projected sample size |
Documentation of health risk behaviors |
388 pre-implementation visits 388 post-implementation visits |
Census of visits12 months pre- and 12 months post-implementation |
Post-implementation: random sample Pre-implementation: matched (as permitted by practice management systems) |
200 pre-implementation visits 200 post-implementation visits |
Proportion of visits for adolescents |
300 pre-implementation visits 300 post-implementation visits |
Census of visits12 months pre- and 12 months post-implementation |
Universe survey (census) |
300 pre-implementation visits 300 post-implementation visits |
Qualitative data from patients, staff, and clinicians will also be collected and analyzed (Table B-4)
Data Collected |
Respondents |
Recruitment |
Iterative refinement of the online interventions and outreach efforts |
2 adolescent representatives from each practice |
Practice to recruit adolescent patients who are considered to be susceptible to risky behavior, and able to use online resources |
Assessment of utility of online interventions for patients |
2 adolescent participants from each practice |
Practice to recruit adolescent patients who are considered to be susceptible to risky behavior, and able to use online resources |
Assessment of effort and workflow concordance of screener and out-of-office outreach |
2 staff members from each practice |
Staff members involved in the RAAPS process |
Assessment of clinical utility of RAAPS and online interventions |
2 clinicians from each practice |
Clinicians in the practice who see adolescent patients and are willing to discuss their experience with RAAPS |
Assessment and refinement of manual that is developed |
Practice manager and practice director |
Each site has one practice director for the site and one lead clinician for the site. |
Of note, data from the RAAPS (Rapid Assessment for Adolescent Preventive Services) questionnaire is collected as a practice improvement intervention to improve the quality of the adolescent’s clinic visit. SNOCAP-USA will assist practices to adopt the electronic version of RAAPS (www.raaps.org) using netbook computers.
Each practice will use its practice management system to create a database of patients to be invited to participate in the survey. This dataset will be used to create address labels for mailing recruitment-consent packets. SNOCAP-USA will cover the postage for mailing these packets. Each packet will consist of an information letter signed by the practice director, two copies of a parental consent –patient assent form (including a link to an online version of the questionnaire if parents wish to review the questions) and a postage-paid return envelope. SNOCAP-USA will receive the consent forms. Contact information for study participants will be stored on a secure server, to which a study ID will be linked. The baseline survey will then be mailed to patient participants, which will be accompanied by at $10 gift card. One week after sending the baseline survey, all participants will receive a thank you postcard. Two-four weeks after the reminder postcard, a replacement copy of the survey will be mailed to participants who have not returned their surveys. Two weeks after the replacement copy, a final reminder postcard will be sent to participants who have not returned their surveys. The six-month follow-up survey will use the same methods except that the first survey mailing will include a $20 gift card.
The primary outcome for this analysis will be six-month changes in behavior. As above, surveys completed at baseline and follow up will yield up to two scores per patient for an estimated 186 observations. The power calculation below describes the precision of the point estimates for behavior change in each domain. A 95% CI for sample size of 186 will range from +/-0.144SD around the point estimate for a continuous outcome.
Secondarily, we will explore:
Whether observed changes for the population as a whole differ from changes observed in control and/or experimental phase of Olson study (Null hypothesis: Observed changes do not differ from those observed by Olson)
Whether observed changes differ between those reporting any clinician visit in the six-month period vs. those reporting no clinician visit in the six-month period. (Null hypothesis: observed changes do not differ in those who have had one or more clinician visits vs. no clinician visits)
Whether observed changes differ between those reporting any use of the out-of-office components (Facebook/twitter/web) vs. those reporting no use of in any of the components. (Null hypothesis: observed changes do not differ in those with any use of these components vs. no use of these components)
Initially, descriptive statistics will be computed on survey responses to describe baseline patient characteristics. In addition, chi-squares and t-tests will be used to determine whether there are differences between dropouts and non-dropouts (follow-up). In general, we will employ general linear mixed effects models utilizing all available data, assuming ignorable missingness (MCAR or MAR).
We will explore whether these outcome variables are normally distributed prior to analysis. In the event that normality assumptions are not met, we will use transformations to normalize distributions, or employ Generalized Linear Mixed Models for dichotomous responses. All hypothesis tests will be two-sided with alpha=.05 or p values reported). Statistical analysis will be carried out using SAS 9.2. Goodness of fit statistics (e.g. AIC, deviance, -2 log likelihood and change in –2LL for nested models) and model fitting diagnostics to assess for influential points, outliers, overdispersion and heteroscedasticity will be used to evaluate alternative model specifications.
We will model repeated measures within each patient as a function of time, using mixed effects regression models to account for repeated observations within individuals. Time will be coded as 0 for baseline and 1 for follow-up. The outcome for patient i measured at time t is Yti. Clinic will be included as a fixed rather than a random effect due to the small number of clinics. Covariates (e.g. gender) will be included if associated with the outcome or dropout at p<.2. Continuous independent variables will be centered at the overall mean (e.g. age).
yij = γ00 + γ01timeij + γ10clinicj + γ20agej + rij ~ N(0, σ2)
The overall hypothesis will be tested as H0: γ01 =0 vs. H1: γ01 ≠ 0. Thus, assuming 186 patients with pre and post intervention measures, t there will be >80% power to detect a 0.21 SD change from pre to post survey means. For outcome measures in which Olson reported six-month changes, the power to detect changes in our project have been calculated (Table B-5). Other outcomes measures of interest, for which change over time have not been reported, are shown in Table B-6.
Domain |
Submeasure |
Operalization |
Baseline in Olson 2008 intervention group Mean, SD |
6-month change in Olson 2008 intervention group Mean, SD1, std. error |
Projected detectable 6-month change at the 95% confidence level with alpha of 0.8 |
Physical activity |
Screen time |
Hours / weekday |
3.93, 3.49 |
0.687, 3.40, 0.202 |
0.71 |
Physically active |
Days / week of >30 minutes activity |
4.86, 1.83 |
0.581, 2.00, 0.290 |
0.42 |
|
Diet |
Sweetened beverage consumption |
Servings / day |
3.36, 3.12 |
-0.151, 2.88, 0.052 |
0.61 |
Milk consumption |
8 oz servings / day |
2.27, 1.64 |
0.190, 1.38, 0.138 |
0.29 |
|
Fruits and vegetable consumption |
Servings/day |
3.91, 2.21 |
0.165. 2.07, 0.080 |
0.43 |
Source |
Domain |
Submeasure |
Operationalizaton |
Cross-sectional prevalence (% or mean, SD) |
Olson-COMBO |
Alcohol |
Alcohol consumption |
Drank alcohol in last month |
16.3% |
Amount consumed (among users) |
Days with > 1 drink in last month |
0.45 ,1.38 |
||
Olson-COMBO |
Tobacco use |
Smoking |
Smoked in past month |
8.8% |
Amount smoked (among smokers) |
Days smoked in last month |
6.43, 10.20 |
||
Colorado YRBS 2007 |
Drugs |
Marijuana use |
Smoked marijuana in last month |
22.8% |
Colorado YRBS 2007 |
Sexual health |
Sexually active |
Sexual intercourse in past 3 months |
33% |
Use of condom (among sexually active) |
Use of condom in last intercourse |
58% |
Method of data collection: The post-visit satisfaction survey is a brief survey that fits on a postcard. Staff will provide patients with the postcard when they provide the netbook computer for RAAPS screening, instructing the patient to return the postcard to the front desk at the end of the visit, or place it in the mail. These surveys will be anonymous, with no study ID, but will be labeled to indicate the practice they came from. The reverse side of the postcard has business reply card information which will allow SNOCAP-USA to pay for the postage of returned cards.
Power calculations: With projected responses for 310 visits, the point estimate for the responses to each dichotomous question will have a 95% CI ranging from +/-.056 of the point estimate, based on a conservative estimate of maximum variability (proportion of .5).
Method of data collection: The effect of the intervention on elucidation and documentation of health risks will be assessed by chart review. For each practice, we will identify adolescent patients who have had visits in both the 12 months pre-intervention and the 12 months post-intervention, and randomly select 50 from this sample. before the intervention, and charts from 50 adolescent patients who presented during the intervention.
The progress note for each of these visits will be reviewed and scored. The scores will range from 0-4, with 1 point assigned for each of four domains assessed in the visit note: (1) physical activity, (2) diet, (3) tobacco, alcohol, and/or drug use, and (4) sexual health. Scoring will be done on-site with Dr. Ross and Mr. Fernald. “Addressing” a domain will be defined as text in the progress note referring to the domain, or notations on screening forms that indicate the clinician has reviewed the domain. (A single mark or initials on a screening form will not count as addressing a specific domain, but check marks or similar notations for specific domains on screening forms will count). At each site, Dr. Ross and Mr. Fernald will first independently review the same set of 10 pre-intervention and 10 post-intervention notes in common, to assess inter-rater consistency. They will then compare their scores, and in the case of discrepancies will come to consensus on the coding strategy before coding the rest of the visits.
Power calculations: In aggregate there will be 200 pairs of subjects. Assuming that the difference in the documentation scores of matched pairs is normally distributed with standard deviation 1, if the true difference in the mean scores of matched pairs is 0.2, we will be able to reject the null hypothesis that this difference in mean scores is zero with probability (power) of 0.804. The Type I error probability associated with this test of the null hypothesis is 0.05
Method of data collection: The three practices other than the school-based practice (which only sees adolescent patients) will use their practice management system to determine the number of clinic visits overall and for patients aged 13, 14, 15, 16, 17, and 18 in two time periods: (1) the 12 months prior to the start date of RAAPS (which will coincide with the date of release and initial promotion of the out-of-office components) and (2) the 12 months following adoption of RAAPS.
Power calculations: The proportion of visits for adolescent patients aged 13-18 in the pre-intervention period will be compared to the proportion of visits for adolescent patients in the post-intervention period. Previous research indicates that adolescents constitute approximately 7% of medical office visits. Given the projection that there will be 300 visits in both the pre-intervention and post-intervention phases, and assuming that adolescents constitute 7% of visits in the three clinics of interest, we project that there will be a total of 4286 visits (300/0.07) both periods. Given a projected sample size of 4286 visits in both the pre-intervention and post-intervention periods (unpaired), this analysis will have 83% power to detect a difference of 1.7% in the post-intervention period with an alpha of 0.05.
Methods of Data Collection: While quantitative survey methods comprise our primary data source for assessment of this project, we plan to also use qualitative methods to gather more detailed information to gain insights and perspectives on the operational aspects of the project. Our interviews and observations will provide vital insights about how programs are implemented in different settings, providing practical guidance for other clinics to be included in the Manual of best practices for adoption of web and screener technologies (a project deliverable).
In general, unless otherwise specified, the following methods will be employed. Interviews will be conducted jointly by Mr. Fernald and either Dr. Ross or Dr. Barton. The interviews will be semi-structured, following an interview guide that allows for flexibility in exploring emerging themes. The interviewer will take field notes on a standard assessment sheet. The interviews will also be audio-recorded for later review, clarification, and analysis. A brief case summary incorporating key findings will be completed within 24 hours of the interview. This will be reviewed by the primary on-site investigators to identify any areas of disagreement or uncertainty about the interpretation of findings.
Iterative refinement of the interventions: We will initially meet in person with the group of eight adolescents (two representatives from each practice). Practices will nominate adolescent patients who are considered to be susceptible to risky behavior, and able to use online resources. We will discuss the purpose of the project, and expectations for feedback from the group. We will introduce our plans and prototypes for the design of the online interventions (web, Facebook, and Twitter), and our plans for outreach and dissemination. We will receive in-person group feedback in this session. Following this, we will regularly and periodically obtain feedback from these representatives using online asynchronous group discussions, following the methodology successfully employed by Co-Investigator Sheana Bull These asynchronous discussions allow for active discussions that allow representatives to use the sites while they discuss them, allow for more longitudinal feedback, and do not require burdensome travel. These online sessions are also auto-documenting.
Perceived utility for adolescents: Near the end of the study period, we will request each practice help to recruit two youth site (8 total participants) for a post-implementation assessment. They will come to the practice for individual sessions in which they will be asked to say aloud what they are thinking about the sites as they navigate them as they typically would. They will explain their understanding of what they are viewing, why they click on the elements they do, the degree to which they find the information of interest and engaging, and its helpfulness and likely impact on their health behavior. These discussions will not ask participants to talk about their personal health information; instead we will focus on the design, content, and methods of the social media. This “talk aloud” method is similar to a cognitive interview used in survey design . Notes will be recorded using pre-printed “wireframes” of the online sites, so that notes can easily be linked to the areas of the sites in question. Signed parental consent and child assent will be obtained prior to data collection (or just a signed consent if individual is 18 years old). To the extent possible, we will conduct the sessions in a private setting.
Perceived Impact on Practice Operations: Near the end of the study period, at each of the four practice sites, we will conduct a series of semi-structured interviews with key informants. Each domain will be assessed with a different and distinct set of questions, and will involve a different and mutually exclusive set of informants (each including two providers from each site, for a total of eight providers in each domain). We will not be asking about specific patients. Signed consents will be obtained prior to data collection.
Clinician interviews: The effect of the interventions on clinical practice will be assessed by conversations with two providers at each site (8 total providers). We will ask semi-structured, open-ended questions about the impact of the in-office component (health risk appraisals) and the out-of-office component (social media) on the delivery of care to their adolescent patients. In particular, we will ask providers to talk about the value of the information provided, the effects on conversations with their adolescent patients, their understanding of health issues youth are concerned about, and what effects they observed on their ability to deliver care. When possible, these interviews will be conducted on-site, in person, in a private setting.
Administrator/ Staff interviews: The effect of the interventions on the check in process and other business processes will be assessed by discussions with the practice manager and a front-desk staff member and by direct on-site observation of processes during site visits (8 total administrators/staff). We will ask semi-structured, open-ended questions about the impact of the in-office and out-of-office components had the operational aspects of the clinic. In particular, we will ask about how in-office components were implemented, how it affected patient flow, how adolescents reacted to the netbooks, how out-of-office components might have affected patient volume in terms of clinic visits or phone calls, and how burdensome the maintenance of the components for their clinic was. Interviews with individuals will be conducted on site in a private setting. While on site, we will also make observations about how the in-office components have been implemented.
Once data have been collected from all sites, a template style of analysis will be used to organize the data for reflection and development of emerging themes . The initial list of codes will be based on broad thematic areas we expect to see in the data based on our questions. Template coding also allows for using additional codes for emerging themes analysts observe early in the coding. Coded data will be further reviewed to refine and develop provisional themes for further reflection. Fernald, Dr. Ross, and Dr. Barton (with additional assistance from Dr. Bull and Dr. Westfall as available) will then reflect on the data as a team to identify cross-site contrasts and commonalities and to derive final conclusions.
The summarized conclusions will be used to develop a manual to assist primary care practices in adopting similar interventions to improve adolescent care. Drafts of these manuals will be provided to the practice directors and practice managers of each of the four practices for member checking. During telephone interviews their feedback will be solicited and the manual will be revised based on their responses.
The methods employed in the behavioral and communication survey (described above in “Information Collection Procedures and Analysis Plan) employ incentives ($10 associated with the first survey and $20 associated with the second survey) and multi-stage mailings (including follow up to non-responders) following the recommendations of Dillman . We will also consult with the PBRN community advisory council on their recommendations on how to improve community acceptance of and participation in the surveys. Response rate for the post-visit survey will be maximized by keeping it very brief and providing options it to be returned to the check-out desk or to be mailed to the research team using a business-reply card.
Our methods will allow us to compare the number of surveys returned to the respondent universe. For the behavioral and communication survey, analytic methods will accommodate missing values (due to skipped questions or failure to return follow up survey) using mixed methods. The representativeness of the survey participants to the clinic at large can also be assessed in part by assessing whether aggregate responses are consistent with the aggregate responses to similar corresponding questions in the RAAPS screener.
The Rapid Assessment for Adolescent Preventive Services (RAAPS) tool is a well established questionnaire which has had substantial testing of its validity, utility, and ease of incorporation into practice settings . It is currently in use in over 30 health centers.
The behavioral and communication survey is of sufficient length, complexity, and effort to require validation of key survey questions and pre-testing.
The primary questions and outcome measures are derived from the COMBO (Common Measures, Better Outcomes) questions for adolescents which were used in the previous study of an electronic screener for adolescents . The COMBO adolescent measures for physical activity, diet, smoking, and risky drinking were used in three separate PBRNs in Round 2 of the Prescription for Health Initiative, were selected specifically to be practical for use in primary care clinics, and were determined to be sensitive to change, relatively brief, and related to public health goals.
Questions related to drug use and sexual health are derived from the CDC Youth Risk Behavior Survey for high school students (ages 13 years and older) . This survey is regularly conducted among high school students and has previously received OMB approval. The rationale for each YRBS question is available at http://www.cdc.gov/HealthyYouth/yrbs/questionnaire_rationale.htm
Questions on patient-physician communication will include questions from the AHRQ’s CAHPS (Consumer Assessment of Healthcare Providers and Systems) Clinician and Group Survey, draft “Visit” questionnaire (questions 18-26), available at http://www.cahps.ahrq.gov/content/products/CG/PROD_CG_CG40Products.asp?p=1021&s=213. Additional questions were adapted from the well validated Perceived Efficacy in Patient-Physician Interactions (PEPPI) questionnaire . Questions on behavioral intentions, self-efficacy, and social norms were adapted from validated questions employed in research on condom use among youth .
The complete adolescent behavior and communication survey has been pre-tested among five adolescents. It took from 9-12 minutes to complete. All adolescents felt that the survey was of appropriate length and content, and easily comprehensible, and unambiguous. Some skipped sensitive questions, which was appropriate as per the instructions adapted from the YRBS. Based on their feedback, the set of questions on behavioral intention, self-efficacy, and social norms related to alcohol was reworded and skip patterns were clarified.
Pre-testing of the brief and simple post-visit satisfaction survey was not performed because it is a very simple and straightforward, six question survey that can be completed in less than one minute.
The committee responsible for revision of survey instruments will consist of the Task Order Leader (Stephen Ross MD), the Director of SNOCAP-USA (David West, PhD), the Project Coordinator (Doug Fernald MA), and Co-Investigator Sheana Bull PhD.
Statistical aspects of the project were developed with consultation from L. Miriam Dickinson, Ph.D., a biostatistician in the University of Colorado Department of Family Medicine.
1 Standard deviation back-calculated from Olson 2008, based on reported six-month changes in intervention group, control group, and p-value of comparison.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| Last Modified By | william.carroll |
| File Modified | 2011-03-08 |
| File Created | 2011-03-08 |