Admongo 2015 Supptg Stmt fin

Supporting Statement for a Paperwork Reduction Act Submission to OMB
Federal Trade Commission’s Admongo Evaluation
(OMB Control No. 3084-NEW)
A. JUSTIFICATION
(1 & 2) Necessity of collection; how, by whom, and for what purpose
As the nation’s consumer protection agency, the FTC is responsible for enforcing laws that
prohibit unfair and deceptive advertising and marketing practices. Part of this mission involves
educating consumers, including young consumers. Most advertising is not fraudulent or
deceptive but nevertheless adopts a point of view that is designed to persuade. Research
indicates that children may not understand that this persuasive intent drives the way a product or
service is presented in an advertisement. 1 Children therefore may process advertisements
without appreciating that the characterization of the product or service is entirely from the
seller’s point of view, which may differ from theirs. 2 Furthermore, due to the increasing
presence of marketing embedded within movies, websites, television shows and video games,
children may have increasing trouble differentiating ads from content. 3
Motivated by these concerns, the FTC launched a youth-directed, multi-media advertising
literacy campaign called Admongo in April of 2010. The online game and accompanying lesson
plans were distributed to every U.S. public school with a fifth or sixth grade class (approximately
100,000 educators in all). Admongo aims to help children from 8 to 12 become more discerning
consumers of marketing information. The program has three broad objectives: (1) raise
awareness of advertising and marketing messages; (2) teach critical thinking skills that will allow
children to better analyze and interpret advertisements; and (3) demonstrate the benefits of being
an informed consumer. The program teaches students specific skills: how to identify ads, how to
identify the ways advertisers target certain groups of consumers, how to spot persuasive
techniques commonly employed by ads, and how to apply an understanding of advertising
techniques to make smarter purchase decisions. The Admongo campaign includes an online
game, in-school lesson plans, sample ads that can be used at home and in the classroom, and
teacher training videos. All materials can be viewed at www.admongo.gov.

See Livingstone, S. and E. J. Helsper. 2006. “Does Advertising Literacy Mediate the Effects of Advertising on
Children? A Critical Examination of Two Linked Research Literatures in Relation to Obesity and Food Choice.”
Journal of Communication. V. 56. 560-584; Hobbs, R. 2004. “Does Media Literacy Work? An Empirical Study of
Learning How to Analyze Advertisements.” Advertising and Society Review. V. 5. Issue 4; Oates, C., M. Blades, B.
Gunter and J. Don. 2003. “Children’s understanding of television advertising: a qualitative approach.” Journal of
Marketing Communications. V. 9. 59-71.
1

See Kunkel, D., B. L. Wilcox, J. Cantor, E. Palmer, S. Linn, and P. Dorwick. 2004. “REPORT OF THE APA TASK FORCE
ON ADVERTISING AND CHILDREN, Section: Psychological Issues in the Increasing Commercialization of Childhood”
(hereinafter, “Kunkel, et al., 2004”); Martin, M. C. 1997. “Children’s Understanding of the Intent of Advertising: A
Meta-Analysis.” Journal of Public Policy and Marketing. V. 16. 205-216.
2

3

See Kunkel et al., 2004.

This submission describes a proposed evaluation of the effectiveness of the online game
component of Admongo. The online game is an interactive teaching tool in which players
advance to higher levels by mastering progressively more sophisticated topics in advertising. In
the proposed evaluation, a randomly selected group of 8-12 year-old students will play the
Admongo game and then take an online ad literacy test. Their scores will be compared to those
of a control group who took the test without exposure to Admongo. The performance of the two
groups will then be analyzed for evidence that the Admongo online game can improve ad
literacy.
The results will provide cost effectiveness data on an important component of the FTC’s youthtargeted outreach. The online game is the one component of the Admongo program that children
can most easily discover, engage with, and learn from on their own. Cost effectiveness data will
enable FTC staff to evaluate both this program and the potential use of other similar programs in
the future. The performance of the control group will provide the agency its first-ever look at the
pre-existing distribution of ad literacy skills among 8 - 12 year old Americans, while the
comparison of control and treatment groups will tell the FTC how one intervention designed to
augment youth ad literacy has fared.
In summary, the collection of this information will enable the agency to improve its design and
targeting of youth-directed consumer education. This type of consumer outreach is a core
component of the FTC’s mission.
(3) Use of automated, electronic, mechanical or other technological collection techniques
This collection is administered entirely through the Internet. A randomly selected half of
participants play an online game that teaches aspects of ad literacy. All participants take an
online ad literacy test, with their answers transmitted over the web and their performance
tabulated by computer. Parents of participants also fill out an online screening questionnaire.
(4) Efforts to identify duplication
This information collection does not duplicate any current or previous collections. The proposed
collection represents the first time the FTC’s Division of Consumer and Business Education has
conducted a rigorous impact evaluation of one of its outreach efforts.
(5) Impacts on small businesses
This information collection does not affect small businesses. The respondents are students ages
8 - 12 and their parents.

2

(6) Consequences if collection is not conducted; technical or legal obstacles to reducing
burden
If the collection is not carried out, then the FTC will forego valuable new information on (i) the
distribution of pre-existing ad literacy skills in the 8 - 12 year-old population; and (ii) the
effectiveness of a taxpayer-funded intervention designed to improve ad literacy in that same
population. Having this information would help the FTC to better understand the pre-existing
knowledge of a population it seeks to educate, and to design more comprehensible and effective
educational materials for young people. These abilities will mean a higher return for every
outreach dollar, as more children are reached more effectively with the same amount of funding.
The estimated burden is already the minimum possible for the proposed study. Further reducing
the burden would require reducing either the sample size or length of the questionnaire and
adversely affect the usefulness of the information collected.
(7) Circumstances requiring collection inconsistent with guidelines
The collection of information in the proposed study is consistent with all applicable guidelines
contained in 5 C.F.R. § 1320.5(d)(2).
(8) Public comments/consultation outside the agency
As required by section 3506(c)(2) of the Paperwork Reduction Act (“PRA”), 44 U.S.C. §§35013521, the FTC published a notice seeking public comment on the proposed collection of
information. See 79 Fed. Reg. 11,789 (March 3, 2014). The FTC received one comment.
The commenter was a private citizen who offered several observations on the proposed study
design. First, the commenter pointed out that the sample is restricted to children with Internet
access at home, limiting the generalizability of the results. In response, we note that although
use of an Internet panel may limit generalizability of results, the household-level information
collected from a screening questionnaire administered to parents should at least provide
information on how the sample differs from the universe of interest. Second, the commenter
noted that an evaluation of Admongo’s effectiveness could be helpful to the FTC’s child-directed
outreach efforts, but that if few children access Admongo, then a study of its effectiveness is not
needed. In response, we point out that the objective of the proposed study is to evaluate the
potential effectiveness of the Admongo online game, which is independent of the actual use of
the game. A finding of a beneficial effect could lead to wider use of Admongo. Third, the
commenter expresses concern that the control and treatment groups may differ in ways that will
confound measurement of Admongo’s effect. In response, we note that participating students
within each age-sex cell will be randomly assigned to control and treatment groups, minimizing
the chances that the groups will differ systematically. And, fourth, the commenter suggested
asking participants’ parents to certify that their children have received no assistance when
completing the ad literacy test. In response, we find this a sensible suggestion and will consult
with the market research company on the feasibility of obtaining such a certification from
parents.

3

Pursuant to OMB regulations (5 C.F.R. Part 1320) that implement the PRA, the FTC is providing
a second opportunity for public comment while seeking OMB approval for the study.
(9) Payments or gifts to respondents.
All students (or their parents) will be compensated in the standard manner by the market research
company that recruits them and runs the experiment. Treatment-group students are expected to
be compensated more than control-group students due to the former group’s substantially larger
time commitment.
(10 & 11) Assurances of confidentiality/matters of a sensitive nature
The ad literacy test will not include sensitive questions. A draft of the test accompanies this
submission. A screening questionnaire, administered online to participants’ parents, also
accompanying this submission, will collect some demographic and economic information about
the household. The FTC will receive the data in anonymized form. The market research
company will assign each player a randomly generated unique user ID, to which the parent
screening questionnaire, the student’s game performance data, and his or her ad literacy test
responses will be matched. Only the market research company (with whom the test subjects had
already willingly shared their PII) will know the identities behind the user IDs.
(12) Estimated annual hours burden
The proposed evaluation will involve 800 students ages 8-12. The half of the sample assigned to
the treatment group will play the Admongo online game for one hour and then take a 20-minute
advertising literacy test immediately afterwards. The time burden for the treatment-group totals
533 hours. The half of the sample assigned to the control group will take the test without playing
the game. The time burden for the control group will be only the time required to take the test 133 hours in total. Finally, a parent of each participating student will be asked to complete a
screening questionnaire, estimated to take 5 minutes. The aggregate time burden to complete the
parent questionnaire totals 67 hours. Therefore, the total time burden for all participants equals
733 hours.
(13) Estimated annual cost burden
Participation will not impose any start-up, capital, or labor expenditures. The costs to
respondents involve only the time expended to play the Admongo online game and/or take the
online advertising literacy test or complete a screening questionnaire. Participation in the
evaluation is voluntary; respondents are drawn from existing pools of Internet panelists (i.e.,
households that have already indicated they are willing and able to take part in Internet research),
and participants and their parents are free to refuse the invitation to participate in any particular
study.

4

(14) Estimate of cost to federal government
The estimated cost to the federal government includes: (i) $20,000 for two children’s marketing
consultants who will help to run the evaluation and perform preliminary tabulation and analysis
of results; (ii) $14,000 for an Internet market research firm that will field the panel and
administer the evaluation; (iii) $8,000 for a web development firm that will program the ad
literacy test; and (iv) $14,640 in FTC staff time. 4 The total estimated cost to the federal
government is therefore $56,640.
(15) Program changes or adjustments
Not applicable. This is a new collection of information.
(16) Plans for tabulation and publication
The FTC plans to use standard regression and statistical hypothesis testing techniques to analyze
the data. FTC staff have not determined whether the results will be published in a report or other
publication, and, if so, in what manner and at what time.
(17) Display of expiration date for OMB approval
The FTC will display the expiration date on the first screen of the online game, parental
screening questionnaire, and the ad literacy test administered to students.
(18) Exceptions to certification
Not applicable.

This estimate is based on 250 hours of FTC staff time (Economists: 40 hours per week x 4 weeks x 1
economist = 160 hours x $60 per hour = $9,600; other staff members’ time: 90 hours total x $56 per hour
=$5,040).
4

5

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
(1) Description of sampling methodology
The FTC proposes to conduct a study of the effectiveness of its Admongo advertising literacy
program for students ages 8 - 12. Thus, the universe of interest is all 8 - 12 year olds in the U.S.
The proposed evaluation will use a randomized controlled experiment to estimate the effect of
playing the Admongo game on a student’s level of ad literacy. Ad literacy will be measured by a
test that has been written specially for this purpose by FTC staff and with the input of two
children’s marketing consultants.
The study will involve approximately 800 students ages 8-12, selected by a marketing research
company from among its existing panelists. To recruit child panelists, the marketing research
company first reaches out to adults through online banner ads, lists obtained through partnership
programs (e.g., airline loyalty participants), referrals, and social media sites. When they sign up
to participate in these studies, adults are then asked a battery of questions about themselves, the
presence of children in the family, and whether they would agree to allow their children to
participate in surveys.
The sample will be selected to match the 8-12 year-old U.S. population along the dimensions of
age, sex, race and ethnicity, and Census region. More precisely, each of four age-sex cells will
contain 200 subjects and will include 15% African American subjects and 15%
Bilingual/English-Dominant Hispanic subjects, and each cell will mirror population-level
frequencies by Census region. 5 Half of each 200-person cell will be randomly assigned to a
treatment group, and the other half to a control group.
Treatment students will be instructed to play the Admongo online game from their homes for one
hour and then to complete an advertising literacy test (also online) within the allotted time (20
minutes). To ensure that each treatment student’s true exposure to the game meets the one-hour
requirement, her time spent playing (and other measures of her performance within the game)
will be monitored and logged by the game’s server. Control students will be provided a link
directly to the test and told simply to complete it within the allotted time. To ensure that control
group members do not play the game, no mention will be made to these students about the
existence of Admongo or its connection to the ad literacy test. To further ensure the integrity of
the evaluation, the market research company will screen out any panelist who has been exposed
to Admongo prior to this study.
Admongo’s effect on ad literacy will be estimated from the difference in test scores between the
control and treatment groups. Additional variables measuring students’ demographic, financial
and family characteristics, to the extent that we are able to capture this information through a
screening questionnaire that is administered to participants’ parents, will increase the precision
of the estimate of Admongo’s impact and will reveal the influence of these factors on ad literacy.

5

The four sex-age cells are boys 8-10, boys 11-12, girls 8-10, and girls 11-12.

6

A power analysis was conducted to determine the minimum detectable effect (MDE) of the
experimental design, i.e. – the smallest difference in mean test score across control and treatment
groups for which there is at least an 80% probability of rejecting the null hypothesis of no effect
at the 5% significance level. For a comparison of the entire treatment group and entire control
group (n=400 in each group), a conservative estimate of the MDE is 4 points out of 100, or
approximately one question on the FTC’s 26-question ad literacy test. 6 For a comparison of
control and treatment means by sex or by age category (n=200 in each age-treatment or sextreatment status group), a conservative estimate of the MDE is 5.6 points, or approximately 1.5
questions. Finally, for a comparison of control and treatment means by age-sex category (n=100
in each age-sex-treatment status group), a conservative estimate of the MDE is 8 points, or
approximately two questions on a 26-question test.
Because study participants will come from households in which an adult has volunteered to be
on a market research panel (and, furthermore, because panel households can refuse to participate
in any given study), the sample may suffer from selection bias and may not constitute a
nationally representative sample of 8-12 year-old American children. Therefore, the estimate of
Admongo’s impact, derived from this sample, will not generalize to the broader audience of all
8-12 year-old Americans. However, the control variables will provide some understanding of
the ways in which study participants differ from the general population of 8 -12 year-olds.
(2) Description of the information collection procedures
The survey instrument is an online advertising literacy test written by the FTC, with input from
children’s marketing consultants and with design help from a major advertising agency. The
central outcome of interest is the proportion of questions answered correctly. Also of interest are
the particular questions, if any, that treatment students tend to answer correctly more often than
control students. Every student’s test will contain the same set of questions, although different
versions of the test may arrange the questions in different sequences to counter any potential
order bias.
The test includes a mix of 35 multiple-choice, true-false, and yes-no questions. Twenty-six
questions are designed with a single right answer in mind, but the remainder are subjective
questions that ask about students’ individual experiences and opinions. These questions will
help the FTC to relate subjects’ attitudes and experiences to the outcome (ad literacy) captured
by the objective questions.
The 26 objective questions focus on recognizing various persuasive techniques; understanding ad
targeting; identifying ads in their various forms; interpreting explicit and implied claims;
understanding advertisers’ responsibilities under the law; and understanding the financial
relationships behind advertising (its role in supporting content, paid testimonials, etc.). The
opinion-based questions touch on the ways that children gather information about products they
want, how children’s experiences with products compare to the impressions given in
advertisements, and whether children receive guidance from their parents about advertising.

The estimate is based on the assumption of a control group population with a mean score of 70% correct and a
standard deviation of 20 percentage points.
6

7

The objective questions elicit a mix of specific factual knowledge (e.g., the meaning of “target
audience” or the name of a specific persuasive technique) and higher-order critical thinking skills
(e.g., understanding why an advertiser might use cross-promotion; predicting the environment
most suited for targeting an ad’s intended audience).
The centerpiece of the ad literacy test is a set of three mock advertisements. Each mock ad is
accompanied by four multiple-choice questions. The mock ads target roughly the 8 - 18 year-old
age range and so present products likely to garner the attention of the 8 - 12 year-olds in the
evaluation. One ad targets females; one targets males, and a third is designed to be genderneutral.
The questions that accompany the mock ads ask students to evaluate the claims, persuasive
techniques, targeting, and presence or absence of purchase-relevant information. These specific
skills are taught in the Admongo lessons and they also constitute key elements of advertising
literacy more generally.
Analysis of data. Analysis will involve estimating a linear regression model in which the
observation is at the level of an individual student. The data is cross-sectional in nature, with
each student appearing just one time in the dataset. The main dependent variable of interest will
be the student’s proportion of questions answered correctly on the ad literacy exam. The
independent variables will include some or all of the following:
•
•

Admongo treatment dummy (1, if the student is in the treatment group, and 0 otherwise)
Student/household-level variables, including:
o student’s age, sex, grade level, and race/ethnicity
o whether the student attends private or public school
o Census region of residence (West, Midwest, Northeast, or South)
o whether there is a computer in the home, whether this computer has Internet
access, and whether the student is allowed to go to kids’ websites on the Internet
o number, age, sex, and grade level of other children in household
o annual household income
o caregiver’s sex, age, educational attainment, employment status, race/ethnicity,
and marital status

The main parameter of interest is the coefficient on the Admongo treatment dummy. A positive
and statistically significant estimate for this coefficient will imply an “Admongo effect” – i.e.,
will support the notion that Admongo participation increases ad literacy, controlling for all other
measurable factors. Interest also centers on the extent to which an Admongo treatment effect
varies with student or household characteristics. Thus, in alternative specifications, the
Admongo treatment dummy will be interacted with variables like age, race/ethnicity, and
household income. Significant coefficient estimates on these interaction terms will provide
evidence that the magnitude of Admongo’s effect varies based on specific observable
characteristics.

8

(3) Methods to maximize response rates/reliability of sample data
Several steps have already been taken by the FTC and its contractors to help maximize the
potential response rate and reliability of the sample data, and more will be implemented when the
evaluation is fielded. Steps already taken include formal piloting of the ad literacy test on a
group of children ages 7-11. The piloting was carried out by children’s marketing consultants
and included timed administration of the draft test and focus-group-style discussions of the
wording and ordering of questions and the clarity of instructions. The piloting led to numerous
refinements to the ad literacy test.
Further steps that will help maximize response rates and reliability include administering the
evaluation only to students whose parents have expressed interest in participating in Internet
studies. These parents have agreed to participate, and to allow their children to participate, in
regular online studies, with minimal topic duplication, and monthly maximum quotas set by the
panel company in order to avoid “wear-out.” Panelists accumulate points each time they take a
survey and points can be applied towards various token awards and small prizes. Other activities
such as puzzles and quizzes keep panelists actively involved and ensure they stay regularly
engaged and eager to participate.
(4) Testing of procedures or methods undertaken
The FTC has piloted a draft version of its ad literacy instrument on two focus groups of children
ages 7 - 11 (a total of nine children participated). The piloting provided feedback on overall
difficulty of questions, question language, and question formats and ordering from members of
the test’s target audience. These insights have been incorporated into a finalized version of the
test instrument. The Admongo online game was also piloted extensively on children prior to
public release (under separate OMB authorization).
(5) Individuals consulted on statistical aspects of the surveys
FTC economist David Givens (Bureau of Economics, Division of Consumer Protection ((202)
326-3397) designed the study and is responsible for its statistical aspects. The following are
additional persons involved with collecting and/or analyzing the resulting survey responses:
Jennifer Leach (Bureau of Consumer Protection, Division of Consumer & Business Education);
Brett Berk and Janet Oak (children’s marketing consultants); Natalie Lauder, Research Now
(Internet panel provider).

9