Investigation of Smart Toys and Additional Toys Child Observations
Abstract:1
Consumer Product Safety Commission (CPSC) staff requests approval from the Office of Management and Budget (OMB) to conduct a one-year new data collection effort. The general objective of this project is to provide the CPSC with information that can be used to help determine the developmentally appropriate ages for selected toys. Findings from this data collection will inform CPSC’s Age Determination Guidelines: Relating Consumer Product Characteristics to the Skills, Play Behaviors, and Interests of Children (“Guidelines”). The Guidelines present age-grading information in the form of recommendations for toys and other articles, children’s toys, childcare articles, and children’s products.
In 2020, CPSC staff released an update to the Guidelines. The revised Guidelines have been extensively updated based on research from the National Institute of Child Health and Human Development (NICHD) with novel and classic toys and children’s observed play patterns at different ages. During the review process, several stakeholders identified categories of toys that are not addressed in the Guidelines, or need updates based on additional research. This research endeavor will begin to fill in the gaps identified during the review by gathering information on the physical characteristics of the selected toys (e.g., size and weight of the toy and its components) and examine whether the various characteristics of these toys are matched to the skills, play behaviors, and interests of children in a particular age groups.
Participation in this data collection is voluntary. In-person data collection sessions will be scheduled for up to 60 children aged 2-4 years old and their caregivers, for a total of 120 participants from the Washington DC area. Legal caregivers will answer a series of screening questions to determine if they meet the criteria for enrollment in the study. Children and caregivers who meet the screening criteria and are willing to participate will complete two in-person data collection sessions, will be enrolled in the study. The first session will be scheduled for 80 minutes, and the second will be scheduled for 80 minutes. Information will be gathered from children by documenting their play patterns with nine toys and noting their ability to interact with selected toys from each of six toy categories (smart toys, take-apart vehicles, musical instruments, figurines, plush toys with electronic components, and manipulatives) as the manufacturer intended. Caregivers will respond to questions asking about their child’s ability to interact with the toys as intended, factors influencing potential purchasing decisions for the specific toys, and whether they would demonstrate how to play with the toys or some of the components. The family will be reimbursed $50 at the end of the first 80-minute data collection session and $100 at the end of the second 80-minute data collection session.
The analysis will examine coded data from the child’s behaviors when interacting with the toys and caregiver’s responses to questions. The Contractor will tabulate descriptive statistics on the coded metrics for each age group, toy, and caregiver responses. When possible, comparisons will be made between children’s play with toys that are intended for their age, versus toys that are intended for children either older or younger. A predefined set of behaviors have been identified as demonstrating a child’s ability to play with the toy as intended. Observation data coded while the child is interacting with the toy will be analyzed and summarized. Some examples of planned descriptive statistics include:
Average play duration by each toy for the different ages.
The age group(s) of children most likely to fully play with the as manufacturer intended.
Percentage of intended behaviors observed by age for each toy.
Percentage of children by age who were able to successfully execute an intended behavior.
Percentage of children who were able to successfully execute an intended behavior after the researcher demonstrated or guided the child through the tasks.
The number of children in each age group who were disinterested with the toy.
The percentage of children who played with the toy in a potentially harmful manner.
The percentage of children in each age group that expressed frustration.
With respect to the caregiver questionnaire, the Contractor will examine the relationship between the child’s actual behavior when interacting with the toys and their caregiver’s perceptions of the child’s abilities to interact with the toy as intended. Specifically, whether caregivers:
Feel the toy was appropriate for their child by age group.
Feel their child could play with the toy in a way that was intended by the manufacturer.
Child could execute all the behaviors needed to interact fully with the toy, without demonstration.
Child could execute all the behaviors needed to interact fully with the toy, after behavior was demonstrated.
Child could execute a single behavior needed to interact with the toy.
Likelihood that they would buy the toy for their child.
Likelihood that they would demonstrate how to interact with the toy.
Findings will provide useful information and enrich the CPSC’s understanding the ages of children who are interested in these toys, which age groups can use the toys as the manufacturer intends, and caregivers’ perceptions of how their child will interact with the toys. CPSC will use the information from this study to inform the development/modification of age determination guidelines for the toys. The Guidelines will be subsequently used by various stakeholders, including manufacturers and consumers of the toys. Manufacturers rely on information contained in the Guidelines when designing toys. That is, toys should be designed and marketed with intention. Other stakeholders who use the Guidelines include CPSC Compliance and Human Factors staff, independent consultants, and testing laboratories who examine toys for the purpose of determining the product’s intended age. Overall, findings may help reduce serious and fatal injuries from children interacting with toys that are not age appropriate.
B. JUSTIFICATION
B.1 Describe the potential respondent universe and any sampling or other respondent selection to be used.
The potential respondent universe is comprised of all children aged 2-4 years of age and their primary caregivers, in the greater Washington, DC area. From this universe, the new data collection will qualify 60 children and their primary caregivers by administering a recruitment eligibility screener.
The new data collection (screening questionnaire) will be administered to an estimated 100 potential caregivers to qualify a total sample of 60 volunteer caregivers and 60 volunteer children. This is a convenience sample. As such, the participants in this research design are not expected nor intended to be a representative sample of all caregivers/children in the United States. That said, care will be taken to recruit participations from a wide cross-section of the population in the study area to ensure people of all demographics (e.g., race, ethnicity, sex, socioeconomic status) have equal opportunity to volunteer to participate.
The following inclusion criteria will be used for eligibility screening:
Caregiver is between ages 18+ years.
Caregiver speaks and reads English.
Caregiver is responsible for a child(ren) who is 2- through 4-years-old when they complete the study.
Caregiver is the primary caregiver for the target child(ren) – s/he is the biological, adoptive, foster, or step-caregiver or a relative of the child(ren), and engages in childrearing activities more than 4 days a week (more than 50 % of the time).
Caregiver routinely purchases toys for the child – s/he seeks information about the toy and engages in decision-making for more than half of the toys that are purchased, either online or in stores.
Caregiver is willing to commit to the observational and the assessment procedures in the lab setting.
The following inclusion criteria will be used for screening to minimize the probability of missing data, and ensure the observational sessions can proceed smoothly:
The child does not have known congenital conditions, ADHD (attention deficit hyperactivity disorder) developmental delay or disabilities, or pervasive developmental disorders (severe impairment in psychosocial development such as autism spectrum disorder). This can mean the child has been diagnosed with one of these conditions or a pediatrician has advised the caregiver to have the child tested.
The child does not have problems with hearing or vision.
The child does not show extreme separation anxiety from the caregiver (i.e., is able to engage with other people or objects, and not unduly upset, when caregiver is not physically close-by).
The child is able to focus on and engage with a toy for more than one minute.
The child does not typically engage in destructive behaviors of objects in the household.
The Study design includes similar numbers of boys and girls within each of the age-group and a sample that is heterogeneous with respect to family sociodemographic and the toy-buying experience.
Gender |
Age |
||
|
2 years |
3 years |
4 Years |
Female |
10 |
8 |
7 |
Male |
10 |
7 |
8 |
Once eligibility is established, the following demographic information will be sought to help ensure diversity of participants enrolled in the study: Child gender, Child/Caregiver race/ethnicity, Total household income (in broad categories), Caregiver education (primary caregiver), Toy purchasing behaviors of the caregiver – how often s/he purchases a toy, whether online or in-store, and general preferences.
The sample size of 60 was selected to ensure the completion of the study within the resources allocated and to have a large enough sample for generalization of the study results.
Data Analysis
Data from child/toy interactions, and caregiver questionnaires will be analyzed individually and collectively to obtain a better understanding of how children interact with the selected toys and determine the developmentally appropriate age for each toy. The analysis will provide an overall synopsis by age group for each toy. In addition to the subjective feedback collected from the caregiver, the Contractor will provide some comparative analysis examining the objective data related to the child’s interactions with the various toys to the subjective responses of the caregivers. That is, the Contractor will compare whether the caregivers’ impressions are consistent with the child’s behavior.
The Contractor will tabulate descriptive statistics on the coded behavior metrics for each age group and toy. When possible, comparisons will be made between children’s play with toys that are intended for their age, versus toys that are intended for children either older or younger. Toys that were not manipulated for at least 15 seconds will be excluded from the quantitative analysis but will be examined more qualitatively. The Contractor may even access the video recording to delve deeper into these toy/child interaction sessions and assess the characteristics of these toys and the age appropriateness.
The data analysis will be organized by toy and the three age groups (2-, 3-, 4- years). The Contractor will aggregate the coded data for each toy from across the age groups. It is important to note that each child will play with multiple toys, and children from different age groups might interact with each toy differently depending on their developmental level. As such, the Contractor will create a crosswalk for analyzing the coded data for the age groups and toys studied. For example, smart toys will have a detailed set of codes addressing their unique characteristics and features and the different behavioral interactions of children across different age groups. Conversely, take-apart vehicles will have their own set of codes addressing their unique characteristics.
The primary interest of this data collection is whether the child of a given age can fully play with the toy as the manufacturer intends. The Contractor will use the coded data for each child/toy interaction and the coded questionnaire responses of caregivers to address two main questions: (1) age appropriateness of the toy and whether there are discrepancies between the determined age based on observations, and the manufacturer’s intended age; and (2) a summary of caregivers’ impressions of the toys and age appropriateness for their child.
From this analysis the Contractor will generate qualitative and quantitative results. For example, the data will be used to calculate the percentage of children in each age group who play with the toy as intended, the number of children in each age group who did not interact with the toy for at least 15 seconds, the percentage of children who did not play with the toy as intended, the percentage of children in each age group that expressed frustration, and so on. Similar analyses as those conducted for the observation data will be conducted on the caregiver questionnaires. That is, the percentage of caregivers who were likely to buy the toy for their child, and percentage of caregivers who felt the toy was appropriate for their child; the percentage of caregivers who mentioned a given factor considered when purchasing, etc. The Contractor will tabulate and produce visual presentations of the quantitative data using descriptive statistics, such as means and frequency plots.
Comparisons will be made between children’s play with toys across the age groups. The recommended age for which a toy is considered appropriate based on the observational analyses will be compared with the manufacturer’s intended age, and discrepancies will be noted.
Integrating information from all sources (observations, interviews, and toy assessments) will provide the most informative answers to the specific research questions. In this way, a fuller picture can be constructed using quantitative, qualitative, and analytic findings to address the effects of the following:
User characteristics;
Toy features;
Information sources used as well as purchasing outlets and their use;
Potential hazards; and
Factors considered during purchase.
B.2 Describe the procedures for the collection of information.
Once potential respondents are identified, they will be administered a recruitment screening questionnaire to determine eligibility for participation in the study (Appendix G). The screening questionnaire collects information on caregiver characteristics such as race/ethnicity, toy purchasing behaviors, and the child characteristics, such as age, developmental conditions, and so on. The screener questionnaire will also ask potential participants if the caregiver is willing to commit to the observational and the assessment procedures of the study and can commit to completing two sessions of the study, so we can verify all scheduled participants can indeed complete the next phase of the study.
If a participant meets the inclusion criteria, the participant will be contacted via phone and email to schedule two study sessions at the Contractor’s office. The first session will be scheduled for 80 minutes and the second will be scheduled for 80 minutes. The Contractor will explain the procedures and go over the confidentiality agreements and answer any questions the participant may have related to scheduling, directions to the location, their participation, and payment. Once scheduled, a confirmation email will be sent to the participant with detailed instructions on how to get to the Contractor’s office, a brief description of the study, including the observational procedures and the safety protocols, and an FAQ to address anticipated questions. The Contractor will also send copy of the consent forms, including the audio/video release for the participant to read through before getting to the lab.
When participants arrive for each session, and are acclimated to the lab setting, the Contractor will review the study procedures and address any questions or concerns the caregiver has about the study. Once the Contractor determines the caregiver is fully informed about the study, s/he will administer the consent form (Appendix C) and audio/video release (Appendix D). Caregivers will indicate their own intention to participate in the study via signing the consent form, and given the young age of our child participants, caregivers will consent for the participating child as well.
Once the caregiver has signed the consent form, the Contractor will ask the child if s/he would like to see some toys and lead the caregiver and child into a laboratory playroom outfitted for observational studies. The protocols use developmentally appropriate lab procedures to sustain the interest of the child. The protocol will include detailed instructions for interacting with caregiver and child participants, administering the questionnaires, observing the child, and handling problematic scenarios. The Contractor also has standardized protocols for accommodating children who experience issues such as separation anxiety from the caregiver, noncompliance, distractibility, and so on.
Each session will be conducted in the Contractor’s Usability Laboratory and will be video/audio recorded. The lab visit will have two simultaneous components – the observational study of child’s interactive behaviors with the toys, and the caregiver filling out a questionnaire. Each of the component is described below.
Observational Coding: The coding will involve one of the Contractor’s trained researchers introducing 9 toys sequentially to the child (4 or 5 toys per session), facilitating the child’s interactions for about 3-15 minutes per toy (depending on toy complexity), and observing and video recording the interaction. The order with which each toy is presented will be randomized to preclude the effects of fatigue and learning from one toy to the next. The toy will be presented unboxed and ready for play. Children will play with one toy at a time. We anticipate conducting 10 sessions per week.
The researcher will remain in the room during the sessions to manually record any observations and ensure the child’s safety. If the child stops interacting with the toy or seeks comfort from the caregiver, the researcher will try to reengage the child and reorient him/her toward the toy. If the trial time for the individual toy has ended, the researcher will introduce the next toy to the child. Approximately 30 seconds before the end of each toy trial, the researcher will let the child know that this trial is over, and the toy will be replaced with another. This procedure will be repeated until the child has had an opportunity to interact with all the toys assigned to the session.
For the observational study, we have developed coding procedures to fully capture the child’s interaction with the toy so the Contractor can determine if the child can interact with the toy as the manufacturer intended in a rigorous manner. The coding checklist contains finite, concrete behaviors that will be noted during the observation session. It is understood that the caregiver and child may need to take breaks (e.g., to soothe a distressed child, or if the child asks them a question). Should the child solicit the caregiver’s help or make efforts to involve her/him in interactions, the RA will instruct caregivers to encourage the child to play independently with the toy.
Coding Checklist: The Contractor will use coding checklists to record real-time observations of the child’s interactions with the toys. Researchers will document whether the child plays with the toys, if he/she can physically and cognitively use the toy as intended and if he/she expresses any socioemotional behavior while interacting with the toy such as crying or throwing the toy in frustration. The coding checklist (Appendix E) is organized around the child’s primary modality of interaction – such as gross motor, fine motor, verbal, emotional, for easy and quick coding. For each modality, the Contractor lists concrete behaviors that are typically shown by children between 2 and 4 years of age when interacting with each of the specific toys, which may be revised based on our pilot observational sessions. The researcher will note the occurrence of behaviors that happen in real time, using a printed version of the checklist for each toy. While every effort will be made to encourage the child to play with the toy longer, a 15-second continuous play will be the minimum requirement to generate valid codes.
Caregiver Questionnaire: Caregivers will complete a set of questions for each toy (Appendix F). When completing the questionnaires, caregivers will have her/his own set of toys in a bag so they can examine the toy while filling out the toy-specific questions. The objective of the caregiver questionnaire is to understand their decisions about toy purchases –whether they believe their child will play with the toy as intended. The questionnaire will be self-administered. The questionnaire will be “toy-centric,” focusing on each of the 9 selected toys with respect to their child and how they might interact with it. The Contractor has strived to make the questionnaires as succinct as possible to ensure the accuracy of data, and to balance the amount of information needed with the length of the questionnaire.
The questionnaire includes questions on: (1) characteristics of the toy pertaining to safety, likeability, and likelihood of purchase, (2) what age child would they buy it for (and skills needed to interact with the toy), and (3) likely response of their own children to the product (to determine if their perception is different from the observed interaction of their child with the toy).
The estimated time it will take the caregiver to complete the questionnaire for each session is approximately 30 minutes. The format of the questions is open-ended for some and forced choice for others. The questions will be structured such that the main response of the caregiver (e.g., yes/no, age, rating on the scale, factors affecting the decision) is recorded, but also document any explanations offered by the caregiver.
Video Recording: The child’s interactions with each toy will be recorded as a backup. The Contractor will use action cameras by GoPro and Contour that provide HD video, flexible mounting, and the ability to attach external microphones for enhanced sound quality in the recorded media, close to where the child participants would play. Each camera includes an SD card for media capture. Media cards will be cleaned at the conclusion of each session, and data will be downloaded to a secure network location for processing immediately after each session. Downloaded data will be renamed to reflect the session details (e.g., toys, participant IDs, date/time, etc.) to ensure unambiguous tracking of each session’s data. Video from the sessions will be saved to the network using the Contractor’s established protocols for security and ensuring participant confidentiality.
Data Security: The Contractor employs several strategies to protect the data collected (e.g., video and audio recordings, as well as other personally identifiable information). These include automatic account and sample access lockout policies, and periodic security scans. All data collected will be housed on servers running on the Contractor’s data centers, which meet the Federal Information Security Management Act (FISMA) moderate standard. Security measures implemented during the study will include, but not be limited to, carefully controlled access to the secure server location associated with the aggregated subject data via project director and system administrator authorization. In addition, participants will be identified by a unique subject numbering convention that will be kept separate from the recruitment data files and used only for study administration personnel requirements.
B.3 Describe methods to maximize response rates.
Participation in the study is voluntary. To recruit a wide range of participants, a variety of methods will be used. Participants will be recruited through advertisements on web sites such as Craigslist, Facebook, etc. Recruiting will also be conducted by contacting potential participants through the Contractor’s maintained databases of previous participants who expressed interest in future studies.
To maximize response rates, once a family meets the inclusion criteria in the screener, their names and sociodemographic characteristics will be added to a list of potential participants. The database will be examined in detail and participants will be selected for inclusion based on their family characteristics with the intent to balance child gender, as well as ensure variability with respect to caregiver education levels, family income, and caregiver toy-purchasing characteristics.
Selected participants will be contacted by the Contractor via phone and email and provided with more detailed information about study requirements and enrollment information. The Contractor will contact all eligible caregivers to ensure they understand exactly what is involved in study participation (including when their participation will start, the number of sessions and how long each session will last), and answer any questions they may have, to minimize non-response at a later stage in the study. The Contractor will enroll caregivers who agree and are available to participate, and schedule two in-person data collection sessions at the Contractor’s facility. The first session will be scheduled for 80 minutes and the second will be scheduled for 80 minutes. The Contractor will make every effort to work with the families so that their session times are convenient for the caregiver.
For each caregiver who agrees to participate, the Contractor will send out a confirmation email within 48 hours of the initial call. The email will include a study agenda, schedule, location of the data collection session, directions to the Contractor’s offices, and a contact telephone number in case of cancellation or questions. Within 24 hours of each scheduled session, the Contractor will also make reminder to increase the likelihood of families keeping their appointments.
Every effort will be made to collect data from each child for all 9 selected toys. However, sessions may have to be discontinued due to child’s non-cooperation, heightened distress, or other reasons. A session will be terminated if the child manifests moderate or severe distress (such as crying inconsolably). In such cases, we will see if the caregiver is willing to return to the lab to continue the session on another day. Based on previous experience, the Contractor recommends that the family be reimbursed $50 at the end of the first data collection session and $100 at the end of the second data collection session.
To further maximize the response rates, the Contractor’s Usability Lab and surrounding rooms will be set up with video-recording equipment and comfortable furnishings to enhance the safety, comfort, and ease of child play. Special floor mats will be installed for very young children (2‑year-olds). Smaller scale tables and chairs will be available in another room for children 3‑ through 4‑years-old. The Usability Lab will be equipped with the resources needed for young children and their families, such as healthy snacks (individually wrapped), tissues, diapers, bottled water, and books.
If families are eligible but are assigned to the reserve sample for various reasons (e.g., they fall into a sociodemographic category that is overrepresented in the sample), they will be sent email informing them that they will be contacted in a few weeks about the study.
B.4 Describe any tests of procedures or methods to be undertaken.
As part of the study development, the Contractor will refine the study procedures by pilot testing the protocol, instructions, and processes used for documenting observed behaviors and administering the caregiver questionnaire with 4 participants (two children & two caregivers). The pilot study will give the Contractor an opportunity to make slight wording changes when needed that will improve overall comprehension without changing the intent of the direction or questions. Piloting will also enable the Contractor to identify anomalies, modify how the researcher will present the toys to the child or the coding scheme and address any unanticipated challenges prior to the start of data collection. Finally, the pilot sessions will allow the Contractor to have a more accurate assessment of the overall time it will take to complete each session as well as each task within the sessions.
The Contractor will enroll two sets of participants, two caregivers and two children, into the study and expose them to the actual test conditions and procedures, following the study protocol. Trained researchers will participate in the pilot sessions. The cameras and other aspects of the lab visit will be made ready before the pilot sessions and be tested as well.
At the end of each pilot session, the Contractor will collect informal feedback from the caregivers as to how the caregiver questionnaire, the presentation of the toys to the child, and flow of the session, could be improved with respect to the reliability and validity of the measures and making the visit more pleasant for the caregiver and the child. In addition, the senior members of the Contractor’s team will provide feedback to the researchers who conducted the session, including procedures for introducing each toy to the child, interacting with the child, safety concerns, recording the observed child behaviors, collecting the questionnaires from caregivers, and coding procedures. The researchers will also check their codes by comparing them to the video recorded, and their work will be monitored and reviewed by the senior members of the Contractor’s research team to ensure consistency. Video recordings of pilot sessions will be used to further train the RAs on how to fine-tune the child/toy interactions and to validate real-time as well as post-observational coding decisions.
The materials and the protocols will be modified based on the results of the pilot sessions.
B.5 Provide the name and telephone number of individuals consulted on statistical aspects of the design.
Khalisa Philips, Ph.D., Ed.M.
Directorate for Engineering Sciences
Consumer Product Safety Commission
4330 East-West Highway
Bethesda, MD 20814
Phone: (301)987-2592
Email: KPhillips@cpsc.gov
Nanmathi Manian, Ph.D.
Senior Study Director
Westat
1600 Research Boulevard
Rockville, MD. 20850
Phone: 301-294-2863
Email: nanmathimanian@westat.com
Doreen De Leonardis, Ph.D.
Senior Study Director
Westat
1600 Research Boulevard
Rockville, MD. 20850
Phone: 301-315-5963
Email: doreendeleonardis@westat.com
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | amanda.kelley |
| File Modified | 0000-00-00 |
| File Created | 2023-12-12 |