C Field Test Analysis Plan

Attachment C

Field Test Data Analysis Plan

The field test will provide an opportunity to examine the performance of ‘fine-tuned’ surveys. These were fine-tuned as a result of the item and validity analysis of the pilot test, which provided input to inform decisions about deleting and revising items and restructuring the surveys. The instruments revised from pilot test input were then put through a round of cognitive interviews, where individuals gave their feedback about readability, clarity, and general understandability. The surveys include a provider (61 items), parent (62 items), and director (53 items) survey.

Table A-1 shows how items in the provider survey were organized into subscales for the analysis. There are 3 content scales: Attitudes, Knowledge, and Practice. Within these, there are 10 subscales (Attitudes - Respect, Commitment and Caring, Openness to Change; Knowledge - Theoretical Knowledge, Knowledge Gathering, Family-Specific Knowledge; and Practice - Relational Skills, Instrumental Support, Joint Goal Setting, and Empower Families). For a subscale to be a latent construct, it must be measured by multiple items. Otherwise, the reliability can’t be estimated. By these criteria, we see that all 10 of the scales are measured by two or more items and will undergo reliability analysis. The number of items per scale ranges from 2 to 10.

Table A-2 shows how items in the parent survey are organized into subscales. There are 3 content scales with 10 subscales in the survey: (Respect, Commitment and Caring, Theoretical Knowledge, Knowledge Gathering, Family-Specific Knowledge, Relational Skills, Instrumental Support, Joint Goal Setting, Empower Families, and Cultural Sensitivity). A comparison of tables A-1 and A-2 shows that the common scales have somewhat different items. This reflects the fact that the surveys had some uncommon items and items that were similar but not identical in wording. Together, the subscales have 62 items. The number of items per scale ranges from 2 to 14.

Table A-3 shows a crosswalk of the items that are parallel between the parent and provider surveys (30 items). By examining these items, we will be able to estimate to what extent parents and providers agree on the three content areas of Attitudes, Knowledge, and Practices related to early child care and education. The frequencies of these items will be examined in a side-by-side comparison of parent and provider responses. There are 8 common subscales across the two surveys in total (in 4 of the subscales there are 2 or more parallel items). For these subscales, total scores will be created and compared between the parent and provider surveys. Comparison of total scale scores defined in this way will give a descriptive comparison of parent and provider experiences and attitudes. Only a subset of the items in the provider and parent surveys will be used for subscale comparison between the two surveys. However, the final psychometric analysis will use all items in the two surveys. Therefore, the comparison of subscales based on parallel items will be for descriptive purposes only.

The analysis plan calls for a target sample of 240 director surveys and a full set of item analyses will be performed on the instrument. The director survey items will be used to describe characteristics of the center (e.g., size, age range of children, race/ethnicity composition, and so on), parent involvement, communication with parents, providing information to families about community support, support of family activities sponsored by the center, and providing parenting information.

Differential Item Function Analysis

If sample sizes permit, the potential bias toward different race/ethnicity and language groups in the assessment will be explored by running Differential Item Function (DIF) analysis. A rule of thumb is that a group size smaller than 20 is too small for a statistical test of group differences. The actual sample size needed for a given power (say 80%) depends on the variability of the outcome and the magnitude of the difference between groups. DIF analysis determines whether a subgroup of individuals responds to an item in a different way than the majority group, indicating possible cultural or language bias in how the item is interpreted by the respondent.¹ In addition to evaluating item functioning between demographic groups, a descriptive comparison between the responses on the English and the Spanish versions of items will be made.

IRT Analysis

Item Response Theory (IRT) is a powerful tool to assess item functioning and the reliability of scales, for items that measure a common, latent scale. Items will be grouped by the subscales that they are hypothesized to measure. Item parameters from the IRT model will indicate how reliable and extreme a particular item is. Estimated scale scores are more reliable than individual item scores and will provide more powerful comparisons between groups. They will also provide more precise estimates for validity correlations of subscales with demographic variables and subscales with each other.

Validity will be assessed by how plausible the mean-level responses for various subgroups are. The field test data include the following demographic information that can be used as a check the validity of the director, provider, and parent surveys:

1. The director survey elicits information about the composition of the center/program: number of children, age range, and percent of children in race/ethnicity categories.

2. The provider survey has information about the teacher’s race/ethnicity, level of education, and whether or not the teacher has a credential from the Child Development Association (CDA).

3. For the parent survey, there will be information about income and race/ethnicity.

Table A-1: Subscales in the Provider Survey

Content Area	Subscale	Teacher Item Numbers	Teacher Item
Attitudes
	Respect
		6e	Beliefs and concepts about child care and education vary by culture
		6f	Even though my professional or moral viewpoints may differ, I accept that parents are the ultimate decision-makers for the care and education of their children.
		7a	When planning activities for children in your program, how often are you able to take into account information parents share about their children?
		7b	When planning activities for children in your program, how often are you able to take into account families’ values and cultures?
		8a	Sometimes it is hard for me to support the way parents raise their children.
		8b	Sometimes it is hard for me to support the way parents discipline their children.
		8c	Sometimes it is hard for me to support the goals parents have for their children.
		8d	Sometimes it is hard for me to work with parents who do not share my beliefs.
	Commitment and Caring
		9a	I teach and care for children because I enjoy it.
		9b	I see this job as just a paycheck.
		9c	I teach and care for children because I like being around children.
		9d	If I could find something else to do to make a living I would.
		10a	Part of my job is to help families get services available in the community
		10b	Part of my job is to offer parents information about community events.
		10c	Part of my job is to respond to issues or questions outside of normal care hours.
		10d	Part of my job is to change my work schedule in response to parents' work or school schedule.
		10g	Part of my job is to talk to parents about how they raise their children.
	Openness to Change
		6a	I am open to using information on new and better ways to teach and care for children
		10e	Part of my job is to learn new ways to teach and care for children.
		10f	Part of my job is to change activities offered to children in response to families’ feedback.
Knowledge
	Theoretical Knowledge
		1b	Since September, how often have you met with or talked to parents about their child's abilities?
		1c	Since September, how often have you met with or talked to parents about their child’s learning?
		1e	Since September, how often have you met with or talked to parents about what to expect at each stage of their child's development?
		1f	Since September, how often have you met with or talked to parents about how their child is progressing towards developmental milestones?
		11a	Have you ever received training or coursework on how to recognize signs of developmental delays in children?
		11b	Have you ever received training or coursework on how to recognize signs of child abuse and neglect?
		11c	Have you ever received training or coursework on how to recognize signs of domestic violence?
		11d	Have you ever received training or coursework on how to recognize signs of substance abuse?
		11e	Have you ever received training or coursework on how to recognize signs of depression or mental health issues in parents?
		11f	Have you ever received training or coursework on how to recognize signs of hunger?
	Knowledge Gathering
		3e	I know the parenting styles of children’s parents.
		3f	I know the employment status of children's parents.
		3h	I know the role that faith and religion play in children's households.
	Family-Specific Knowledge
		3a	I know if children have siblings.
		3b	I know if children have other adult relatives living in their households.
		3c	I know their parents’ schedules.
		3d	I know the marital status of children's parents.
		3g	I know their financial situation.
		3i	I know their cultures and values
		3j	I know what their families do outside of the education and care setting to encourage their children's learning.
		3k	I know how parents discipline their child.
		3l	I know changes happening at home.
		3m	I know health issues their children have such as food allergies or asthma.
Practices
	Relational Skills
		1a	Since September, how often have you met with or talked to parents about their child's experiences in the education and care setting?
		1d	Since September, how often have you met with or talked to parents about problems their child is having in the education and care setting?
		1g	Since September, how often have you met with or talked to parents about goals they have for their child?
		1h	Since September, how often have you met with or talked to parents about how their child is progressing towards the parents’ goals?
		4d	Since September, how often have you been able to suggest activities for parents and children to do together?
		5a	How often are you able to answer parents' questions when they come up?
		8a	When planning activities for children in your program how often are you able to take families' values, cultures, and beliefs into account?
		8b	When planning activities for children in your program how often are you able to take parents' ideas into account?
		8c	When planning activities for children in your program how often are you able to take information parents share about their children into account?
		12	How easy or difficult is it for families to reach you or someone at your program during the day if they have a question or if a problem comes up?
	Instrumental Support
		4a	Since September, how often have you been able to help children settle in when they are dropped off?
		4b	Since September, how often have you been able to share information with parents about their children's day?
		4c	Since September, how often have you been able to offer parents books and materials on parenting?
	Joint Goal Setting
		2a	Since September, how often have you met with or talked to parents about your expectations for the children in your care?
		2b	Since September, how often have you met with or talked to parents about the rules you have for children in your care?
		2c	Since September, how often have you met with or talked to parents about how they feel about the education and care you provide?
		5b	How often are you able to work with parents to develop strategies they can use at home to support their child's learning and development?
		5c	How often are you able to set goals with parents for their child?
		5d	How often are you able to offer parents ideas or suggestions about parenting?
		5e	How often are you able to provide parents the opportunity to give feedback on your performance?
	Empower Families
		6b	I encourage parents to provide feedback on my care and teaching practices.
		6c	I encourage parents to make decisions about their children’s education and care.
		6d	When it comes to their children, the parents are the experts.
		13a	Since September, have you encouraged families to seek or receive services?
		13b	Since September, have you made appointments or arrangements for families to receive services they need?
		13c	Since September, have you helped families find services they need?

Table A-2: Subscales in the Parent Survey

Content Area	Subscale	Item Number	Item Label
Attitudes
	Respect
		12a	My childcare provider or teacher judges my family because of our faith and religion.
		12b	My childcare provider or teacher judges my family because of our culture and values.
		12c	My childcare provider or teacher judges my family because of our race/ethnicity.
		12d	My childcare provider or teacher judges my family because of our financial situation.
	Commitment and Caring
		7b	My childcare provider or teacher is flexible in response to my work or school schedule.
		8a	My childcare provider or teacher is open to learning new ways to teach and care for my children.
		9a-k	My provider is: Caring, Understanding, Rude, Flexible, Dependable, Trustworthy, Impatient, Unfriendly, Respectful, Judgmental, Available.
		10	My childcare provider or teacher sees this job as just a paycheck.
Knowledge
	Theoretical Knowledge
		1b	Since September, how often have you met with or talked to your childcare provider or teacher about your child's abilities?
		1d	Since September, how often have you met with or talked to your childcare provider or teacher about your child's learning?
		1f	Since September, how often have you met with or talked to your childcare provider or teacher about what to expect at each stage of your child's development?
	Knowledge Gathering
		3f	How comfortable would or do you feel sharing with your childcare provider or teacher your employment status?
		3i	How comfortable would or do you feel sharing with your childcare provider or teacher the role that faith and religion play in your household?
	Family-Specific Knowledge
		3a	How comfortable would or do you feel sharing with your provider or teacher if your child has siblings?
		3b	How comfortable would or do you feel sharing with your provider or teacher if you have other adult relatives living in your household?
		3c	How comfortable would or do you feel sharing with your provider or teacher your household schedule?
		3d	How comfortable would or do you feel sharing with your provider or teacher your marital status?
		3e	How comfortable would or do you feel sharing with your provider or teacher your personal relationship with a spouse or partner?
		3g	How comfortable would or do you feel sharing with your provider or teacher your financial situation?
		3h	How comfortable would or do you feel sharing with your provider or teacher your family life?
		3j	How comfortable would or do you feel sharing with your provider or teacher your family’s culture and values?
		3k	How comfortable would or do you feel sharing with your provider or teacher what you do outside of the education and care setting to encourage your child’s learning?
		3l	How comfortable would or do you feel sharing with your provider or teacher how you discipline your child?
		3m	How comfortable would or do you feel sharing with your provider or teacher problems your child is having at home?
		3n	How comfortable would or do you feel sharing with your provider or teacher changes happening at home?
		3o	How comfortable would or do you feel sharing with your provider or teacher health issues your child has such as food allergies or asthma?
		6g	How often does your childcare provider or teacher remember personal details about your family when speaking with you?
Practices
	Relational Skills
		1a	Since September, how often have you met with or talked to your provider about your child's experiences in the education and care setting?
		1c	Since September, how often have you met with or talked to your provider about your child's behavior?
		1e	Since September, how often have you met with or talked to your childcare provider or teacher about goals you have for your child?
		4	If you had a problem with your provider, how comfortable would you feel talking to him or her about it?
		5c	How often does your childcare provider or teacher suggest activities for you and your child to do together?
		6a	How often does your childcare provider or teacher ask about your family?
		6c	How often does your childcare provider or teacher listen to your ideas about ways to change or improve the care and education your child receives?
		7e	My childcare provider or teacher uses my feedback to adjust the education and care provided to my child.
		7g	My childcare provider or teacher asks me questions to show he/she cares about my family.
		13	How easy or difficult is it for you to reach your childcare provider or teacher during the day if you have a question or a problem comes up?
		14	On a scale of 1-5, where 1 is the worst you can imagine and 5 is the best you can imagine, how would you describe your relationship with your childcare provider or teacher?
	Instrumental Support
		5a	How often does your childcare provider or teacher share information with you about your child’s day?
		5b	How often does your childcare provider or teacher offer you books or materials on parenting?
		5d	How often does your childcare provider or teacher ask you about the cultural values and beliefs you want him/her to communicate to your child?
		7d	My childcare provider or teacher tells me how my child is progressing towards goals or developmental milestones.
		11a	I trust that my childcare provider or teacher can maintain a safe environment for my child.
	Joint Goal Setting
		1g	Since September, how often have you met with or talked to your childcare provider or teacher about your vision for your child’s future?
		2a	Since September, how often have you met with or talked to your childcare provider or teacher about your provider's expectations for your child?
		2b	Since September, how often have you met with or talked to your provider about the rules your provider has for children in his or her care?
		2c	Since September, how often have you met with or talked to your provider about how you feel about the care and education your child receives?
		6b	How often does your childcare provider or teacher work with you to develop strategies you can use at home to support your child's learning and development?
		6e	How often does your childcare provider or teacher provide you with opportunities to make decisions about your child's education and care?
		6f	How often does your childcare provider or teacher provide you with opportunities to give feedback on his or her performance?
		7k	My childcare provider or teacher encourages parents to provide feedback on the way he/she cares for and teachers children
		8b	My provider and I work together to make sure my child has the best care and support.
		11b	I trust that my childcare provider or teacher has my child’s best interest at heart.
	Empower Families
		6d	How often does your childcare provider or teacher offer you ideas or suggestions about parenting?
		6h	How often does your childcare provider or teacher contradict you in front of your child?
		7a	My childcare provider or teacher respects me as a parent.
		7c	My childcare provider or teacher treats me like an expert on my child.
		7f	My childcare provider or teacher encourages me to be involved in all aspects of my child’s care and education.
	Cultural Sensitivity
		7h	My childcare provider or teacher reflects the cultural diversity of students in activities.
		7i	My childcare provider or teacher shows respect for different ethnic heritages.
		7j	My childcare provider or teacher is respectful of religious beliefs.
		7l	My childcare provider or teacher communicates the ethnic values and beliefs I want my child to have.

Table A3: Items that are parallel between the Parent and Provider Surveys

Content Area	Subscale	Parent Item Numbers	Parent Item	Provider Item Numbers	Provider Item
Attitudes
	Commitment and Caring
		10	My childcare provider or teacher sees this job as just a paycheck.	9b	I see this job as just a paycheck.
		7b	My childcare provider or teacher is flexible in response to my work or school schedule.	10d	Part of my job is to change my work schedule in response to parents' work or school schedules.
Knowledge
	Theoretical Knowledge
		1b	Since September, how often have you met with or talked to your childcare provider or teacher about your child's abilities?	1b	Since September, how often have you met with or talked to parents about their child's abilities?
		1d	Since September, how often have you met with or talked to your childcare provider or teacher about your child's learning?	1c	Since September, how often have you met with or talked to parents about their child's learning?
		1f	Since September, how often have you met with or talked to your childcare provider or teacher about what to expect at each stage of your child's development?	1e	Since September, how often have you met with or talked to parents about what to expect at each stage of their child's development?
	Knowledge Gathering
		3f	How comfortable would or do you feel sharing with your childcare provider or teacher your employment status?	3f	I know the employment status of children’s parents.
		3i	How comfortable would or do you feel sharing with your childcare provider or teacher the role that faith and religion play in your household?	3h	I know the role that faith and religion play in children’s households.
	Family-Specific Knowledge
		3a	How comfortable would or do you feel sharing with your provider or teacher if your child has siblings?	3a	I know if children have siblings.
		3b	How comfortable would or do you feel sharing with your provider or teacher if you have other adult relatives living in your household?	3b	I know if children have other adult relatives living in their households.
		3c	How comfortable would or do you feel sharing with your provider or teacher your household schedule?	3c	I know their parents’ schedules.
		3d	How comfortable would or do you feel sharing with your provider or teacher your marital status?	3d	I know the marital status of children’s parents.
		3g	How comfortable would or do you feel sharing with your provider or teacher your financial situation?	3g	I know their financial situation.
		3j	How comfortable would or do you feel sharing with your provider or teacher your family’s culture and values?	3i	I know their cultures and values.
		3k	How comfortable would or do you feel sharing with your provider or teacher what you do outside of the education and care setting to encourage your child’s learning?	3j	I know what their families do outside of the education and care setting to encourage their children’s learning.
		3l	How comfortable would or do you feel sharing with your provider or teacher how you discipline your child?	3k	I know how parents discipline their child.
		3n	How comfortable would or do you feel sharing with your provider or teacher changes happening at home?	3l	I know changes happening at home.
		3o	How comfortable would or do you feel sharing with your provider or teacher health issues your child has such as food allergies or asthma?	3m	I know health issues their children have such as food allergies or asthma.
Practices
	Relational Skills
		1a	Since September, how often have you met with or talked to your childcare provider or teacher about your child's experiences in the education and care setting?	1a	Since September, how often have you met with or talked to parents about their child's experiences in the education and care setting?
		1e	Since September, how often have you met with or talked to your provider about goals you have for your child?	1g	Since September, how often have you met with or talked to parents about goals they have for their child?
		5c	How often does your childcare provider or teacher suggest activities for you and your child to do together?	4d	Since September, how often have you been able to suggest activities for parents and children to do together?
		13	How easy or difficult is it for you to reach your provider during the day if you have a question or if a problem comes up?	12	How easy or difficult is it for families to reach you or someone at your program during the day if they have a question or if a problem comes up?
	Instrumental Support
		5a	How often does your childcare provider or teacher share information with you about your child's day?	4b	Since September, how often have you been able to share information with parents about their children's day?
		5b	How often does your childcare provider or teacher offer you books or materials on parenting?	4c	Since September, how often have you been able to offer parents books and materials on parenting?
	Joint Goal Setting
		2a	Since September, how often have you met with or talked to your childcare provider about your provider's expectations for your child?	2a	Since September, how often have you met with or talked to parents about your expectations for the children in your care?
		2b	Since September, how often have you met with or talked to your childcare provider about the rules your provider has for children in his or her care?	2b	Since September, how often have you met with or talked to parents about the rules you have for children in your care?
		2c	Since September, how often have you met with or talked to your childcare provider or teacher about how you feel about the care and education your child receives?	2c	Since September, how often have you met with or talked to parents about how they feel about the education and care you provide?
		6b	How often does your childcare provider or teacher work with you to develop strategies you can use at home to support your child’s learning and development?	5b	How often are you able to work with parents to develop strategies they can use at home to support their child’s learning and development?
		6f	How often does your childcare provider or teacher provide you with opportunities to give feedback on his or her performance?	5e	How often are you able to provide parents the opportunity to give feedback on your performance?
	Empower Families
		7c	My childcare provider or teacher treats me like an expert on my child.	6d	When it comes to their children, the parents are the experts.
		7f	My childcare provider or teacher encourages me to be involved in all aspects of my child’s care and education.	6c	I encourage parents to make decisions about their children’s education and care.

1 The idea is to determine if members of different groups have different item response functions. A logistic regression approach is recommended. See: Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.

21