U. Site-specific MDI Tables

Appendix U. Site-specific MDI Tables

This page has been left blank for double-sided copying.

Appendix U. Table B2.1.a. Colorado minimum detectable impacts based on administrative data

Test	Assumption	Total sample size	MDI (percentage points)
Test 1 (Denver, Broomfield, Larimer)	100 percent of anticipated sample	9,000	2.7 pp.
	75 percent of anticipated sample	6,750	3.2 pp.
	50 percent of anticipated sample	4,500	3.9 pp.
	15 percent of anticipated sample	1,350	7.1 pp.
Test 2 (Montrose)	100 percent of anticipated sample	2,500	4.0 pp.
	75 percent of anticipated sample	1,875	4.6 pp.
	50 percent of anticipated sample	1,250	5.7 pp.
	15 percent of anticipated sample	375	10.4 pp.

Note: Test 1 includes 5 research groups, while Test 2 includes 3 research groups. MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level, and are based on the following assumptions: (1) R-squared (the proportion of variation in the outcome explained by covariates) is 0.05; and (2) average take-up of SNAP E&T in the control group is 10 percent. (Colorado Department of Human Services estimates between 2 and 7 percent of work registrants participate in SNAP E&T across the four counties; we assume a higher take-up rate to be conservative.) Because we are calculating MDIs for a binary outcome, the MDI is highest for an outcome with a mean of 0.5 (since this would maximize the variance of the outcome). Assuming a higher take-up rate in this context (in which take-up is below 0.5) therefore increases the MDI, which is conservative because it requires a larger impact of the intervention to detect significant effects.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

Appendix U. Table B2.1.b. Connecticut minimum detectable impacts based on administrative data

Assumptions		MDI (percentage points)
	Total responding sample size	3 research groups	2 research groups
7-month enrollment period (July-January)	808	11.8	9.6
6-month enrollment period (July-December)	588	13.8	11.3
3-month enrollment period (July-September)	428	16.2	13.2

Note: MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level, and are based on the following assumptions: (1) an R-squared (the proportion of variation in the outcome explained by covariates) of 0.05, (2) a binary outcome with a mean value of 0.50, and (3) a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting. Sample sizes are based on estimates provided by Connecticut Community College leadership of the number of community college students enrolled in SNAP E&T from July 2023 to January 2024.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

.

Appendix U. Table B2.1.c. District of Columbia minimum detectable impacts based on administrative data

Pre-post analysis			RCT analysis
Total number of individuals in pre-intervention group	Total number of individuals in post-intervention group	MDI (percentage points)	Size of treatment and control groups	MDI (percentage points)
300	240	11.8	120 and 120	17.7
200	160	14.5	80 and 80	21.7
175	140	15.5	70 and 70	23.2
150	120	16.8	60 and 60	25.1

Note: The pre-post analysis compares a percentage outcome among the pre- and post-intervention groups. The size of the post-intervention group assumes that 20 percent of individuals in the pre-intervention group continue to participate in SNAP E&T in the post-intervention period; these individuals will be excluded from the analyses of outcomes in the post-intervention period due to exposure to both case management approaches. MDIs are based on a 0.80 power level and assume the mean value of the binary outcome is 0.50, baseline variables explain 5 percent of the variation in the outcome, response rates of 100 percent for outcomes based on administrative data, as well as a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

Appendix U. Table B2.1.d. Kansas minimum detectable impacts based on administrative data

	Based on starting sample size of 1,200
Assumption	Sample size for each group	MDI (percentage points)
All individuals (100 percent of starting sample)
Estimate effect of receiving behaviorally informed text message appointment reminders compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities	300 and 300	11.2
Estimate effect of receiving behaviorally informed text message nudges compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities	300 and 300	11.2
Estimate effect of receiving behaviorally informed text message appointment reminders and nudges compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities	300 and 300	11.2
Estimate effect of receiving behaviorally informed text message reminders or nudges compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities	900 and 300	9.1

Note: MDIs are calculated for a two-tailed test with 0.80 power level at a 0.05 significance level and assume: (1) the mean value of the binary outcome is 0.50, (2) baseline variables explain 5 percent of the variation in the outcome, (3) response rates are 100 percent for outcomes based on administrative data, as well as (4) the design effect is 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

Appendix U. Table B2.1.e. Massachusetts minimum detectable impacts based on administrative data

	Based on starting sample size of 30,000
Assumption	Sample size for each group	MDI (percentage points)
Compare text message treatment group 1 to text message treatment group 2 to estimate the effect of message content on expressing interest in learning more about E&T services	12,000 and 12,000	1.8
Compare (1) the combination of individuals who do not pass the survey screener and those who pass the screener and are placed in the control group for the assessment and (2) the text message control group to estimate the effect of the outreach message on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T1.)	9,000 and 6,000	2.3
Compare (1) the combination of individuals who do not pass the survey screener and those who pass the screener and are placed in the control group for the assessment and (2) the text message control group to estimate the effect of the outreach message on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T2.)	9,000 and 6,000	2.3
Compare (1) the combination of individuals in the assessment treatment group who were deemed not to be work ready and those in the assessment treatment group who were deemed to be work ready but were assigned to the career center control group and (2) the assessment control group, to estimate the effect of the assessment on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T1.)	2,100 and 3,000	3.9
Compare (1) the combination of individuals in the assessment treatment group who were deemed not to be work ready and those in the assessment treatment group who were deemed to be work ready but were assigned to the career center control group and (2) the assessment control group, to estimate the effect of the assessment on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T2.)	2,100 and 3,000	3.9
Compare the career center treatment group and the career center control group to estimate the effect of the warm handoff referral on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T1.)	900 and 900	6.4
Compare the career center treatment group and the career center control group to estimate the effect of the warm handoff referral on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T2.)	900 and 900	6.4

Note: MDIs are calculated for a two-tailed test with 0.80 power level at a 0.05 significance level. MDIs assume: the mean value of the binary outcome is 0.50, baseline variables explain 5 percent of the variation in the outcome, response rates of 100 percent for outcomes based on administrative data, as well as a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

.

Appendix U. Table B2.1.f. Minnesota-Hennepin minimum detectable impacts based on administrative data

Assumption	Total sample size	MDI (percentage points)
100 percent of projected sample	4,700	5.6
75 percent of projected sample	3,525	6.5
50 percent of projected sample	2,350	8.0

Note: Power calculations do not adjust for multiple comparisons. MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level and are based on the following assumptions: (1) R-squared (the proportion of variation in the outcome explained by covariates) is 0.05; (2) Average take-up of SNAP E&T in the control group is 50 percent; (3) there is no design effect from weighting because these outcomes will be based on administrative data.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

Appendix U. Table B2.1.g. Minnesota-Rural minimum detectable impacts based on administrative data

Total sample size	MDI (percentage points)
4,500	4.1
3,375	4.7
2,250	5.8
1,125	8.1

Note: MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level, and are based on the following assumptions: (1) an equal number of individuals are randomly assigned to the treatment and control groups (2) R-squared (the proportion of variation in the outcome explained by covariates) is 0.05; (3) Average take-up of SNAP E&T in the control group is 50 percent; (4) there is no design effect from weighting because these outcomes will be based on administrative data.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

Appendix U. Table B2.1.h. Rhode Island minimum detectable impacts based on administrative data

		Based on starting sample size of 5,000
Comparison	Main outcomes	Sample size for each group	MDI (percentage points)
All individuals (100 percent of starting sample)
Compare Treatment Group 2 (email web link) to control group to estimate the effect of message content on expressing interest in learning more about E&T services	Percentage of individuals who view the website Percentage of individuals who complete and submit contact form in website link Percentage of individuals who enroll in SNAP E&T	750 and 1,000	6.6
Compare Treatment Group 1 (text web link) to Treatment Group 2 (email web link) to estimate the effect of type of outreach messaging on outcomes	Percentage of individuals who view the website Percentage of individuals who complete and submit contact form in website link Percentage of individuals who enroll in SNAP E&T	750 and 750	7.1
Compare (1) the combination of individuals in Treatment Group 3 (text) who do not reply and those who do reply but are assigned to receive the existing assessment and (2) Treatment Group 1 (text web link) to estimate the effect of replying to outreach messages on outcomes	Percentage of individuals who complete and submit contact form in website link in the email or who reply to the text; percentage that completes form or responds within 1 month of receiving the message Percentage of individuals who enroll in SNAP E&T	700 and 750	7.2
Compare (1) the combination of individuals in Treatment Group 4 (email) who do not reply and those who do reply but are assigned to receive the existing assessment and (2) Treatment Group 2 (email web link) to estimate the effect of replying to outreach messages on outcomes	Percentage of individuals who complete and submit contact form in website link in the email or who reply to the text; percentage that completes form or responds within 1 month of receiving the message Percentage of individuals who enroll in SNAP E&T	700 and 750	7.2
Compare Treatment Group 3A (text) who receives enhanced assessment and Treatment Group 3B who receives current assessment to estimate the effect of the enhanced assessment on outcomes measuring whether individuals are a “better fit” with providers. (Both groups initially received a text message inviting them to reply “Yes” to learn more about E&T.)	Percentage of individuals who finish an assessment Percentage of individuals who are referred to a provider Percentage of individuals who start intake at a provider Percentage of individuals who stay with a provider Percentage of individuals who get assigned to a component Percentage of individuals who start a component Percentage of individuals who remain in the component for a certain period of time Percentage of individuals who experience an inter-provider referral Percentage of individuals who are referred to another provider or back to DHS	450 and 450	9.1
Compare Treatment Group 4A (email) who receives the enhanced assessment and Treatment Group 4B who receives the current assessment to estimate the effect of the enhanced assessment on outcomes measuring whether individuals are a “better fit” with providers. (Both groups initially received an email inviting them to reply “Yes” to learn more about E&T.)	Percentage of individuals who finish an assessment Percentage of individuals who are referred to a provider Percentage of individuals who start intake at a provider Percentage of individuals who stay with a provider Percentage of individuals who get assigned to a component Percentage of individuals who start a component Percentage of individuals who remain in the component for a certain period of time Percentage of individuals who experience an inter-provider referral Percentage of individuals who are referred to another provider or back to DHS	450 and 450	9.1

Note: MDIs are calculated for a two-tailed test with 0.80 power level at a 0.05 significance level. MDIs assume the mean value of the binary outcome is 0.50, baseline variables explain 5 percent of the variation in the outcome, response rates of 100 percent for outcomes based on administrative data, as well as a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.