1205-0359 Supporting Statement Part B_FINAL 2.16.17

Benefits Timeliness and Quality (BTQ) Review System

OMB Control No. 1205-0359

February 2017

SUPPORTING STATEMENT

Benefits Timeliness and Quality (BTQ)

Review System

Part B

OMB Control No. 1205-0359

2. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling

State Workforce Agencies (SWAs) use data collected through required Unemployment Insurance (UI) reports to measure and evaluate performance primarily through analyses of universes of UI workload and timeliness data. There are three areas, however, where it is necessary to select random samples to assess performance:

• adjudication quality reviews of separation issues

• adjudication quality reviews of nonseparation issues

• lower authority appeals quality reviews.

The BTQ collection of information is based solely on review of existing working SWA agency records and does not involve surveying individuals, administering questionnaires or any other form of survey research. Samples are selected quarterly from records reported on the ETA 9052 and ETA 9054 reports, which are censuses of nonmonetary determinations and lower authority appeals, respectively.

State samples are stratified by the type of adjudication, with equal numbers of separation issues and non-separation issues selected. The size of samples that states use to assess the quality of nonmonetary adjudications is dependent on the total number of adjudications that the state processes. States with fewer than 100,000 annual nonmonetary determinations are classified as small states, and the resulting annual sample sizes for both separation issues and non-separation issues are 240. States with 100,000 or more annual nonmonetary determinations are classified as large states, and the resulting annual sample sizes for both separation issues and non-separation issues are 400.

It should be noted that the classification of states as “large” or “small” only impacts the number of cases sampled -- “large” states will sample more cases than “small” states -- and does not produce separate strata. Within each state, however, the sample is stratified on separation issues and non-separation issues.

The size of the samples that states use to assess the quality of appeals is determined only by the total number of appeals that the state processes. States with fewer than 40,000 annual lower authority appeals are classified as small states and will sample 80 cases per year. States with 40,000 or more annual lower authority appeals are classified as large states and will use a sample of 160 cases per year. Appeals samples are not stratified.

Table showing Quality Review Universes and Samples

Universe Sizes from CY 2015

Items to Be Reviewed	Range of Annual Universe*	Annual Sample Size	Range of Quarterly Universe*	Quarterly Sample Size
Nonmonetary Adjudications (27 Small States)	15,590 – 97,052	240 (See note 1)	3,679 – 26,072	60 (See note 2)
Nonmonetary Adjudications (26 Large States)	107,091 – 1,016,233	400 (See note 3)	21,580 – 275,149	100 (See note 4)
Lower Authority Appeals (44 small states)	1,012 – 34,939	80	253 – 8,735	20
Lower Authority Appeals (9 large states)	40,005 – 209,001	160	10,041 – 52250	40

Notes:

* Excluding Virgin Islands. Source for nonmonetary (separation and nonseparation) adjudications is the ETA 9052 report. Source for lower authority appeals is the ETA 9054 report.

(1) Annual nonmonetary samples are stratified by separation (120 cases) and nonseparation (120 cases).

(2) Quarterly nonmonetary samples are stratified by separation (30 cases) and nonseparation (30 cases).

(3) Annual nonmonetary samples are stratified by separation (200 cases) and nonseparation (200 cases).

(4) Quarterly nonmonetary samples are stratified by separation (50 cases) and nonseparation (50 cases).

Both the adjudications and appeals quality measures have been designated as UI Performs Core Measures. Core Measures have national minimum performance criteria established, as opposed to individual SWA-negotiated criteria. Accurate estimates of quality are thus extremely important, but must be balanced with the costs of conducting the reviews. For this reason, BTQ has implemented varying sample sizes for both adjudications and appeals quality reviews based on the size of activity in each SWA.

Adjudication quality samples are stratified into separation and nonseparation issues to allow for separate estimates of quality. For SWAs where the total population of nonmonetary adjudications reported on the ETA 9052 report equaled or exceeded 100,000 in the prior CY, the quarterly sample size is 50 separation issues and 50 nonseparation issues. For SWAs where the population was less than 100,000, the quarterly sample size is 30 separation issues and 30 nonseparation issues.

Lower Authority Appeals quality samples are not stratified. The quality sample pool for the ETA 9057 report includes single and two party appeal hearings. The quality sample pool excludes redeterminations, withdrawals, dismissals, decisions with no hearings held, and episodic claims programs such as Extended Benefits, Emergency Unemployment Compensation, Disaster Unemployment Assistance, Trade Adjustment Assistance, and Trade Readjustment Allowances. For SWAs where the total population of lower authority appeals reported on the ETA 9054 report equaled or exceeded 40,000 in the prior calendar year, the quarterly sample size is 40 appeals decisions. For SWAs where the population was less than 40,000, the quarterly sample size is 20 appeals decisions.

SWAs can draw more than the minimum required quality review samples for adjudications, in order to obtain more precise information about how operational decisions impact quality.

Because these samples are drawn from universes of records maintained by SWAs, the response rates are 100% except for cases of lost documents. In CY 2015, 42 SWAs reported no nonmonetary adjudications with missing documentation. For the 10 SWAs reporting missing documentation, the number of adjudications missing documentation, among both separation and non-separation cases, ranged from 1 to 9. No state had higher than a 7.5% missing documentation rate. A table with detailed information on the incidence of missing information by state is provided in Section B-3.

2. Information Collection Procedures

All SWAs are expected to maintain computer files containing all adjudications. Stratifying the samples by separation and nonseparation issues is done on the basis of the adjudication issue code, which is stored with every adjudication record. It is thus straightforward for SWAs to extract two files containing all separation and nonseparation adjudications where the notice date falls within the quarter being sampled. SWAs then either 1) randomize the file using a computer program and select the first n records, with n equal to the quarterly sample of nonmonetary adjudications or lower authority appeals, as discussed in the previous section; or 2) draw a systematic sample by calculating a sampling (skip) interval (universe divided by desired sample size), selecting a random number between 1 and the interval number for the initial selection and then selecting every nth record to produce a systematic a random sample.

Not all SWAs have computerized records of Lower Authority Appeals decisions. SWAs that maintain automated records of appeals decisions draw the quarterly sample using the procedures (random file or systematic sampling method) described above. SWAs that do not maintain automated records of appeals decisions must use a manual systematic sampling approach by counting the total number of appeals decisions for the quarter, calculating the sampling interval and manually selecting the appropriate appeals decisions for review.

Minimum performance criteria for UI Performs Core Measures were published in UIPL 14-05, issued February 18, 2005. The nonmonetary quality standards are set at 75% of cases scoring 95 or more points. The lower authority appeals quality Core Measure minimum performance criterion is 80% of the cases scoring at least 85% of the potential points. Given the sample sizes and the target scores, the confidence intervals for the estimates of annual performance are as follows:

95 Percent Confidence Intervals for Estimated Percentages

Of Adjudications or Appeals Passing Quality Reviews

Measurement	Small States (Quarterly / Annual)	Large States (Quarterly / Annual)
Adjudications: Separation Issues Nonseparation Issues Universe of All Adjudications	75% + 15.8 / + 7.8 75% + 15.8 / + 7.8 75% + 11.0 / + 5.5	75% + 12.1 / + 6.0 75% + 12.1 / + 6.0 75% + 8.5 / + 4.2
Lower Authority Appeals	80% + 18.0 / + 8.8	80% + 12.6 / + 6.2

Note: The confidence intervals for nonmonetary adjudications assume simple random samples are selected and that 75 percent of the sample determinations receive passing scores (>= 95 points). The confidence intervals for lower authority appeals assume simple random samples are selected and that 80 percent of the sample determinations receive passing scores (>= 85 percent of the potential points). The 75 percent and 80 percent values are the minimum performance criteria established for these measures under UI Performs. Confidence intervals are expressed as + percentage points. See Appendix B-1 for the statistical formulas used to estimate annual population parameters and sampling variability.

Example: For a small state sample (30 cases) of separation adjudications, let:

n = the number of separation adjudications sampled for the quarter.

m = the number of completed sample separation adjudications for the quarter.

x = the number of separation adjudications in the quarter that meet the quality criterion of >= 95 points.

the proportion of separation adjudications in the quarter that meet quality criterion.

Assuming that the sampling fraction, f = n / N is negligible and that all sample cases were evaluated (that is, m = n -- no cases could not be evaluated due to missing documentation and no cases failed to meet the definition for inclusion in the population), then:

= (.75 * .25) / 29

= .1875 / 29

= .006466

The 95% confidence interval is:

= 1.96 * .0804

= .1576 or 15.8 percentage points.

Operational definitions of the populations of nonmonetary determinations are provided in ETA Handbook 301, 5th ed., chapter II, and sampling instructions are provided in Appendix A of the handbook. Operational definitions of the populations of lower authority appeals and sampling instructions are provided in ETA Handbook 382, 3rd ed., section II, and Appendix A.

Sampling frames are validated as part of UI Data Validation to insure that they include all of the records meeting the nonmonetary determinations and lower authority appeals operational definitions. The BTQ validation procedures are documented in the UI Data Validation Benefits Handbook, ET Handbook 361, Module 4.

3. Methods to Maximize Response Rates

The Department is aware of missing administrative records that could impact the effectiveness of BTQ and Appeals sampling. The data collection instruments record the incidence of missing records. For nonmonetary adjudications states must select additional sample cases in the subsequent quarter to make up for the cases that could not be scored because the case materials could not be found. For example, if during the review of a state’s 50 separation cases, 3 were identified as “case material not found” and therefore could not be evaluated for quality, the separation sample selected for the following quarter would be 53 cases. If 2 of the 50 nonseparation cases were identified as “case material not found” and not be evaluated for quality, the nonseparation sample selected for the following quarter would be 52 cases.

When a state enters its results to the database, a message will be generated stating that the scores for the quarter are inconclusive if either of two conditions is met:

1. If the total number of separation cases and/or the total number of nonseparation cases that are not scored because the case material cannot be found, or because they are outside the scope of this review, or because there is “no issue” exceeds 16.7% of either sample (separation or nonseparation) for small states and 25% of either sample (separation or nonseparation) for large states.

2. If the number of separation cases and/or the number of nonseparation cases that are not scored because the case material cannot be found exceeds 10% of the sample (separation or nonseparation). This 10% threshold for cases that are not scored because the case material cannot be found applies separately from the 16.7% / 25% thresholds for all nonscored cases.

The table below shows a state by state breakdown for CY 2015 of the incidence of cases in which material cannot be found and the sampled record may not be processed. Since this survey is drawn from a known and finite universe of administrative data, these cases represent the only instance that could be considered a non-response. This proxy for a non-response rate is computed below. No state reported greater than a 4.3% incidence of materials not found, precluding them from analysis. When the separation issue and non-separation issues are combined, no state had greater than a 4.3% incidence of materials not found, precluding them from analysis.

	Cases Disqualified for Analysis from				Cases Disqualified for Analysis from
	the Sample of Separation Records				the Sample of Non-separation Records
		Cases w/				Cases w/
	Sample	Material	Percent		Sample	Material	Percent
	Cases	Not Found	Missing		Cases	Not Found	Missing
AK	120	0	0.0%		120	0	0.0%
AL	120	0	0.0%		120	0	0.0%
AR	120	0	0.0%		120	0	0.0%
AZ	200	0	0.0%		200	0	0.0%
CA	200	0	0.0%		200	0	0.0%
CO	200	0	0.0%		200	0	0.0%
CT	120	0	0.0%		120	1	0.8%
DC	121	1	0.8%		121	1	0.8%
DE	121	1	0.8%		120	0	0.0%
FL	200	0	0.0%		200	0	0.0%
GA	200	0	0.0%		200	0	0.0%
HI	120	0	0.0%		120	0	0.0%
IA	120	0	0.0%		120	3	2.5%
ID	120	0	0.0%		120	0	0.0%
IL	200	0	0.0%		200	0	0.0%
IN	120	0	0.0%		120	0	0.0%
KS	120	0	0.0%		120	0	0.0%
KY	120	0	0.0%		120	0	0.0%
LA	210	3	1.4%		210	9	4.3%
MA	200	0	0.0%		200	0	0.0%
MD	200	0	0.0%		200	0	0.0%
ME	120	0	0.0%		120	0	0.0%
MI	200	0	0.0%		201	0	0.0%
MN	200	0	0.0%		200	0	0.0%
MO	200	0	0.0%		200	0	0.0%
MS	120	0	0.0%		120	0	0.0%
MT	120	0	0.0%		120	1	0.8%
NC	200	0	0.0%		202	3	1.5%
ND	120	0	0.0%		120	0	0.0%
NE	120	0	0.0%		120	0	0.0%
NH	129	0	0.0%		129	2	1.6%
NJ	200	0	0.0%		200	0	0.0%
NM	120	0	0.0%		120	0	0.0%
NV	120	0	0.0%		121	1	0.8%
NY	201	0	0.0%		200	0	0.0%
OH	200	0	0.0%		201	0	0.0%
OK	120	0	0.0%		120	0	0.0%
OR	200	0	0.0%		200	0	0.0%
PA	201	0	0.0%		201	0	0.0%
PR	120	0	0.0%		120	0	0.0%
RI	120	0	0.0%		120	0	0.0%
SC	200	1	0.5%		201	1	0.5%
SD	120	0	0.0%		120	0	0.0%
TN	120	0	0.0%		120	0	0.0%
TX	200	0	0.0%		200	0	0.0%
UT	202	0	0.0%		204	0	0.0%
VA	200	0	0.0%		200	0	0.0%
VT	120	0	0.0%		120	0	0.0%
WA	200	0	0.0%		200	0	0.0%
WI	200	0	0.0%		200	0	0.0%
WV	120	0	0.0%		120	0	0.0%
WY	120	0	0.0%		120	0	0.0%
US	8185	6	0.07%		8191	22	0.3%

For Appeals, a minimum random sample of 20 cases (40 for states with annual workloads greater than 40,000) are drawn and evaluated each quarter. These samples are not stratified.  If the state cannot produce its prescribed number of cases, the National Office does not permit the state to substitute cases.  Instead it instructs the state to score and evaluate the cases that it has and divide the final score by the actual number of cases retrieved.  This helps the National Office be aware of issues related to the state’s maintenance of its hearing records.  During the Annual Appeals Review, 10 random cases (20 for states with annual workloads greater than 40,000) are drawn and re-evaluated from the quarterly samples already reviewed by the state.  The same rule is applied when cases are missing.  Along with issues that arise from the review, states are notified of issues pertaining to their sample submission and the regional offices are engaged to address any systemic problems.   

The following table summarizes the number and percentage of CY 2015 lower authority appeals cases that could not be evaluated because either:1) tape Inaudible; 2) tape missing; 3) documents missing; 4) tape Inaudible and documents missing; or 5) tape and documents missing.

Cases Disqualified for Analysis
from Lower Authority Appeals
Samples
State	Sample	Missing	Percent Missing
AK	80	1	1.3%
AL	80	7	8.8%
AR	80	0	0.0%
AZ	80	2	2.5%
CA	160	4	2.5%
CO	80	0	0.0%
CT	80	0	0.0%
DC	80	0	0.0%
DE	80	0	0.0%
FL	160	4	2.5%
GA	160	0	0.0%
HI	80	2	2.5%
IA	80	0	0.0%
ID	80	1	1.3%
IL	160	0	0.0%
IN	80	2	2.5%
KS	80	2	2.5%
KY	80	2	2.5%
LA	80	1	1.3%
MA	80	5	6.3%
MD	80	0	0.0%
ME	80	0	0.0%
MI	80	0	0.0%
MN	80	0	0.0%
MO	80	1	1.3%
MS	80	1	1.3%
MT	80	1	1.3%
NC	80	1	1.3%
ND	80	4	5.0%
NE	84	1	1.2%
NH	80	8	10.0%
NJ	160	11	6.9%
NM	80	1	1.3%
NV	80	1	1.3%
NY	160	1	0.6%
OH	80	0	0.0%
OK	80	0	0.0%
OR	80	0	0.0%
PA	160	9	5.6%
PR	80	4	5.0%
RI	80	2	2.5%
SC	80	1	1.3%
SD	80	4	5.0%
TN	80	1	1.3%
TX	160	0	0.0%
UT	80	1	1.3%
VA	80	0	0.0%
VT	80	0	0.0%
WA	80	2	2.5%
WI	80	0	0.0%
WV	80	2	2.5%
WY	80	1	1.3%
US	4804	91	1.9%

The Department will continue to work with SWAs to correct administrative record control problems.

4. Tests of Procedures or Methods

The BTQ appeals and adjudications quality instruments are variations of instruments that have been in use for over twenty years. The revisions were subjected to a field test, which occurred in six SWAs over five quarters in 1993 and 1994. The data collection instrument for the ETA 9056 Nonmonetary Determination Quality Review is provided in ET Handbook 401, 4th ed., section V, chapter 6, p. 2. The data collection instrument for the ETA 9057 Lower Authority Appeals Quality Review is provided in ET Handbook 401, 4th ed., section V, chapter 7, pp. 2-3.

5. Individuals Consulted on Statistical Aspects of the Design

Daniel Sommers

Mathematical Statistician

Division of Performance Management

Office of Workforce Security

202-693-3197

Sommers.Daniel.J@dol.gov

Contact Information for the Collection

Ed Medlin

Unemployment Insurance Program Specialist

Division of Unemployment Insurance Operations

Office of Workforce Security

202-693-3259

Medlin.Edward@dol.gov

Appendix B-1

Equations for Population Parameter Estimates

The following formulas are used to produce the annual estimates of the proportions of nonmonetary adjudications and lower authority appeals meeting quality standards, based on the quarterly samples.

The following notation will be used:

H = the number of calendar quarters for which the estimate is

being made.

N_h = the number of adjudications or appeals in quarter h.

X_h = the number of adjudications or appeals in quarter h which meet the quality criteria.

P_h = X_h/N_h = the proportion of adjudications or appeals in quarter h which meet the quality criteria.

N_ = = total number of adjudications or appeals in the period.

X_ = = total number of adjudications or appeals which meet the quality criteria in the period.

The parameter to be estimated, P, is the proportion of adjudications or appeals that meet the quality criteria during the period. We wish to estimate:

P = X_/N_ =

Now let:

m_h = the number of completed sample adjudications or appeals for quarter h.

m_ = = total number of completed sample adjudications or appeals in the period.

x_h = the number of adjudications or appeals in quarter h which meet the quality criteria.

= proportion of adjudications or appeals in quarter h which meet the quality criteria.

If it is assumed that non-response is random, then

.

It follows that is unbiased for P.

Furthermore, as sampling is independent within each quarter (stratum), it follows that:

where f_h = m_h/N_h. The usual estimator for is

.

If f_h is negligible then

can be used for variance estimation.

Proportions for Subgroups

Samples of nonmonetary determinations and lower authority appeals may contain elements that do not meet the definition for inclusion in the population. These “foreign” elements are not included in the estimates of the proportion of the population meeting the quality criteria. The sample elements that meet the operational definition of the population constitute the sample subgroup for which population parameters are estimated.

Building on the notation above, for the k^th subgroup and the h^th week let:

N_hk = the number of adjudications or appeals.

X_hk = the number of adjudications or appeals which meet the quality criteria.

P_hk = X_hk/N_hk = the proportion of adjudications or appeals which meet the quality criteria.

Then for the k^th subgroup we have

N_k = = total number of adjudications or appeals in the quarter.

X_k = = total number of adjudications or appeals which meet the quality criteria in the quarter.

The parameter to be estimated, P_k, is the proportion of adjudications or appeals in subgroup k that meet the quality criteria during the quarter. Analogous to previous work, we can write

P_k = X_k/N_k = .

Note that neither X_k nor N_k is known. For the k^th subgroup, h^th quarter, let

m_hk = the number of completed sample adjudications or appeals for quarter h.

x_hk = the number of adjudications or appeals in quarter h which meet the quality criteria.

Assuming nonresponse is random:

is unbiased for X_k and

is unbiased for N_k.

The ratio estimator is approximately unbiased for P_k, and

where f_hk = m_hk/N_hk and θ_hk = N_hk/N_h.

Assuming that f_hk is negligible, an estimate for the variance is given by

where:

and

^{  .

Confidence Intervals

The 95% confidence interval for any estimate (u) is:

Coefficient of Variation

The coefficient of variation (cv) of an estimate u is:

14