Targeted Evaluations of State Vocational Rehabilitation (VR) Agency Practices

SURVEY OF VOCATIONAL REHABILITATION AGENCY
PRACTICES IN COOPERATIVE ARRANGEMENTS
AND QUALITY ASSURANCE

OMB NUMBER--TO BE ADDED
EXPIRES--DATE TO BE ADDED

Paperwork Burden Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a
collection of information unless such collection displays a valid OMB control number. The valid
OMB control number for this information collection is 1820-xxxx. The time required to complete
this information collection is estimated to average 90 minutes per response, including the time to
review instructions, search existing data resources, gather the data needed, and complete and
review the information collection. If you have any comments concerning the accuracy of the
time estimate(s) or suggestions for improving this form, please write to: U.S. Department of
Education, Washington, D.C. 20202-4537.
If you have comments or concerns regarding the status of your individual submission of
this form, write directly to: Steve Zwillinger, OSERS, U.S. Department of Education, 400
Maryland Avenue, S.W., PCP/ 5066, Washington, D.C. 20202-2800.

The Rehabilitation Services Administration (RSA) is supporting a 24-month study called
Targeted Evaluations of State Vocational Rehabilitation Agency Practices, conducted by RTI
International and InfoUse to collect information about vocational rehabilitation (VR) agency
practices in three areas. The three topics to be addressed include state agencies’ use of:
•

Quality assurance (QA) procedures used by VR agencies to ensure that they deliver
quality services in accordance with established policies and procedures. Quality
assurance may include case file reviews, staff training, use of data from case
management systems, input from staff and consumers, performance measurement
systems, strategic planning initiatives, performance-based budgeting systems, and a
variety of other procedures.

•

Third-party cooperative arrangements through which state agencies enter
partnerships involving funds transfers whose intent is to deliver specialized rehabilitation
services to target groups, such as transitional youth or individuals with severe and
persistent mental illness.

•

State-operated Comprehensive Rehabilitation Centers (CRCs), which provide
coordinated programs of vocational and medical rehabilitation to VR consumers in eight
states.

The study will allow analysts to examine the effects of these practices on VR program outcomes
and consumers served, and provide information to RSA that may be useful in its efforts to assist
state agencies in ensuring effective and efficient delivery of VR services. The study will also
identify promising practices, which could be adopted by other Centers and/or VR agencies.
As one component of the study, RTI/InfoUse is fielding a survey of state VR agencies to
examine current practices in two of the three areas under investigation: use of third-party
cooperative arrangements and quality assurance procedures in place in VR agencies. We are
requesting that your VR agency’s staff complete this survey in order to provide information that
may be useful to RSA in fulfilling its technical assistance and other management
responsibilities.
The survey contains two sections, as follows:
SECTION A: USE OF COOPERATIVE ARRANGEMENTS TO MEET MATCHING
REQUIREMENTS
The purpose of Section A is to obtain information on VR agencies’ use of cooperative
arrangements in order to meet federal match requirements. In addition to information on the
sources and amounts of funds used for match, this section requests information on the nature
and implementation of cooperative arrangements used for matching purposes. The section
requests information on the following topics:
1.
2.
3.
4.
5.
6.
7.

Sources of VR agency’s match for the 110 Program
Issues regarding availability of state-appropriated funds
Number and types of partner agencies
Waivers of statewideness
Administrative location of programs funded under cooperative arrangements
Monitoring and oversight of cooperative programs
Purposes and perspectives regarding cooperative programs

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

1

8. Characteristics of, and services provided to, consumers who participate in cooperative
programs
9. Outcomes
10. Order of Selection
Agencies whose percentage of match from cooperative arrangements is 10 percent or
more of their total match will complete all of Section A.
Agencies whose percentage of match from cooperative arrangements is less than 10 but
greater than 0 will complete only the following questions in Section A, then skip to
Section B: Questions 2, 3, 24, 25, 28, and 33-38.
Agencies whose percentage of match from cooperative arrangements is 0 will complete
only Question 1 in Section A, then skip to Section B.
SECTION B: QUALITY ASSURANCE ACTIVITIES
Section B of the survey focuses on a range of quality assurance (QA) activities and systems
that agencies may use to support continuous improvement of their services. Section B asks
(1) how QA activities have affected your agency’s performance on measures or indicators used
to monitor performance and compliance with applicable laws, policies, and procedures; (2) what
QA activities your agency has found to be effective and ineffective; and (3) whether your agency
has implemented any QA activities that you consider innovative or exemplary. Specific topics in
this section include:
1.
2.
3.
4.
5.
6.
7.
8.

Case file reviews
Case management systems
Strategic planning and performance measurement
Program evaluation and special studies
Personnel assignments, training, and performance evaluation
Management and monitoring of vendor (VR service provider) contracts
Input from consumers and other customers
Effects of QA activities and promising practices

Specific instructions and definitions appear throughout the survey.
We are requesting that you complete the questionnaire, which is in Word format, and return it
to RTI by (date). We suggest that you download the file, rename it, and print out a copy to
review before entering your responses in the Word document.
Please send the completed questionnaire electronically to VRPRACTICES@RTI.ORG.
Alternatively, you may print the form and mail it to Barbara Elliott, RTI International, 179
Johnson Bldg., PO Box 12194, RTP, NC 27709. Please retain a copy for your records.
When submitting your questionnaire electronically, please include requested attachments in the
same message if they are available electronically. If attachments are not available electronically,
please submit them separately by mail. Agencies submitting their questionnaire by mail should
include attachments in the same package.
If you have any questions about the survey, or exactly what information it requires, please do
not hesitate to call Barbara Elliott, Project Director, at 919/541-6313 (8:30 a.m. to 5:00 p.m.
Eastern time). Thank you for your assistance.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

2

SECTION A: USE OF COOPERATIVE ARRANGEMENTS TO MEET MATCHING
REQUIREMENTS
The purpose of Section A is to obtain information on VR agencies’ use of cooperative
arrangements in order to meet federal match requirements. In addition to information on the
sources and amounts of funds used for match, this section requests information on the nature
and implementation of cooperative arrangements used for matching purposes.
Complete Question 1 to describe your agency’s sources of matching funds for fiscal year (FY)
2007.
If the percentage of match from cooperative arrangements represents at least 10 percent of the
total as indicated in Question 1, complete the remainder of Section A.
If the percentage of match from cooperative arrangements is less than 10 but greater than 0,
complete only the following questions in Section A, then skip to Section B: Questions 2, 3, 24,
25, 28, and 33-38.
If the percentage of match from cooperative arrangements is 0, complete only Question 1 in
Section A and then skip to Section B.
Your responses to the remainder of Section A should refer only to programs funded through
cooperative arrangements used for matching purposes.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

3

DEFINITIONS OF CATEGORIES OF MATCHING FUNDS
Source: Rehabilitation Services Administration
Political Subdivisions: These are funds transferred from a political subdivision (such as a city or county) for the
discretionary use of the VR agency in order to provide services to the political subdivision’s VR eligible
population, usually its employees. (Do not include in this category transfers of funds from other state agencies.)
Transfers from Other State Agencies: In general, these are agreements where a state agency provides
matching funds to the VR agency for serving VR-eligible consumers from the other state agency. The
agreement may require the VR agency to dedicate counselors to serving the other state agency’s consumers,
or include other such requirements, but under no condition will the other agency provide VR services to VR
consumers.
These agreements are not to be confused with third-party cooperative arrangements where another state
agency or a local public agency is providing or administering VR services to the VR agency’s consumers.
If the transfer of matching funds is through a third-party cooperative arrangement (as described below), report
those funds under third-party cooperative arrangements, not as transfers from other state agencies.
Cooperative Arrangements: Section 34CFR 361.28 is the best reference for an explanation of what constitutes
cooperative arrangements wherein a VR agency enters into an arrangement with another state agency or a
local public agency that is furnishing part or all of the non-federal share of matching funds in exchange for
providing VR services to applicants for, or recipients of, services from the state VR agency under the VR
program.
The services provided by the cooperating agency must be new services (to the agency) that have a vocational
rehabilitation focus or existing services that have been modified, adapted, expanded, or reconfigured to have a
vocational rehabilitation focus. The services provided by the cooperating agency are only available to
applicants for, or recipients of, services from the VR agency. The expenditures for, and staff providing, services
under the cooperative arrangement are under the administrative supervision of the VR agency and all state
plan requirements, including a state’s Order of Selection, apply to all services provided under the cooperative
arrangement.
If the cooperative arrangement does not comply with the requirements of 34 CFR 361.25 related to
statewideness, then a waiver of statewideness must be obtained in accordance with 34CFR 361.26.
Matching Funds for Establishment Projects: The regulatory requirements for establishment projects providing
VR services for groups of individuals with disabilities are found in section 34 CFR 361.49. They entail “the
establishment, development, or improvement of a public or other nonprofit community rehabilitation program
that is used to provide vocational rehabilitation services that promote integration and competitive employment,
including, under special circumstances, the construction of a facility for a public or nonprofit community
rehabilitation program.” The match rate for establishment projects is 21.3%; the match rate for construction
projects is 50%.
Section 34 CFR 361.5(b)(17)(ii) describes how federal financial participation is available at the applicable
matching rate for levels of staff costs that gradually decline over a maximum period of four years.
Section 34 CFR 361.60 (b)(3)(i) provides the authority for using contributions from private entities to establish
community rehabilitation programs or construct a facility for community rehabilitation program purposes.
Gifts and bequests: These are discussed under 34 CFR 361.60 (b)(3) - Contributions by private entities. The
main caveat with these funds is that they must not benefit the donor in any way. See 34 CFR 361.60 (b)(3)(iii).
Randolph Sheppard Set-Aside: This matching source comes from the expenditures made by “set-aside” funds
of the Randolph Sheppard program (34CFR395.9) for which federal financial participation under the Basic
Support Program is allowed: management services, initial stocks and supplies, operational costs during the
initial establishment period not to exceed 6 months, as well as the purchase and repair of equipment.
Surface Transportation Act Funds: These are funds collected by state VR agencies administering the
Randolph Sheppard program from vending machines in rest areas on the interstate highway system.
State-Appropriated Funds: These are funds allocated to the state VR agency by state appropriation. For some
state VR agencies this is the only source of matching funds and nationwide this represents the biggest source
of matching funds. This includes any expenditures of state funds made on behalf of the state VR agency from a
central account maintained by the state’s Department of Treasury for the purpose of paying expenses common
to all state agencies, such as fringe benefits, indirect costs, etc.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

4

Section B: Quality Assurance Activities
Section B of the survey focuses on a range of quality assurance (QA) activities and systems
that agencies may use to support continuous, or ongoing, improvement of their services. The
activities may include, but are not limited to:
•
•
•
•
•
•
•
•

Case file reviews
Case management systems
Strategic planning and performance measurement
Program evaluation and special studies
Personnel assignments, training, and performance evaluation
Management and monitoring of vendor (VR service provider) contracts
Input from consumers and other customers
Effects of QA activities and promising practices

Section B also asks (1) how QA activities have affected your agency’s performance on
measures or indicators used to monitor performance and compliance with applicable laws,
policies, and procedures; (2) what QA activities your agency has found to be effective and
ineffective; and (3) whether your agency has implemented any QA activities that you consider
innovative or exemplary.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

5

SECTION A: USE OF COOPERATIVE ARRANGEMENTS
TO MEET MATCHING REQUIREMENTS
SOURCES OF AGENCY MATCH FOR 110 PROGRAM
1. Please complete the following table regarding your agency’s match sources for FY
2007. Definitions of each source of match are in the instructions. Enter whole
numbers with no dollar signs or commas.

Source of match

Amount

Number of
arrangements/
projects

a. Political subdivisions
b. Transfers from other state agencies
c. Cooperative arrangements
d. Matching funds for establishment projects
e. Gifts and bequests
f. Randolph Sheppard set-aside
g. Surface Transportation Act Funds
h. State appropriated funds
The total of your matching funds from
all sources, automatically calculated, is
shown below.
$0
Your percentage of funds from
cooperative arrangements,
automatically calculated, is shown
below.
If the percentage is 10 or more,
complete all of Section A.
If the percentage is less than 10 but
greater than 0, complete only the
following questions in Section A, then
skip to Section B: Questions 2, 3, 24,
25, 28, and 33-38.
If the percentage is 0, skip to Section B.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

6

ISSUES REGARDING AVAILABILITY OF STATE-APPROPRIATED FUNDS
2. Please briefly describe how your state’s economic situation affects the availability of
state-appropriated funds for meeting VR matching requirements.

3. Do you anticipate future changes that may free up state funds for matching?
Yes

No

Please explain why or why not.

NUMBER AND TYPES OF PARTNER AGENCIES
4. Please complete the following table regarding the number of state-level and locallevel cooperative arrangements by type of partner agency and total funds received
through each type of arrangement. Enter whole numbers with no dollar signs or
commas.
NOTE: The total number of state and local level arrangements reported here should equal the number of
arrangements reported in Question 1.c. The total amount of state and local funds reported here should equal the
amount reported in Question 1.c.

Agency type
a. State education agency
b. Local education agency
Postsecondary institution
c. 4-year college/university
d. 2-year college
e. Vocational/technical
institute
f. State mental health agency
g. Local mental health agency
h. Developmental disabilities
council
i. Corrections agencies
j. Other (specify)
k. Other (specify)
l. Other (specify)
m. Other (specify)
n. Other (specify)
o. Other (specify)
p. Total

State level
Number of
arrangements Total funds

Local level
Number of
arrangements Total funds

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

7

5. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

WAIVERS OF STATEWIDENESS
6. How many of your cooperative arrangements are operating under a waiver of
statewideness (i.e., a waiver of the requirement to offer services in all areas of the
state)? (Please use information from your most recent state plan, with updates as
approved by RSA.)

7. If you need to explain or add any comments concerning your response to the question
above, please use this space.

ADMINISTRATIVE LOCATION OF PROGRAMS FUNDED UNDER COOPERATIVE
ARRANGEMENTS
8. What division of the agency has administrative responsibility for programs funded
under cooperative arrangements?

9. Does the agency have staff designated for this purpose?
Yes

No

(IF NO, SKIP TO Q 11)

If yes, please indicate:
a. Number of full-time staff:
b. Number of part-time staff:

10. Where are these staff located? (Total should equal the number reported in Q9.)
a. Number in central office:
b. Number in field offices:

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

8

MONITORING AND OVERSIGHT OF COOPERATIVE PROGRAMS
11.

What types of performance requirements apply to cooperative arrangements
programs in your agency?
(Check all that apply)
Reasonable costs of services in comparison with other providers of like services
Costs per person served
Costs per outcome
Targets for numbers served
Targets for number of employment outcomes
Targets for number of other appropriate outcomes (specify):
Other, specify

12. Does your agency negotiate these performance requirements?
Yes

No

If yes, how often?

13.

What types of oversight activities do you use for monitoring the operations and
outcomes of programs implemented under cooperative arrangements?
(Check all that apply)
Regularly scheduled monitoring visits
Specialized on-site performance reviews
Expenditure reports:

Monthly

Quarterly

Other (specify)

Performance reports:

Monthly

Quarterly

Other (specify)

Reports on consumer outcomes
Analysis of outcomes of these consumers compared to outcomes of consumers in other similar
programs
Case reviews
Formal evaluations
Regular meetings with partner agencies
Other, specify

14. Do these oversight activities apply to all programs operating under cooperative
arrangements?
Yes

No

If no, explain briefly:

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

9

15. Does your agency use established criteria to determine whether contract
performance is adequate to justify renewal of programs operated under cooperative
arrangements?
Yes

16.

No

(SKIP TO Q 17)

What criteria does your agency use for this purpose?
(Check all that apply)
Achievement of performance targets in terms of number of consumers served
Achievement of performance targets in terms of number of employment placements or other
specified successful outcomes
Achievement of performance targets in terms of expenditures
Cost efficiency in comparison with similar programs not operated under a cooperative
arrangement
Other (specify)

17. Has your agency identified cooperative programs whose performance is not
adequate?
Yes

18.

No

(SKIP TO Q 20)

If so, what actions do you take if performance is not adequate based on established
requirements?
(Check all that apply)
Renegotiate performance requirements
Implement corrective measures (specify)
Place the program on notice or probation
Terminate the contract
Other (specify)

19.

Please indicate whether you have implemented any of the actions above in the past
three years. (Check “yes” or “no” for each item)
a. Renegotiated performance requirements ....

Yes

No

b. Implemented corrective measures ...............

Yes

No

c.

Placed a program on notice or probation .....

Yes

No

d. Terminated a program’s contract ................

Yes

No

e. Other (specify)

Yes

No

....................................

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

10

20.

Under what conditions do you conduct evaluations of all cooperative programs or a
group of cooperative programs (for example, programs operated in postsecondary
institutions, programs serving persons with mental illness)?
If the agency does not conduct such evaluations check here

and skip to Q 21.

a. Please describe the conditions under which you conduct such evaluations.
b. Please describe the purpose of such evaluations.
c. How often do you conduct such evaluations?
d. How do you use the results? (Briefly describe)

21. If you need to explain or add any comments concerning your responses to any
question in the “Monitoring and Oversight of Cooperative Programs” section, please
use this space.

PURPOSES AND PERSPECTIVES REGARDING COOPERATIVE
ARRANGEMENTS
22.

Please indicate which of the following describe the purposes of your agency’s
cooperative arrangements contracts.
(Check all that apply)
To meet federal matching requirements
To increase the agency’s ability to deliver vocational rehabilitation services to eligible individuals
with disabilities
To increase the agency’s ability to deliver VR services to individuals with particular types of
impairments or needs (e.g., transitional youth, persons with severe mental illness)
To improve the quality of services to target groups through coordination with other relevant
agencies (e.g., education agencies, mental health commissions)
Under waivers, to expand substantially the number of persons whom the agency can serve
Other (specify)

23.

In your view, which of these purposes is the single most important justification for
your agency’s third-party cooperative arrangements?
(Check one)
To meet federal matching requirements
To increase the agency’s ability to deliver vocational rehabilitation services to eligible individuals
with disabilities
To increase the agency’s ability to deliver VR services to individuals with particular types of
impairments or needs (e.g., transitional youth, persons with severe mental illness)
To improve the quality of services to target groups through coordination with other relevant
agencies (e.g., education agencies, mental health commissions)
Under waivers, to expand substantially the number of persons whom the agency can serve
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

11

24. Please briefly describe your perspectives on the advantages of delivering VR
services through the mechanism of third-party cooperative arrangements.

25. Please briefly describe the chief disadvantages of delivering VR services through the
mechanism of third-party cooperative arrangements.

CHARACTERISTICS OF, AND SERVICES PROVIDED TO, CONSUMERS WHO
PARTICIPATE IN COOPERATIVE PROGRAMS
This section requests demographic, disability, and services information on your agency’s
consumers who participate in cooperative programs.

CONSUMER CHARACTERISTICS
26.

Please indicate the number of consumers in each of the categories below. Include
participants in all of your cooperative programs (all programs funded under
arrangements reported in Question 4). Totals for consumers by primary disability,
significance of disability, age, ethnicity, and educational attainment should be equal.

Characteristic
Primary disability
a. Visual impairment
b. Physical impairment
c. Communicative disorder
d. Cognitive disorder
e. Mental or emotional (psychosocial) disability
f. Unknown
g. Total
Significance of disability
h. Most significant
i. Significant
j. Not significant
k. Unknown
l. Total
Age
m. 24 or under (transitional youth)
n. 25-50
o. Over 50
p. Unknown
q. Total

Consumers in
cooperative programs
in FY 2007

Number receiving SSI/SSDI
Race
r. White
s. Black or African American

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

12

t. American Indian or Alaska Native
u. Asian
v. Native Hawaiian or other Pacific Islander
w. Unknown
Ethnicity
x. Hispanic or Latino
y. Not Hispanic or Latino
z. Unknown
aa. Total
Educational attainment
bb. Less than high school
cc. High school diploma/GED
dd. Some postsecondary
ee. Postsecondary vocational credential
ff. Postsecondary academic credential
gg. Unknown
hh. Total

27. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

28. Does your agency have information available on the statewide distribution of the
population by disability type and significance?
Yes

No

If yes, please attach or indicate where RTI can obtain, this information.

CONSUMER SERVICES
29.

Please indicate the number of consumers participating in cooperative programs
during FY 2007 who received each service listed below.

Service
a.
b.
c.
d.
e.
f.
g.
h.
i.
j.
k.

Consumers in
cooperative programs
in FY 2007

Postsecondary occupational/vocational training
College/university training
Secondary occupational/vocational training
Support services
Secondary education
Work experience
Career development
Basic/remedial education
Job-related services (e.g., readiness, Job-search, placement)
Medical services
Psychological services

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

13

l. Counseling and guidance
m. Supported employment

30. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

OUTCOMES
31.

These tables request information on outcomes achieved by individuals who
received services from a program funded under cooperative arrangements in FY
2007 and who left VR services during the fiscal year.
Number of consumers
closed from cooperative
programs in FY 2007

Outcome
Employment outcome
a. Competitive employment
b. Supported employment
c. Self-employment
d. Business Enterprise Program
e. Homemaker/unpaid family worker
f. No employment outcome
g. Total

Consumers closed from
cooperative programs in FY 2007
who achieved employment outcomes
Average weekly
earnings

Outcome
a.
b.
c.
d.

Average
hours/week

Competitive employment
Supported employment
Self-employment
Business Enterprise Program

32. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

ORDER OF SELECTION
33. Does your agency currently operate under an Order of Selection (OOS)?
Yes

No (IF NO, SKIP TO SECTION B)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

14

34. How many years have you operated under OOS?
35. What has been the size of the waiting list over the past few years? (If there has not
been a waiting list, please enter “0.”)
FY 2005:
FY 2006:
FY 2007:

36. Do you anticipate that the waiting list will increase in FY 2008?
Yes

No

If yes, by approximately how many individuals?

37. Has operation under OOS affected your ability to operate cooperative programs due
to the requirement to first serve persons with most significant disabilities?
Yes

No

If “yes,” please explain how. If “no,” please explain why not.

38. Has operation of cooperative programs affected your OOS status?
Yes

No

If “yes,” please explain how.
If “no,” please explain why not.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

15

SECTION B: QUALITY ASSURANCE ACTIVITIES
CASE FILE REVIEWS
1. Which of the following topics are addressed by your agency’s case file reviews? (If
your agency does not conduct case file reviews, check here
and skip to Q13.)
(Check all that apply)
Adequacy of documentation
Timeliness of service delivery
Appropriateness of services
Review of fiscal status
Identification of promising practices
Compliance:
With federal laws
With state laws
With VR policies and procedures
With fiscal policies
Evaluation:
Of regional performance
Of local office performance
Of individual counselor performance
Identification of needs:
For changes in agency policies
For specialized or targeted services
For increased availability of certain services
For staff training
Other:
Other (specify)

2. Does your agency have a formal protocol for conducting case file reviews?
Yes

No

If yes, please attach a copy to the completed survey.

3. Please indicate whether central office, regional, or local staff have primary
responsibility for your agency’s case file reviews.
(Check one)
Central office staff
Regional or district managers
Local office managers
Local supervising counselors
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

16

4. How often do these staff members conduct case file reviews?
(Check one)
Quarterly
Biannually (twice a year)
Annually
Frequency varies by location
Other (specify)

5. How many case files were included in the most recent review conducted by these
staff members? If the number varies by location, please check here
and provide
an estimated or average number.
cases

6. How were case files selected for the most recent review conducted by these staff
members? (Check all that apply)
Sample of closed cases
Sample of cases at certain status codes (specify)
Sample of individual counselors’ cases
Sample of cases receiving certain services (specify)
Other (specify)

7. Please indicate whether any additional staff members (other than those identified in
Question 3) conduct separate case file reviews. If not, check here
and skip to
Q11. (Check all that apply)
Central office staff
Regional or district managers
Local office managers
Local supervising counselors
Other (specify)

8. How often do these additional staff members conduct case file reviews?
(Check one)
Quarterly
Biannually (twice a year)
Annually
Frequency varies by location
Other (specify)

9. How many cases were included in the most recent review conducted by these
additional staff members? If the number varies by location, please check here
provide an estimated or average number.

and

cases

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

17

10.

How were cases selected for the most recent review conducted by these additional
staff members? (Check all that apply)
Sample of closed cases
Sample of cases at certain status codes (specify)
Sample of individual counselors’ cases
Sample of cases receiving certain services (specify)
Other (specify)

11. In the recent past, has your agency identified specific areas for improvement on the
basis of case file reviews?
Yes

No

If yes, please list the specific area(s):

12. If you need to explain or add any comments concerning your responses in the “Case
File Reviews” section, please use this space.

CASE MANAGEMENT SYSTEMS
13.

Does your agency use a commercially developed case management system (such
as AWARE, CRIS, or VR System 6) or an internally developed system?
(Check one)
AWARE
CRIS
VR System 6
Other commercially developed system (specify)
Internally developed system
Not applicable—do not use commercially or internally developed system

14.

Please indicate whether your case management system:
(Check all that apply)
Allows state administrators to monitor performance of local offices and counselors
Allows local administrators to monitor the performance of individual counselors
Allows counselors to monitor their progress toward performance goals
Provides counselors with reports needed for managing casework
Allows administrators to create ad-hoc reports
Allows administrators to monitor time in status
Includes edit checks that allow cases to move from one status to another only under certain
conditions

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

18

Includes edit checks for out-of-range responses (i.e., responses that do not fall within an
expected range) to critical items
Requires validation (e.g., supervisor approval) of out-of-range responses to critical items
Permits monitoring of office- or counselor-level budgets

15.

What methods does your agency use to verify the accuracy of data reported
through the case management system?
(Check all that apply)
Comparison with hard-copy records
Verification by local supervisors
Documentation provided by employers or consumers
Edit and range checks
Other (specify)

16. If you need to explain or add any comments concerning your responses in the “Case
Management Systems” section, please use this space.

STRATEGIC PLANNING AND PERFORMANCE MEASUREMENT
17. Does your agency have established strategic goals?
Yes

No

If yes, please attach a list of the goals to the completed survey.

18.

Please indicate whether the goals were established by:
(Check one)
The VR agency itself
The DSA or a parent agency
A combination of the two
The state legislature
Other (specify)

19.

How often your does your agency monitor progress toward these goals?
(Check one)
Monthly
Quarterly
Biannually (twice a year)
Annually
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

19

20.

What performance measures or indicators does your agency use to monitor
performance?
(Check all that apply)
Federal standards and indicators
Other measures or indicators developed by the VR agency itself
Other measures or indicators developed by the DSA or a parent agency
Other (specify)

If you use measures other than the federal standards and indicators, please
attach a list of the measures to the completed survey.
21.

For which of the following purposes does your agency use performance data?
(Check all that apply)
Evaluating performance on specific measures
Evaluating progress toward strategic goals
Analyzing consumer characteristics
Analyzing services provided
Analyzing fiscal issues
Identifying promising practices
Case flow forecasting and analysis
Other (specify)
Please give an example of uses of performance data (optional)

22.

In the recent past, has your agency identified specific areas for improvement
through use of strategic goals or performance measures?
Yes

No

If yes, please list the specific area(s):

If yes, please provide an example, including a detailed description, of how areas for
improvement were identified (optional)

23.

Has your agency established standards for time in status (other than those
specified in federal regulations); e.g., the amount of time a case should remain at
status 02 (applicant) or status 10 (IPE development)?
Yes

No

If “yes,” please describe:

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

20

24.

Does your agency operate under a performance-based budgeting system? (A
performance-based budgeting system is defined as one in which agency budgets are
linked to performance goals.)
Yes

No

25. If you need to explain or add any comments concerning your responses in the
“Strategic Planning and Performance Measurement” section, please use this space.

PROGRAM EVALUATION AND SPECIAL STUDIES

26.

Has your agency recently completed, or is it currently conducting, a formal
evaluation of the VR program as a whole?
Yes

27.

No (IF NO, SKIP TO Q 28)

If yes, please indicate the type(s) of evaluation conducted.
(Check all that apply.)
Evaluation of program implementation (process evaluation)
Evaluation of consumer outcomes
Trends analysis
Analysis of cost or efficiency
Management analysis
Analysis of organizational effectiveness
Other (specify)

Please attach a report of a recently completed evaluation (optional).

28. Has your agency recently completed, or is it currently conducting, a special study
(or studies) on specific topics?
Yes

No (IF NO, SKIP TO Q 30)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

21

29.

If yes, please indicate the topic(s) studied.
(Check all that apply)
Specific services:
Supported employment
Transition services
Services for individuals with certain types of disabilities
Specify:
Services provided through cooperative program(s)
Services provided by CRPs
Other services (specify)
Other topics:
Accuracy of RSA-911 data
Timeliness of service delivery
Implementation of agency policies and procedures
Other (specify)

30.

In the recent past, has your agency identified specific areas for improvement on the
basis of formal evaluations or special studies?
Yes

No

If yes, please list the specific area(s):
31. If you need to explain or add any comments concerning your responses in the
“Program Evaluation and Special Studies” section, please use this space.

PERSONNEL ASSIGNMENTS, TRAINING, AND PERFORMANCE EVALUATION
32.

Does your state VR agency staff include individuals whose time is devoted
primarily to quality assurance activities?
Yes

33.

No (IF NO, SKIP TO Q 35)

In what office are these staff members housed?
(Check all that apply)
DSA
VR agency director’s office
VR agency field services director’s office
Planning or evaluation division
Statistics/MIS division
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

22

34. Please indicate the number of FTEs devoted primarily to quality assurance activities:
FTEs

35.

On what topics related to quality assurance does your agency provide field staff
with training? (Check all that apply)
Agency performance measures or strategic goals
Agency policies and procedures
Effective practices
Problems identified in case reviews or data analysis
Other topics related to quality assurance (specify)

36.

Which of the following indicators does your agency consider in counselor
performance appraisals? (Check all that apply)
Number of employment outcomes
Cost per employment outcome
Number of consumers served
Cost per consumer served
Consumer satisfaction
Other performance indicators (specify)

37. If you need to explain or add any comments concerning your responses in the
“Personnel Assignments, Training, and Performance Evaluation” section, please use
this space.

MANAGEMENT AND MONITORING OF VENDOR (VR SERVICE PROVIDER)
CONTRACTS

38. Has your agency established guidelines concerning reasonable service costs (e.g.,
price sheets, market comparisons, approved ranges of costs) for VR vendors
(defined as providers of VR services)?
Yes

No

Please attach a copy.

39. Approximately how many vendors do business with your agency?
Vendors

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

23

40. Does your agency make performance reports or ratings for individual vendors (e.g.,
vendor “report cards”) available to consumers?
Yes

No

41. Does your agency use performance-based contracts for vendors?
Yes

No (IF NO, SKIP TO Q 43)

Please attach a copy.
42. Does your agency use a standard format or template for these contracts?
Yes

No

If yes, please attach a copy.

43.

Please indicate whether your requirements for vendors include:
(Check all that apply)
Detailed descriptions of services to be provided
Requirement for accreditation by a national organization (e.g., CARF, JCAHCO, NAC, ASHA,
ACVREP, as appropriate)
Use of specific fiscal and/or administrative procedures
Use of specific referral, intake, and/or orientation procedures
Submission of regular performance reports to the state VR agency
Prescribed format for, or electronic submission of, invoices
Staffing requirements (academic or other training, certification, or licensing)
Other (specify)

44. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

45.

How does your agency obtain consumer feedback on vendors’ services?
(Check all that apply)
Surveys conducted by state VR agency
Surveys conducted by State Rehabilitation Council
Surveys conducted by vendor
Focus groups
Counselor must contact consumer before invoice is paid
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

24

46. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

47.

How does your agency monitor the performance of vendors?
(Check all that apply)
Monitoring visits by VR agency staff
Central office staff analyzes reports submitted by vendors
Local office staff analyzes reports submitted by vendors
Externally contracted audits
Other (specify)

48. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

INPUT FROM CONSUMERS AND OTHER CUSTOMERS
49.

To what extent is your State Rehabilitation Council involved in evaluation of VR
agency performance?
(Check one)
Not involved

(PLEASE ENTER ANY COMMENTS, THEN SKIP TO Q 53.)

Limited extent
Moderate extent
Great extent

50. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

51.

In the recent past, what role has the State Rehabilitation Council played in
evaluation of VR agency performance?(Check all that apply)
Suggested areas for evaluation
Helped carry out evaluation activities (e.g., helped design instruments, conducts focus groups)
Reviewed evaluation results
Disseminated evaluation results to legislature, advocacy groups, or public
Made specific recommendations for changes in agency policy or practice based on evaluation
results
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

25

52. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

53.

Please indicate whether your State Rehabilitation Council is involved in the
following aspects of consumer satisfaction surveys.
(Check all that apply)
Developing the survey instrument
Conducting the survey
Analyzing survey data
Other (specify)

54. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

55.

Which of the following methods does your agency use to obtain input from
consumers and other customers?
(Check all that apply)
Consumer satisfaction survey of:
Applicants
Individuals in extended evaluation or plan development
Individuals receiving services under an IPE
Individuals closed rehabilitated (status 26)
Length of time after closure:
Individuals closed not rehabilitated before initiation of services under an IPE (status 30)
Length of time after closure:
Individuals closed not rehabilitated after initiation of services under an IPE (status 28)
Length of time after closure:
Individuals at other statuses (e.g., post-employment services)
Individuals who dropped out between application and eligibility
Individuals receiving specific services (specify services)
Other consumer satisfaction survey (specify)
Other methods:
Consumer focus groups—specify topic(s)
Survey of employers
Survey of referring or partner agencies
Survey of VR agency field staff
Survey of vendors
Other (specify)

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

26

56. If you need to explain or add any comments concerning your responses to the
question above, please use this space.

57. In the recent past, has your agency identified specific areas for improvement
through analysis of consumer satisfaction data?
Yes

No

If yes, please list the specific area(s):
If yes, please give an example of how areas for improvement were identified
(optional)

EFFECTS OF QA ACTIVITIES AND PROMISING PRACTICES
58. Does your agency have written policies and procedures, or a manual, that describe
the quality assurance activities it uses?
Yes

No

If yes, please attach a copy.
59.

In your opinion, to what extent have your agency’s quality assurance activities
improved compliance with applicable laws, regulations, and/or policies?
(Check one)
Not at all (GO TO Q 61)
Limited extent
Moderate extent
Great extent

60. Please give an example of how your agency’s quality assurance activities have
improved compliance with applicable laws, regulations, and/or policies.

61.

In your opinion, to what extent have your agency’s quality assurance activities
improved performance on measures or indicators used to monitor performance?
(Check one)
Not at all (GO TO Q 63)
Limited extent
Moderate extent

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

27

Great extent

62. Please give an example of how your agency’s quality assurance activities have
improved performance.

63. What quality assurance activity (or activities) has your agency found to be most
effective?

64. What quality assurance activity (or activities) has your agency found to be
ineffective?

65. Please describe any quality assurance practices your agency uses that you consider
exemplary or innovative.

66. If you need to explain or add any comments concerning your responses in the
“Effects of QA and Promising Practices” section, please use this space.

Please provide contact information for an agency representative in case we have
follow-up questions.
Agency name:
State:
Contact person
Name:
Email address:
Telephone:

The contact person’s name will be used only for RTI’s internal purposes, in case followup questions are necessary.

SURVEY OF VOCATIONAL REHABILITATION AGENCY PRACTICES IN COOPERATIVE ARRANGEMENTS AND QUALITY ASSURANCE

28