Supporting Statement
Biennial Review of Defense Agencies and DoD Field Activities
A. Justification
1. Circumstances of Information Collection
The following document details an extension of a currently approved collection (OMB Control Number 0704-0422) for the Biennial Review of Defense Agencies and DoD Field Activities. This review includes respondents who are not federal employees. During the previous three cycles of the Biennial Review (2001-2002, 2003-2004, 2005-2006) several Defense Agencies and DoD Field Activities identified organizational customers who are not federally employed, but whose feedback would also be of value for the Biennial Review. This statement describes the basics of the overall Biennial Review methodology, as it would apply to these respondents and requests a new burden approval for the 2008-2009 Biennial Review Organizational Customer Survey.
The Biennial Review is a performance assessment effort mandated by Section 192(c) of Title 10, United States Code, which states that “periodically (and not less than every two years), the Secretary of Defense shall review the services and supplies provided by each Defense Agency and Department of Defense (DoD) Field Activity. The statute establishes two purposes for the Biennial Review: (1) to ensure “there is a continuing need for each such Agency and Field Activity;” and (2) to ensure that “the provision of those services and supplies by each such Agency and Field Activity, rather than by the Military Departments, is a more effective, economical, or efficient manner of providing those services and supplies or of meeting the requirements for combat readiness of the armed forces.” In a memorandum dated June 9, 2001, the Deputy Secretary of Defense stated that an additional purpose of the Biennial Review is to provide useful organizational-customer feedback to the Directors of the Defense Agencies and DoD Field Activities regarding the level of satisfaction held by their organizational customers. In addition to meeting these performance assessment requirements, the Biennial Review also provides agencies and field activities with the opportunity to obtain feedback for purposes of quality improvement and/or planning.
New Biennial Review Construct. The last four Biennial Reviews have used a standardized survey to assess the levels of satisfaction held by the organizational customers on the products and services provided by each Defense Agency and DoD Field Activity. This survey served as the core data-gathering methodology for the Biennial Review. Effective immediately, a change has been made to the Biennial Review. While a survey of the organizational customers of the Defense Agencies and DoD Field Activities shall continue to be used in the Biennial Review process, the survey component will be completed every other Biennial Review. In this manner, organizational customers will be surveyed every four years, and this shall be known as the Biennial Review with Organizational Customer Survey (BR OCS). Alternating with the BR OCS, the Biennial Review shall be conducted without the survey component, and this shall be known as the Biennial Review with Senior Management Assessment (BR SMA).
Since the 2003-2004 Biennial Review included a survey component, the 2005-2006 Biennial Review, covering the period from January 2005 to December 2006, did not contain a survey. The 2005-2006 BR SMA report included a summary and analysis of Secretary of Defense decisions that demonstrated fulfillment of the requirements of Section 192(c) since the previous Biennial Review. The 2005-2006 BR SMA report was published in March 2007.
The current Biennial Review covers the period from January 2007 to December 2008, and it will be the 2008-2009 BR OCS. The report will include a summary and analysis of Secretary of Defense decisions that demonstrate fulfillment of the requirements of Section 192(c) since the 2005-2006 BR SMA report. A survey of organizational customers will be conducted, and will be followed by a full analysis of the survey results by business line. Where possible, trend analysis will be included on each business line from previous Biennial Reviews. The survey process used for the 2003-2004 Biennial Review shall serve as the model for this report.
In conducting the Biennial Review, the Director of Administration and Management, Office of the Secretary of Defense (OSD), requires each Defense Agency and DoD Field Activity to identify their major mission activities as “business lines,” to identify the specific “products and services” within that business line, and to identify organizational customers who can provide feedback required to meet the purposes of the review. The Biennial Review uses standard questionnaire content to answer the main legislative requirements, coupled with standard questions to help guide improvement efforts and questions unique to the business line of each agency or activity. Each business line has a unique survey (hence, a family of surveys); for the 2003 review there were 58 viable business lines. With respect to this OMB request, 8 business lines had more than 9 non-federal respondents identified on their customer list. The other 50 business lines have no non-federal respondents identified, or have fewer than 10. A sample survey is included in this submission. Some agencies or field activities elect to add custom questions of their own to gather feedback from the respondent population. These are usually additional quality improvement questions or are done to capture additional information about the respondents needed for planning purposes. The survey sample provided includes a sample of custom questions provided by the Defense Contract Audit Agency.
USES OF THE DATA
Note that the OSD Biennial Review survey is not intended to generate statistics about these populations. We are not interested in being able to generalize to the population of customers. Rather we are using the survey as a tool for obtaining customer feedback for performance improvement activities. The goal is to collect information from interested customers in order to help Agencies/DoD Field Activities improve the products and services that they supply to their customers. We are using the survey, therefore, as a management tool (rather than as a “research” tool).
CURRENT STATUS
OSD is currently working with the various Agencies and DoD Field Activities to obtain their customer lists and also receive information with which to generate their surveys. This information includes their updated lists of products and services, as well as whether the Agency/Field Activity would like to add any “custom” questions to the core Biennial Review survey. As of January 22, 2009, OSD has received information from the following Agencies and DoD Field Activities:
Defense Advanced Research Projects Agency (DARPA)
Defense Business Transformation Agency (BTA)
Defense Commissary Agency (DeCA)
Defense Contract Audit Agency (DCAA)
Defense Contract Management Agency (DCMA)
Defense Information Systems Agency (DISA)
Defense Legal Services Agency (DLSA)
Defense Logistics Agency (DLA)
Defense Media Activity (and AFIS legacy products and services) (DMA)
Defense Security Cooperation Agency (DSCA)
Defense Security Service (DSS)
Defense Technical Information Center (DTIC)
DoD Education Activity (DoDEA)
DoD Test Resource Management Center (TRMC)
Missile Defense Agency (MDA)
National Geospatial-Intelligence Agency (NGA)
Office of Economic Adjustment (OEA)
Pentagon Force Protection Agency (PFPA)
TRICARE Management Activity (TMA)
OSD is continuing to work with the following Agencies and DoD Field Activities to receive their final customer lists and survey information:
Defense Finance and Accounting Service (DFAS)
Defense Intelligence Agency (DIA)
Defense POW/MP Office (DPMO)
Defense Technology Security Administration (DTSA)
Defense Threat Reduction Agency (DTRA)
DoD Human Resources Activity (DHRA)
National Reconnaissance Office (NRO)
National Security Agency/Central Security Service (NSA)
Washington Headquarters Services (WHS)
Some Agencies/DoD Field Activities have more than one “business line”; OSD will be sending out one survey for each business line. The type of customers can vary greatly across business lines, and so some business lines may have many non-Fed customers, while some may not have any. Included in this submission we have provided an Excel table that indicates for each business line whether it includes any non-Fed customers, and also whether it includes any “custom questions” submitted by the Agency/DoD Field Activity. Note that we have also included a pdf sample of each survey that would be sent to a non-Fed customer. The information we have provided to OMB is based on the 19 Agencies/DoD Field Activities that have submitted their information to OSD. For the remaining 9, OSD will send updates to OMB as they come available to us.
2. Purpose and Use of Information
The information collected is used in the broader context of the Biennial Review process. The overall Biennial Review data serve the specific purpose of meeting the legislative requirement described in the preceding section, i.e., the data are used by OSD and DoD management to determine the continuing need for and effectiveness of the agencies and field activities. In this respect, the Biennial Review is only one of many data sources used in the overall management of Defense Agencies and DoD Field Activities. The 1997 and 1999 Biennial Reviews were conducted with only federal government employees providing feedback. The 2001, 2003, and 2005 Biennial Reviews included non-federal government respondents.
OSD has taken advantage of the Biennial Reviews requirement to create a system of standardized feedback that is used by agencies and field activities to guide their planning and quality improvement efforts. There is no specific guidance in the authorizing legislation which indicates that non-federally employed individuals should be included in the surveys conducted to answer the review questions. However, in those business lines where these individuals are either the main type of customer (e.g., Industrial Security Program for the Defense Security Service) or are key participants in a business process (e.g., as contractors or stakeholders), it is very valuable to agencies and field activities to add in feedback from organizational customers who are not federal employees. The data from these individuals will be combined with data from military and civilian federal employees to provide a balanced view of the performance of each business line.
A sample report from the Defense Contract Audit Agency is included in this submission.
3. Use of Information Technology
The survey process is optimized to allow respondents to make as much use of information technology as they desire. All respondents have the opportunity to complete the survey using Internet access after receiving the preliminary notification letter (71 % of respondents utilized Internet in the 2003 review to complete the survey). All survey operations are coordinated using a central database, which enables the survey operations staff to mail surveys and reminders to only those who have not responded.
The publication of the findings for the overall Biennial Review is shared via CD-ROM. A limited number of paper copies are made available to the distribution list on request.
4. Efforts to Identify Duplication
There is no other survey process in DoD that duplicates the methods and purposes of the Biennial Review as a standardized assessment of Defense Agency and DoD field Activity performance. This is equally true for the federally-employed respondents as it is for the non-federal respondents covered by this request. A key characteristic of the Biennial Review is a focus on organizational customers and stakeholders who are in a position to provide feedback on agency and field activity performance at a level of an overall business line. It is clearly differentiated from surveys of retail customer satisfaction. The same standards have been applied to the selection of respondents who are not federal government employees.
Individual agencies and field activities are strongly advised not to duplicate the format or questions of the Biennial Review in their individual organizational surveys. It is possible that some agencies or field activities will have survey efforts more specific to their own interests that could be sent to some of the same respondents. However, the trend for federal government respondents is that agencies and field activities are electing to use the Biennial Review for their own organization-specific surveys of customers at this level.
In addition, Defense Agencies and DoD Field Activities are highly encouraged to add custom questions to the Biennial Review survey. This will help a particular agency or field activity focus on a specific area of concern. For the 2003 Biennial Review, 11 Defense Agencies and DoD Field Activities added custom questions to the Biennial Review survey.
5. Involvement of Small Entities
There are no characteristics of the survey design or purpose that would indicate any noticeable number of small entities in the non-federal respondents covered by this request. The survey focuses on organizational customers.
The business line most likely to include small entities is the Defense Security Services Industrial Security Program. Any participating small entities on this list have already demonstrated an ability to handle much more elaborate information request burdens by virtue of their participation in gaining approval to handle classified material. If the products or services associated with this business line are burdensome, this survey process provides a very time-efficient method of providing feedback on that burden.
6. Consequences if Information Collected Less Frequently
Law mandates the Biennial Review process at a two-year review cycle. The next Biennial Review will cover the 2007-2008 timeframe. The use of survey findings by agencies and activities to make improvements to their programs require additional time to make changes to the business processes that are identified as needing attention. Exclusion from the cycle would extend the period during which legitimate concerns of non-federal respondents could be addressed through this feedback process beyond two years.
7. Consistency with the Guidelines in 5 CFR 1329.5(d)(2)
This information collection fully complies with 5 CFR 1329.5(d)(2).
8. Consultation Outside the Agency
The 60-day notice for the proposed data collection is currently published in the Federal Register: August 29, 2008 (73 FR 50947-50948). A copy of the notice is provided. No comments were received.
The data collection instruments are written using plan, coherent, and unambiguous terminology which has been reviewed and modified using a walk-through cognitive lab methodology for the general version of the Biennial Review instructions and surveys. Comments from previous Biennial Reviews were incorporated into the survey instructions and design. Also, each business line survey is reviewed by the agency or field activity that conducts it.
9. Payments to Respondents
OSD is not requesting approval for payment to respondents for provision of this information.
10. Assurance of Confidentiality
All results will protected to the extent permitted by the law. Although identifying information is associated with individual responses during data collection, it is only used to ensure control of the survey process and proper association of an individual’s responses with the correct business line. That information remains with the survey operations contractor. No individual survey item responses are ever reported to the participating agencies or field activities. Information reported out will be aggregated to the entire group of private sector respondents for any affected business line. No individual responses or aggregate responses from which individuals could be identified will be reported. Comments are summarized in published reports. Comments are edited to remove identifying information prior to release to agencies and field activities, unless the respondent specifically grants permission to share as written.
11. Questions of a Sensitive Nature
There are no questions in the proposed surveys that ask for sensitive information. The only potentially sensitive information involved in this data collection is the honest evaluation of performance by Defense Agencies and DoD Field Activities. Since respondents’ identities are protected to the extent permitted by law and no individual information will be shared with the agencies and field activities, it is not anticipated that there will be any problems in dealing with sensitive information.
12. Estimates of Annualized Hour Burden
Below is the computation of burden based on the number of non-federal respondents nominated for each business line. In 2003-2004 the Defense Agencies and DoD Field Activities nominated over 11,300 customers across 75 viable business lines. Participation for both groups of respondents is voluntary. The number of non-federal government customers nominated for any specific business line is dependent on the particular mission area, and ranges from approximately 400 contractor organizations sampled for the DSS - Industrial Security Program to business lines with two or three nominated. In many cases, there are too few non-federal government customers nominated to anticipate that the results for these respondents will be reported separately. This issue is discussed in more detail in section B-1.
No detailed estimates of wage costs for specific business lines were made, since the salaries of persons surveyed ranges across a mix of job levels and organizations and are not pre-determined in the data collection plan. If the average wage cost/hour of a respondent capable of providing the desired feedback was $75, the total burden for the respondents would be $46,875 based on a 100% response rate.
Number of respondents: 2500.
Responses per respondent: 1.
Total annual responses: 2500.
Hours per response: 15 minutes.
Total hours: 625.
13. Estimates of Annualized Cost Burden to Respondents
There are no capital or startup costs for respondents completing the survey. Paper surveys will be returned in postage paid envelopes. There are no ongoing record keeping requirements.
14. Estimates of Annual Cost to Government
There are no separate costs to the government associated with this request. The cost of adding the non-federally employed respondents to the study is minimal. While there will be additional postage costs for mailing notification letters, surveys, and reminders, the survey support contractor bid the project as a fixed price contract, so there is no additional government cost for adding these respondents beyond the costs of the main survey effort. All reporting and related uses of the information will be included within the planned costs of producing the overall report. The most recent contract cost the government $350,000 per year, with four options to renew for a total cost of $1.75 million.
15. Changes in Burden
The burden estimate represents a modest decrease, reflecting a decrease in the number of non-federal respondents.
Time Schedule, Analysis Plans, and Publication
Time Schedule
The 2008-2009 Biennial Review will be conducted for a three-month period. Data from non-federal respondents are collected simultaneously.
Analysis Plans
The data analysis primarily involves computing descriptive statistics that reflect proportions of the respondents selecting specific response alternatives for each question. The report production process is highly automated and will begin immediately upon the completion of the field period, including initial cleaning of data. It is anticipated that the results of the non-federally employed respondents will be reported as a distinct category of respondents only where there are a noticeable proportion of the overall customer population. The analysis period is projected to take two months across the anticipated 75 business lines. Beginning in the 2003-2004 Biennial Review and continuing for the review covered in this request, overall and component satisfaction percentages are analyzed for significant changes of trends using a confidence interval method (95%, two-tailed coverage). Trends are only tested for significant changes with the immediately preceding Biennial Review data when data from both cycles meet minimum data reporting standards used for the entire report.
Publication
The publication of the results is a two-step process. Once the analysis is complete, reports for each business line are generated and provided to each Defense Agency and DoD Field Activity for review. They are allowed a two-week period to review their findings and provide comments for inclusion in the final report. Once the agency and field activity comments are provided, the findings will be published and circulated to the distribution list on CD-ROM, with a limited number of paper copies available. OSD’s current plan is that Agencies/DoD Field Activities with fewer than 15 completed surveys for a given business line will not receive a separate report (this is being done to protect respondents’ identities to the extent permitted by law).
17. Display of Expiration Date
All materials transmitting the data set to respondents and any provider instruments created to capture information display the expiration date of OMB approval.
18. Exceptions to Certification Statement
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
B. Statistical Methods
Note—the statistical and survey methods used for the respondents who are not federally employed are identical to those used for federal employees. These methods have been used successfully for the 1999, 2001, and 2003 Biennial Review efforts.
1. Respondent Universe and Sampling Methods
Most business lines involve a complete census of organizational customers, since the total numbers nominated are generally small enough that a census is possible within the same level of resources that would normally be used to generate target samples of 400. This is true for both federally-employed and non-federal respondents. For non-federal respondents, all customer organizations of relevance to each business line were included. For any business line where the customer census is larger than 400 (e.g., the Defense Security Service–Industrial Security Program serving contractors who have facility security clearances), customers will be sampled randomly.
As indicated in A1 of the supporting statement, the purpose of the survey is not to generate generalizable statistics about a population of inference. This survey is merely a tool to collect input from customers to assist in process improvement efforts. The goal is to collect voluntary information from customers who are interested in providing feedback – their feedback would then be used by Agencies and DoD Field Activities for improving products and services, and also be used by OSD to oversee the performance improvement process across DoD.
2. Information Collection Procedures
The primary strategy for each business line is a standard survey containing core questions used for all business lines, ratings of specific products and services unique to the business line, and custom questions unique to the business line used by the participating Defense Agency or DoD Field Activity for its own purposes. A standard five stage Dillman method survey approach is used that includes web access to survey completion at any point in the process. The nomination letter (pre-notification) includes information for access to the web site. Respondents not responding receive a paper survey, a reminder, a second paper survey, and a second reminder per the stages of the Dillman method.
3. Methods to Maximize Response Rates
Two types of efforts have been made to maximize response rates. The notification materials include a memorandum from senior Defense leadership (the Deputy Secretary of Defense signed a memo for the 2001-2002 review, and the Deputy Director, Administration and Management signed a memo for the 2003-2004 review). This will increase response rates by emphasizing the legitimacy and importance of the project. Copies of these letters are included in this submission.
The other strategy is to use a five-stage Dillman survey methodology for managing survey operations. This includes the notification letter, followed by a survey, followed by a reminder, followed by a survey, followed by a reminder. A unique feature of the study involves giving respondents the survey website address, their user ID, and password in the notification letter. Respondents who choose to can log into the website and complete a survey without having to receive a paper copy. Respondents who complete a survey online are removed from additional mailings. The initial notification letter, survey cover letter, and reminder letter are included in this submission.
In 2003-2004, the overall response rate was 41 %. However, many individual business lines had much higher response rates, ranging up to 70%. Ten of 56 viable business lines had response rates of 60% or higher. The overall response rate was equivalent to previous Biennial Reviews. The response rate for non-federal respondents will likely be the same as the overall average, although two business lines with a large number of non-federal customers (Office of Economic Adjustment and the DSS Industrial Security Program had response rates of 67% and 60% respectively).
4. Tests of Procedures
No new tests of procedures are proposed. The planned approach was used for the 1999, 2001, and 2003 Biennial Review supported by Westat’s Organization and Management Research Group (OMRG). All procedures proposed reflect currently accepted best practice for this type of data collection effort.
5. Statistical Consultations
Most business lines included surveyed the entire population of organizational customers identified by the agencies and activities. Given the relatively small size of most of those populations, the only statistical techniques employed represent summary data describing the proportions of respondents indicating general levels of satisfaction or agreement on the standard questions. Minimum respondent numbers are imposed on reporting of results to prevent misleading or erroneous conclusions. No complex statistical procedures have been employed. For previous Biennial Reviews, W. Wade Martin, Ph.D.; Kimya Lee, Ph.D.; and Mr. John Helmick, MS from the Westat OMRG have provided advice for the project, in consultation with the Westat Statistics Group. No changes to these procedures are currently planned.
| File Type | application/msword |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |