NIST Clinical Quality Assurance Program Impact and Needs Assessment Supporting Statement

OMB Control No. #0693-0033 – NIST Generic Clearance for Program Evaluation Data Collections

Expiration Date 03/31/2016

National Institute of Standards and Technology (NIST)

Clinical Quality Assurance Program (ClinQAP)

Impact and Needs Assessment Questionnaire

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

A questionnaire was developed for the current and potential participants in the Clinical Quality Assurance Program, (ClinQAP) which is provided by the Chemical Sciences Division (CSD) of the Material Measurements Laboratory (MML) at the National Institute of Standards and Technology (NIST).

The ClinQAP is currently comprised of three subprograms that routinely provide interlaboratory assessments for measurements of micronutrients (e.g. vitamin A, vitamin C, vitamin E, carotenoids, coenzyme Q10), vitamin D metabolites, and fatty acids in human serum and plasma matrices. The longest-standing of these three subprograms, the Micronutrients Measurement Quality Assurance Program (MMQAP), was initiated in 1984 and has been operating essentially unchanged since 2000; 78 and 43 comparability studies have been conducted for fat-soluble vitamins and vitamin C, respectively, to date. The Vitamin D Metabolites Quality Assurance Program (VitDQAP) was formed in 2009 and has provided eleven comparability studies, and the Fatty Acids Quality Assurance Program (FAQAP) has conducted two studies to date. The MMQAP, VitDQAP, and FAQAP are all in various “life-stages” ranging from mature (MMQAP) to infancy (FAQAP), but all have the common goal of improving the comparability of clinical measurements for selected analytes.

The CSD is currently performing needs- and impact-assessments for many of its programs and services and is currently formulating a strategic plan. Internal assessments have been made by the Technical Project Leaders (TPLs) of the three respective subprograms, and all agree that the current ClinQAP has some potential limitations that should be addressed. Primarily, there is concern that the ClinQAPs currently address only a narrowly-focused group of analytes and matrices and hence a limited portion of the clinical research community. Furthermore, since the mature programs have already accomplished many of their intended research goals (i.e., improvement in the comparability of laboratory measurements), the continuation of the subprograms in their present format, particularly the MMQAP and the VitDQAP, might not be technically justifiable.

Rather than speculate on the community’s needs and whether these needs align with the CSD’s research interests and goals, the TPLs of the subprograms have developed a questionnaire for the current and potential participants of the ClinQAP. The questionnaire has two intentions: 1) to assess if the current ClinQAPs are necessary for meeting the performance validation requirements of the participating laboratories and 2) to monitor and support the emerging measurement needs of the clinical community with the goal of expanding the potential research areas and hence impact of the ClinQAP.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The questionnaire was developed by the ClinQAP program coordinator in collaboration with the three subprogram TPLs. When combined, the program coordinator and TPLs have over 50 years of experience coordinating quality assurance studies for the nutritional and clinical communities, and have maintained positive, collegial relationships with the participating laboratories. Therefore, the program coordinator and TPLs are well-qualified to construct a questionnaire to meet the intentions described in Question #1. The questionnaire passed several rounds of testing through the TPLs and was also tested with other CSD staff members with interest in clinical chemistry and the field of metabolomics, which are potential growth areas for the ClinQAP. Suggestions for improvement were incorporated into the questionnaire during each iteration of internal review and testing.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

When the questionnaire was first released in November, 2014, the current and prospective MMQAP, VitDQAP, and FAQAP participants received a preliminary email from their respective TPLs through their regular communication channels. For example, the VitDQAP has a dedicated NIST email address, vitdqap@nist,gov, which is exclusively used to communicate with participants of the VitDQAP. The email contained a preliminary announcement of the ClinQAP and the questionnaire had the following content:

“Dear Colleagues,

As you might be aware, NIST currently offers three quality assurance programs (QAPs) for the determination of micronutrients, vitamin D metabolites, and fatty acids in serum and plasma matrices, the MMQAP, VitDQAP, and the FAQAP, respectively. Given the similarity in the operations of these programs, NIST is consolidating them into one larger program, the NIST Clinical Quality Assurance Program, or ClinQAP. The primary goals of the ClinQAP are to support the comparability of clinical measurements through the subprograms, MMQAP, VitDQAP, and the FAQAP, and to monitor and support the emerging measurement needs of the clinical community.  You will begin to notice the transition to this program over the next year, where the 2015 comparability study communications will begin using this terminology.  Beyond 2015, the ClinQAP will begin using a uniform reporting system that is different than what the individual programs are currently using.  We will provide additional details and instructions when we are ready to implement these changes.

 

In the meantime, we have designed a questionnaire requesting your feedback.  The major goals of the questionnaire are to assess the impact of the MMQAP, VitDQAP, and FAQAP and to identify measurement needs of the community that are not currently being addressed through the existing subprograms.  The questionnaire can be accessed from our website, http://www.nist.gov/mml/csd/clinqap.cfm, or by using the direct link below:

 

https://mmlapps.nist.gov/questionnaire/

 

We greatly appreciate your feedback and request your participation in this questionnaire by December 31, 2014.

If you have any questions regarding the questionnaire, don’t hesitate to email us at clinqap@nist.gov.

 

Sincerely,

<the TPLS of the MMQAP, VitDQAP, or FAQAP>”

The email requesting feedback via the questionnaire was sent initially to the approximately 150 current and prospective participants of the ClinQAP. After that time, no additional email messages were sent, but the link to the questionnaire was kept active on the program website. Therefore, anyone who encounters the online collection instrument is able to provide feedback, irrespective of their status in the ClinQAP. To date (2/11/16), 53 responses to the survey have been received. Of those, all but five respondents indicated that they are currently participating in at least one of the subprograms. The response rate is therefore estimated to be 30%. While this is lower than the 75% response rate that we originally projected, this is satisfactory to the TPL’s of the subprograms, who wish to keep the questionnaire active.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

The questionnaire is being conducted through a web interface, and the responses are collected into a database that is stored on a local server that is managed by MML. A CSD staff computer scientist who generated the web interface for the questionnaire is currently the only person who has access to the database containing the responses. However, per the request of the subprogram TPLs, the staff computer scientist downloads the results from the server into a usable, searchable file in Microsoft Excel. The collection instrument contains many open-ended questions, and a wide-range of responses has been received so far that are not easily generalized to the participant population. Furthermore, the identities of the participants in the ClinQAP are kept anonymous, and reporting the results of the questionnaire could potentially reveal their identities. Therefore, the results from the questionnaire are not being formally reported, but rather the TPLs review the individual responses periodically to assess:

1) the approximate percentage of respondents that feel the current subprograms are necessary/not necessary for meeting their laboratory performance validation requirements. If, for example, the majority of the participants in the current MMQAP indicate that services offered by the College of American Pathologists (CAP) are sufficient to meet their accreditation needs for vitamins A and E, we may stop offering comparability studies for these particular analytes. The MMQAP participants would then be informed that the decision was made to phase-out the program for vitamins A and E based on the feedback received in the questionnaire.

2) if there are any predominant trends in the participant responses to the open-ended questions. If, for example, several of the participants express difficulty measuring toxic metals (e.g., arsenic) in dried blood spots, we may design and coordinate an interlaboratory comparison study to address this measurement challenge.

3) the subset of participants who are interested in metabolomics research and/or measurements of human urine, who may then be invited specifically to participate in a pilot interlaboratory study that has been recently proposed in the CSD.

3