February 3, 1999 Meeting Summary:

Table of Contents

I. Record of Attendance

II. Welcome and Introductory Information

III. Presentations and Committee Discussion

• Clinical Laboratory Improvement Amendments of 1988 (CLIA) Update

  • Centers for Disease Control and Prevention (CDC)

  • Health Care Financing Administration (HCFA)

  • Food and Drug Administration (FDA)

• HCFA Inspections of Cytology Laboratories

• Update on Research on CLIA Cytology Requirements

  • Public Comments

  • Committee Discussion

• HCFA Validation Inspections of Accredited Laboratories

  • Public Comments

  • Committee Discussion

IV. Concluding Remarks

VI. The Addenda

Record of Attendance

Clinical Laboratory Improvement Advisory Committee

The Secretary of Health and Human Services is authorized under Section 353 of the Public Health Service Act, as amended, to establish standards to assure consistent, accurate, and reliable test results by all clinical laboratories in the United States. The Secretary is authorized under Section 222 to establish advisory committees.

The Clinical Laboratory Improvement Advisory Committee (CLIAC) was chartered in February 1992 to provide scientific and technical advice and guidance to the Secretary and the Assistant Secretary for Health regarding the need for, and the nature of, revisions to the standards under which clinical laboratories are regulated; the impact on medical and laboratory practice of proposed revisions to the standards; and the modification of the standards to accommodate technological advances.

The Committee consists of 20 members, including the Chair. Members are selected by the Secretary from authorities knowledgeable in the fields of microbiology, immunology, chemistry, hematology, pathology, and representatives of medical technology, public health, clinical practice, and consumers. In addition, CLIAC includes three ex officio members, or designees: the Director, Centers for Disease Control and Prevention; the Commissioner, Food and Drug Administration; the Administrator, Health Care Financing Administration; and such additional officers of the U.S. Government that the Secretary deems are necessary for the Committee to effectively carry out its functions. CLIAC will also include a non-voting liaison representative who is a member of the Health Industry Manufacturers Association and such other non-voting liaison representatives that the Secretary deems are necessary for the Committee to effectively carry out its functions.

Due to the diversity of its membership, CLIAC is at times divided in the guidance and advice it offers to the Secretary. Even when all CLIAC members agree on a specific recommendation, the Secretary may not follow their advice due to other overriding concerns. Thus, while some of the actions recommended by CLIAC may eventually result in changes to the law, the reader should not infer that all of the advisory committee's recommendations will be automatically accepted and acted upon by the Secretary.

WELCOME AND INTRODUCTORY INFORMATION

Prior to calling the meeting to order, CLIAC Chair Dr. Morton Schwartz acknowledged the death of Dr. Lemuel Bowie, a member of CLIAC, on December 25, 1998. Dr. Schwartz recognized Dr. Bowie for his excellent work in the field of laboratory medicine and expressed appreciation for his contributions to this Committee. Dr. Schwartz then called the meeting to order. Dr. Edward Baker, Director, Public Health Practice Program Office (PHPPO), welcomed the CLIAC members and noted that this was to be the last meeting chaired by Dr. Schwartz. He expressed gratitude to Dr. Schwartz for his years of dedicated service to the Committee, and for his wisdom, advice and leadership in dealing with many complicated issues pertaining to the Clinical Laboratory Improvement Amendments of 1988 (CLIA) regulations. Dr. Baker then announced that Dr. Toby Merlin will assume the role of CLIAC Chair at the next meeting. He also introduced Dr. Robert Martin, the new Director of the Division of Laboratory Systems (DLS), and Dr. Devery Howerton, who will be the new Branch Chief for the Laboratory Practice Standards Branch (LPSB). Following Dr. Baker, the CLIAC members made self-introductions and disclosure statements of their relevant financial interests as they relate to the topics to be discussed during the CLIAC meeting.

PRESENTATIONS AND COMMITTEE DISCUSSION

Clinical Laboratory Improvement Amendments of 1988 (CLIA) Update

Centers for Disease Control and Prevention (CDC)  - Addendum A

Dr. Thomas Hearn, Branch Chief, Laboratory Practice Assessment Branch (LPAB), DLS, PHPPO, updated the Committee on two of CDC's areas of responsibility relevant to CLIA. His first report was on the status of the possible transfer of the CLIA test categorization and waiver activities from the CDC to the Food and Drug Administration (FDA). Dr. Hearn noted that the transfer is being addressed by the CDC, FDA, and Health Care Financing Administration (HCFA), and that the process for the transfer is still being worked out by the agencies involved. He then briefed the CLIAC on several issues that relate to genetic testing, including the recent CLIAC recommendations to add specific requirements for genetic testing to the CLIA regulations. These recommendations are currently being evaluated by the Department of Health and Human Services for incorporation into the regulations where appropriate. The CDC is also involved in two research activities to gather information on current laboratory practices in molecular genetic testing. One of these is a project to identify and characterize the focus of quality assurance and proficiency testing programs for molecular genetic testing, in an effort to identify a test or group of tests that could be used to monitor quality in molecular genetic testing. The other is a survey conducted by Dr. Margaret McGovern, Mount Sinai School of Medicine, of quality assurance practices in biochemical genetic testing laboratories. Both of these projects will provide information that will assist in determining appropriate quality standards for genetic testing laboratories.

Ms. Carol Cook, Health Scientist, LPSB, DLS, PHPPO, reported on CDC's publication of the proposed model certification program for embryo laboratories, developed as part of the implementation of the Fertility Clinic Success Rate and Certification Act of 1992 (FCSRCA). The proposed model program was published as a notice with opportunity for comment in the Federal Register on November 6, 1998. After analyzing the 15 comment letters received in response to the notice, the CDC will revise the model as appropriate, and publish the final model certification program in the Federal Register. The finalized version will be distributed to State officials as described in the FCSRCA.

Health Care Financing Administration (HCFA)  - Addendum B

Ms. Judy Yost, Director, Division of Outcomes and Improvements (DOI), Center of Medicaid and State Operations (CMSO), HCFA, summarized HCFA's current CLIA implementation activities. She referred the Committee to the statistical report containing data on laboratory certification, CLIA-exempt states, accreditation organizations, survey deficiencies, and enforcement. She stated that HCFA and CDC are working on the final regulation for quality control (QC), and that development of this regulation is being given the highest priority . She added that the framework for the QC regulation is based on previous recommendations made by the CLIAC. Ms. Yost also mentioned that the CLIAC recommendations concerning genetic testing are being considered for incorporation in the CLIA regulations. Ms. Yost then reported that HCFA would meet the March 31, 1999, deadline to have their computer systems certified for AY2K@ and that HCFA and CDC have sent out letters to manufacturers and proficiency testing providers, urging them to ensure that their systems will be operational in 2000. Last, Ms. Yost discussed program integrity/fraud and abuse investigations being conducted by HCFA in coordination with other government agencies, including the Department of Justice. She emphasized, as in previous CLIAC meetings, that these investigations are not conducted as part of, or in conjunction with, CLIA inspections.

Following Ms. Yost's presentation, several CLIAC members had questions about the HCFA statistical data. Dr. Baker noted the decline in survey deficiencies over time, an indicator of improvement in laboratory quality. He posed the question of what is the ultimate goal in the rate of these deficiencies? Ms. Yost said that although the goal is to have 100% of laboratories in compliance with CLIA (and thus 0% deficiencies), it is an incremental process with many factors that affect the data. Several members commented on the fact that the statistics differ geographically, and two Committee members stated that inspection statistics may not always be a true reflection of quality assurance practices.

Food and Drug Administration (FDA)   - Addendum C

Dr. Steven Gutman, Director, Division of Clinical Laboratory Devices, Office of Device Evaluation, Center for Devices and Radiological Health, FDA, reported on new initiatives that are affecting the FDA process for review and classification of clinical laboratory devices and reagents. He mentioned the FDA's regulation on analyte specific reagents, which includes requirements for in-house (home brew) laboratory tests, and gave the website for the Health Industry Manufacturers Association (www.himanet.com) as a source of information on these regulations . He described the FDA's compliance policy guides, which explain the three ways in which products may be marketed - for research, investigational, or clinical use. Next, Dr. Gutman outlined changes in the 510(k) and premarket approval processes, intended to make them more streamlined and user friendly. Last, he discussed the Hazard Analysis Critical Control Points program, being piloted for medical devices. In this program focused on preventing problems, manufacturers of medical devices identify critical control points and their limits in the manufacturing process, and assure that they are in control. If a device failure or problem is detected, corrective action can be taken immediately to prevent further problems from occurring.

A CLIAC member asked whether there will be significant changes in the CLIA test categorization process when it is transferred to the FDA. Dr. Gutman responded that the transfer will be primarily administrative with no major changes intended. He added that efforts are being taken to maintain consistency with the process currently in place, including use of the same scoring system. He did note that there will be products to be categorized for CLIA that are currently exempt from FDA review.

HCFA Inspections of Cytology Laboratories  - Addendum D

Ms. Cheryl Wiseman, Health Insurance Specialist, DOI, CMSO, HCFA , presented information on HCFA's specialized reviews of cytology laboratories. HCFA has a contract to conduct on-site surveys of cytology laboratories for complaint investigation, by recommendation of the regional office or state survey agency, or random selection. HCFA determines which laboratories are to be inspected and directs the contractor to conduct the surveys. The survey team consists of a team leader, cytotechnologist screeners, and a board-certified pathologist with extensive training and expertise in cytopathology. The surveys are outcome-oriented and take an educational approach in assessing the laboratory's compliance with the CLIA requirements. As seen in the data Ms. Wiseman presented, the percentage of laboratories with condition level deficiencies has decreased over time, an indicator of improved laboratory performance.

Several CLIAC members asked for clarification of the statistical data, and asked what HCFA's goals are in conducting these surveys. Ms. Wiseman explained that they are trying to survey cytology laboratories in different geographic areas and laboratories that routinely screen different test volumes. Ms. Yost also pointed out that in most cases, HCFA inspectors do not have specialized expertise in cytology, and the contractor surveys are focused on cytology practices and reporting, including a review of previously reported cases. The Committee also discussed the potential value of conducting outcomes research to determine whether implementation of CLIA has resulted in an increase in the early detection of abnormal Papanicolaou (Pap) smears, and a decrease in invasive cervical cancer.

Update on Research on CLIA Cytology Requirements  - Addendum E

Ms. Rhonda Whalen, Senior Health Scientist, LPSB, DLS, PHPPO, gave a brief chronology of cytology proficiency testing (PT) with respect to CLIA. She summarized the applicable regulations and the steps in implementing the regulations, including research projects that have been conducted in an effort to develop a cytology proficiency testing program that can be administered on a national level.

Dr. Richard Keenlyside, Medical Officer, LPAB, DLS, PHPPO, discussed the results of a CDC study (conducted under contract by Analytical Sciences, Inc.) that compared cytology screeners work performance to their scores on a CDC glass slide proficiency test and a computer proficiency test developed by the CDC. In this study, the work performance of 85 cytology screeners was measured using a pair of cytoevaluators to retrospectively rescreen 500 Pap smears evaluated by each cytology screener. The cytology screeners were proficiency tested on-site using a set of 10 glass slides and in Atlanta using a computer test consisting of 10 challenges. The scoring system, which consisted of the four diagnostic categories outlined in the CLIA regulations, was used to evaluate cytology screener work performance and to score each proficiency test. Data from the study demonstrated that there is a low, but definite (unlikely to be a chance finding), correlation between an individual's work performance and the results of both glass slide and computer proficiency tests. Dr. Keenlyside noted that measurement uncertainties exist in a single 10 challenge proficiency test that need further investigation. Information from cytology PT should, therefore, be used carefully in conjunction with other performance measures as part of a laboratory's quality assurance program.

Public Comments  - Addendum F

Dr. Diane Davey, representing the American Society of Cytopathology, commented on the cytology study conducted by the CDC. She raised the following concerns, which were echoed in a second public comment by Dr. George Birdsong, a cytopathologist from Grady Memorial Hospital in Atlanta, Georgia.

There needs to be a higher level of correlation between PT and actual work performance for cytologists to feel comfortable with computerized proficiency testing. CDC must carefully consider the implications of failing cytology PT when it could mean the potential loss of an individual's job.

The study data indicates many cytologists may be considered proficient by one test and deficient by another test. Using the CLIA system for scoring the computer-based PT, approximately 50% of individuals would fail. This does not represent the true rate of problems in cytology. Either the computerized program, or the grading system, or both, should be changed.

Data showing the correlation between glass slide PT and computer-based PT were not presented, and the regulations currently specify glass slides.

Although slides were referenced prior to testing, they were not validated by consensus of participants. There can be much variation in diagnostic slides.

Pathologists were not accurately represented in the study. Evaluation of pathologists' locator skills is not a valid measure of pathologist performance, since most pathologist do not screen slides.

The lack of pathologist review of discordant cases (disagreement between cytologist and cytoevaluator) does not represent current practice where the pathologist is responsible for the final determination of any abnormal reports. It is important to correlate slide diagnosis with actual clinical follow-up.

Classification of HSIL cases as LSIL or ASCUS should not be considered a misclassification.

The number of slides (500) evaluated may be too low in some cases. The prevalence of disease in a population (or number of abnormals) should be used to determine the number of slides needed to accurately measure work performance.

Committee Discussion

A few CLIAC members commented on the most clinically significant error in reading Pap smears, which occurs when a high grade case is reported as normal, and asked whether the correlations for this had been evaluated for glass slide versus computer-based testing. They stressed that meaningful results in this study may be missed by focusing on things that occur with higher frequency but are of lower significance. Dr. Keenlyside agreed with these comments, but noted that since the numbers in this study were small, it may not be possible to evaluate an even smaller subset of data. One Committee member asked if there was a clinical utility in having the computer images trace a case over time so that the review was not just a measure of performance on one point in time. Another Committee member expressed several concerns with this study, some of which were the same as those mentioned by Dr. Davey. Additional points included :

A 10 slide PT is not a valid measure of an individual's ability to read Pap smears, especially in light of the fact that the percentage of abnormals in a proficiency test is much higher (60%) than what is seen in the workplace (approximately 95% normal). When one fourth of the slides are abnormal it does not reflect the monotony of the workday.

The skills and responsibilities of cytotechnologists and pathologists differ, and it is not appropriate that they be tested in the same way.

There is a need to examine why there were discrepancies that were two diagnostic categories apart. In the workplace, all abnormal slides are reviewed by a pathologist. The cytoevaluators who reviewed discrepancies were not pathologists. The study should have included follow up on the 43 slides called normal by cytologists but determined to be HSIL by the cytoevaluators.

It is important to use validated slides and validated computer images. In some cases the computer images may not reflect the microscopic image of certain cells. Also, fewer glass slide PT sets should have been used and the test sets should not have such a variable performance.

PT will identify individuals who have problems but continually testing individuals who consistently pass PT is not necessary.

In conclusion, several CLIAC members stated that there are a number of issues regarding cytology PT that must be considered. One member suggested that perhaps PT should be used as a screening mechanism to determine which laboratories need closer scrutiny or further review. One member stated that blind PT may never work in cytology and that on-site surveys may be the best way to measure laboratory quality. Another member stated that it might be helpful to have a presentation to CLIAC on the Maryland cytology PT experience. Others emphasized that more information is needed on cytology PT.

HCFA Validation Inspections of Accredited Laboratories  - Addendum G

Ms. Sandra Farragut, Health Insurance Specialist, DOI, CMSO, HCFA, gave an overview of HCFA's validation survey process, in which on-site inspections of a sample of accredited laboratories are performed to determine if CLIA equivalency is being sustained by the accreditation organization. She explained that the review teams used by HCFA include both technical and nontechnical personnel in an effort to be objective. When conducting the validation inspections, the team looks for serious, or Acondition level@ deficiencies, which are deficiencies that have the potential for directly affecting the accuracy and reliability of test results. If HCFA finds a condition level deficiency that was not identified by the accreditation organization, a disparity is noted, and disparity rates are determined for each accreditation organization.

Ms. Farragut presented data from the FY >97 validation inspections and said that the surveys for FY >98 should be completed soon. Regarding the FY >97 data, she stated that of 14,000 accredited laboratories, approximately 50% are accredited by the Commission on Office Laboratory Accreditation (COLA), and 25% each are accredited by the College of American Pathologists and the Joint Commission on Accreditation of Health Care Organizations. Among the American Association of Blood Banks, the American Osteopathic Association, and the American Society for Histocompatibility and Immunogenetics, approximately 400 laboratories are accredited. Ms. Farragut also mentioned that at this time, HCFA is revising the validation survey process in an effort to improve communication with the accreditation organizations and decrease the disparity rate.

Public Comments  - Addendum H

Dr. Stephen Kroger, Chief Executive Officer, COLA, commented on Ms. Farragut's presentation. He commended HCFA for their approach to these inspections, and gave a brief summary of COLA's analysis of the increase in the disparity rate for their organization from FY >96 to FY >97. He explained the steps COLA took to investigate this increase, and actions taken in response to their findings. He concluded with several recommendations to improve the current validation survey process.

Committee Discussion

The Committee asked for clarification of several points with respect to validation surveys. Ms. Yost referenced them to the CLIA regulations, where the process is specifically described. She stated that in performing these inspections, HCFA is mainly looking for trends, and acknowledged that there will always be some inherent differences between a validation survey and the inspection performed by the accreditation organization. She added that HCFA's plan to implement a process for conducting surveys simultaneously by both groups may result in differences being resolved immediately. Simultaneous surveys would also be a more streamlined process for laboratories being inspected. CLIAC members asked whether validation surveys are performed in CLIA-exempt states, and whether joint surveys are an option for states. Ms. Farragut reported that results of validation surveys show that states are doing about the same or slightly better than the accreditation organizations. She also said that simultaneous surveys may be an option in the exempt states. Several CLIAC members noted that there are states conducting surveys for HCFA that have higher deficiency rates and suggested that it may provide useful information to send surveyors from a region or state with laboratories identified as having many disparities to another area not known for having problems, and have them conduct inspections in that region or state. This could illustrate whether the disparities are true problems or a result of inconsistencies in the inspection process.

Concluding Remarks

As Dr. Schwartz prepared to adjourn his last CLIAC meeting, he noted some of the things he had learned in his experience as Chair of this Committee. Among the many, he mentioned the significance of a level playing field, the difficulties of providing laboratory services in rural areas of this country, and that compromise is important. He expressed his appreciation to the CDC , HCFA, and the FDA for providing support to the CLIAC during the time he served as Chair, and adjourned the meeting.

I certify that this summary report of the February 3, 1998, meeting of the Clinical Laboratory Improvement Advisory Committee is an accurate and correct representation of the meeting. 

                                  /S/ Morton K. Schwartz, Ph.D. 
                               Chairman