|
Abstract
In order to help implement
the cytology proficiency testing (PT) requirements of the Clinical Laboratory
Improvement Amendment of 1988 (CLIA), the Centers for Disease Control and
Prevention (CDC) collected and referenced approximately 1500 cervical cytology
slides in six months. Referencing of the glass slides included stratification
into the four CLIA categories and obtaining consensus by three pathologists.
Slides were first screened
by cytotechnologists blinded from the diagnosis of the contributing
laboratory. If the original diagnosis did not match the diagnosis given by the
cytotechnologists, the slides were eliminated. Three cytopathologists then
looked at the slides and either agreed or disagreed with the diagnosis. The
cytopathologists were also asked to identify slides that were unsuitable for
proficiency testing.
Cytotechnologists
eliminated 33% of the slides from further review (26% did not match the
original diagnosis and 7% were considered poor quality). The cytopathologists
reviewed the remaining 947 slides, with 756 receiving final diagnostic
consensus.
Slides in the categories
for unsatisfactory and low grade squamous intraepithelial lesions (LG-SIL)
were the hardest to achieve consensus. It was anticipated that it would be
difficult to obtain consensus of the unsatisfactory category since the
professional community has not reached agreement on the criteria for
unsatisfactory. The difficulty in achieving consensus on slides in the LG-SIL
was not anticipated. There was a tendency for our evaluators to either
overcall or undercall the low grade lesions.
Introduction
In December 1987, the Wall
Street Journal published a series of articles about problems with the
evaluation of pap smears. It reported "cases of cancer missed due to
excessive workloads of cytotechnologists, lack of quality control procedures,
and poorly educated personnel". The articles were followed by a televised
report of poor quality control and a lack of government regulations. This
media attention prompted congressional hearings that led to the passage of the
Clinical Laboratory Improvement Amendments of 1988 (CLIA). CLIA directed the
Secretary of Health and Human Services (HHS) to establish individual workload
limits for a 24 hour period; to establish quality control measures; to set
personnel standards; and to provide for annual proficiency testing of
cytotechnologists and pathologists.
The regulations for
implementing CLIA were published in the Federal Register of February
28, 1992. The regulations described the program content of a cytology PT to
include 10 and 20 glass slide tests comprised of at least one challenge from
each of the four diagnostic categories extracted from the Bethesda system. The
use of glass slides was written into the regulation to fulfill the statutory
requirement "under normal working conditions".
To date there is one state
PT program, Maryland that meets the CLIA criteria. HHS still has the
responsibility to implement PT for the large number of cytotechnologists and
pathologists who are currently not being tested. In this ongoing effort to
promote the development of PT programs, CDC complied a collection of glass
slides that have been reviewed by three pathologists and have consensus in one
of the four diagnostic categories.
METHODS
A request for proposals
(RFP) to solicit the purchase of gynecologic cytology slides for proficiency
testing, self assessment, and training was sent out on May 19, 1994. This RFP
provided a total of 1500 slides.
Demographic information
from each slide was entered in a database. The database allowed us to track
the slides through the review process and to print reports that facilitated
the review.
The first step in the
process was an initial screening of the 1500 slides by cytotechnologists. This
was accomplished through an agreement with the American Society for
Cytotechnology (ASCT). A total of eleven cytotechnologists took part in this
initial screening which took place at CDC on two separate dates. The
cytotechnologists who screened the slides were blinded from the original
diagnosis obtained from the contributing laboratory. The cytotechnologists
dotted significant cells on the slides and placed them in to one of the four
diagnostic categories identified in CLIA. The Cytotechnologists Review Form is
shown in Figure 1. The diagnosis obtained from the cytotechnologists review
was compared to the contributing laboratories diagnosis. If the contributing
laboratories diagnosis did not match with the diagnosis given by the
cytotechnologists, the slides were eliminated from further review.
The remaining slides were
then reviewed by pathologists. A total of thirteen pathologists were used in
the three review sessions that also took place at CDC. Three pathologists
looked at each slide and either agreed or disagreed with the diagnosis. The
form for this step is shown in Figure 2. The pathologists were also asked to
make comments and to identify slides that were not suitable for PT.
RESULTS
A total of 1500 slides were
included in the review, 756 (50%) slides met consensus for one of the four
diagnostic categories. The cytotechnologists eliminated 497 (33)% of the
slides. Of the 33%, 460 were eliminated when the diagnosis of the
cytotechnologists did not match that of the contributing laboratories and 37
were eliminated due to poor quality (IE., needed recoverslipping, stain was
not optimal, or was thought to be problematic as a PT slide). In addition 77
(5%) of the slides were returned to the original laboratory at the request of
the laboratory. Figure 3 contains a list of the referencing descriptions
applied to the slides.
Of the 1500 slides the
diagnosis from the contributing laboratories indicated 200 slides were from
the A category, 689 slides were from the B category, 302 slides were from the
C category and 308 slides were from the D category. The slides are identified
by categories in Figure 4 . Consensus was obtained on 34% of the slides in
category A, 59% of the slides in category B, 29% of the slides in category C
and 66% of the slides in category D.
The pathologists reviewed
947 slides with 756 (80%) receiving full consensus. Since most of the slides
were eliminated in the blind review this meant the pathologists elimnated an
additional 12% of the total slides.
Of the 947 slides reviewed
by the pathologist consensus was obtained in 60% of the slides in category A,
87% of the slides in category B, 45% of the slides in category C, and 90% of
the slides in category D. The results of the pathologists review is shown in
Figure 5. Slides that did not receive consensus were diverted to the training
branch and will be used to produce training materials that will be distributed
through the National Laboratory Training Network.
The intent of the stringent
review process was to obtain slides that are fair and equitable for testing.
Inter-observer variability makes the process difficult. In particular it was
hard to achieve consensus of the category A and C slides. There is still
disagreement on what constitutes an unsatisfactory slide and how to apply the
10% and 25% acceptability rules. The most surprising finding was the amount of
disagreement between our pathologist for LG-SIL. Cases were often undercalled
to ASCUS or overcalled the HG-SIL.
National testing and
training efforts can help standardize the diagnostic features used by the
different laboratories and eliminate the problems associated with
inter-observer variability.
REFERENCES:
- Public Law 100-578, The
Clinical Laboratory Improvement Amendments of 1988, October 31, 1988.
- HSQ-176-FC; 57 FR 7002,
February 28, 1992.
- Kurman, Robert J. and
Solomon, Diane, The Bethesda System, Springer-Verlag, New York
(1994).
|