Inter-scorer Reliability Fact Sheet

Applicable Accreditation Standards

Scoring personnel must be one of the following: RST, RPSGT, CPSGT, respiratory therapists with the sleep disorders specialist certification (either CRT-SDS or RRT-SDS), or medical staff members/PhDs board-certified in sleep medicine (as defined in Standard B-2). Non-registered sleep technologists (as defined in Standard B-10) may score only under the supervision of one of the above while adhering to Standard B-10.

Inter-scorer reliability (ISR) must be determined between each scorer and the facility director or a medical staff member board-certified (as defined in Standard B-2) in sleep medicine or a corporate-appointed physician or PhD board-certified in sleep medicine; such individual must be an employee of the corporation and serves as the reference sleep specialist. ISR assessment must be conducted for each individual facility. In cases where a corporate appointed individual serves as the reference standard for ISR, the facility director or medical staff member must attest in writing that he/she has reviewed the results of the ISR assessment and will take corrective action when results fall below the facility’s level of acceptable agreement as defined in its quality assurance program.

For comprehensive PSG, the following parameters must be compared: sleep staging epoch-by-epoch agreement, respiratory events, leg movements and arousals. Sleep technologists must be blinded to the scoring of the facility director/medical staff member and to all other scoring technicians. Comparisons between each scorer and the facility director/medical staff member must be made on 200 consecutive 30-second epochs in each of three polysomnograms per quarter, for a total of 12 polysomnograms per year. Comprehensive PSG studies must report agreement between scorer and the facility director/medical staff member as percent concordance defined as the quotient of the total number of epochs of agreement for a given parameter and the total number of epochs in the analysis sample multiplied by 100. Sleep-related breathing event comparisons for laboratory PSG must include analysis by total number of events and by the following event types: obstructive apnea, central apnea and hypopnea. If the facility reports respiratory effort related arousals, this event must be included in the comparison. Use of the AASM Inter-Scorer Reliability program fulfills the requirements of this standard.

What is the intent of the ISR requirement?

The ISR program is an integral part of a facility’s quality assurance program. Close agreement of scoring personnel is necessary for accurate interpretation and quality patient management. Scoring technologists’ participation in an ISR program encourages continuous improvement of sleep scoring skills.

Which sleep facility staff members must participate in ISR?

  • All scoring technologists are required: Technical staff members who score sleep studies for the sleep facility must participate in the ISR program. This includes technical staff members that score as little as one record. It also includes staff that score for the sleep facility, but may work remotely or at a corporate location, such as subcontractors.
  • Facility gold standard: The facility director or medical staff member board-certified (individual meeting Standard B-2) serves as the facility’s gold standard and is required to participate in the ISR program. A corporate appointed board certified sleep specialist can also serve as the gold standard and must participate.

When must the sleep facility perform ISR?

ISR must be conducted on a quarterly basis as a part of the facility’s quarterly quality assurance program. Three polysomnograms are chosen randomly per quarter for use in ISR assessments for a total of 12 polysomnograms per year.

How must the sleep facility conduct ISR?

Quarterly Comparison

  • For each of the three polysomnograms compared quarterly, 200 consecutive epochs will be selected for comparison.
  • The facility gold standard and all scoring technologists must independently score the 200 consecutive epochs.
  • Scoring must include four parameters: staging (epoch by epoch), respiratory events, leg movements and arousals.

Quarterly Reporting

  • Comparison data, including a percentage agreement, must be tabulated between the gold standard and each of the scoring technologists for each of the four parameters. This data must be detailed in the facility’s quarterly report for each of the three polysomnograms compared.
  • For staging, epoch by epoch comparison must be reported.
  • The facility’s ISR policy must include a goal percentage agreement (to be set by the facility) for each of the four parameters compared. The quarterly report should indicate whether each scoring technologist has met the goal percentage and, if not, what actions were taken to address inconsistencies.
  • The quarterly report must be signed and dated by the gold standard indicating review and corrections if necessary.

AASM ISR Program: The AASM’s optional online mechanism for meeting the ISR accreditation standards.

The AASM posts a set of 200 epochs to the online ISR program each month. The AASM Gold Standard has scored the epochs already. Participating facilities must have all scoring technologists score the 200 epochs. The technologists’ scores are compared to the gold standard and a percentage agreement report is generated. The sleep facility’s gold standard must review their sleep technologists’ results and sign the report as part of the quarterly quality assurance program.

The AASM ISR program, like accreditation, is location specific. This means that although your staff completes their assessments through the AASM ISR program, the assessments cannot count for or be combined with any other accredited facility. Each accredited facility location has to purchase a separate, active ISR account to meet the standard. Once each facility has a separate, paid account for AASM ISR, then you may link staff to each location. Linking Staff to each location ensures your staff only need to complete the monthly assessment one time and their score will be applied to each location to which they are linked. Standard F-8 from the AASM Standards for Accreditation states that “ISR assessment must be conducted for each sleep facility.” Because your accreditation is location-specific, you must also have separate ISR accounts. AASM ISR is an option to fulfill Standard F-8 and is not required to fulfill the Standard. Nevertheless, each location must complete an inter-scorer reliability program each month, at each location.

Educational and reporting features of the AASM ISR Program include:

  • Monthly record review videos, that provides insight to the Gold Standard Panel decision process.
  • Free Continuing Education Credits – 12 CECs per year.
  • Monthly scoring records are provided as well as national gold-standard scores.
  • Immediate feedback provided to technologists and supervisors.
  • Center and national results are provided for each record.

Key Things to Keep In Mind

  • A complete ISR report is signed and dated by the gold standard reference sleep specialist.
  • ALL scoring technologists must participate in ISR.
  • A complete ISR report reflects a full quarter (three months) of data.
  • ISR assessment must be conducted for each sleep facility.
  • For facilities using AASM’s ISR Program with multiple related entities; each facility must be independently enrolled after the facility enrolls each site then sleep technicians may then be linked to each location.
  • Subcontracted scorers are still required to conduct ISR using assessments from the facility or through facility AASM ISR Program enrollment.
  • Corporate appointed reference sleep specialist can serve as gold standard, provided the facility director or medical staff member attests in writing that the results were reviewed and action will take place if result fall below expectations. The facility “reference sleep specialist” must meet standard B-2 – this individual cannot be a technologist.
  • Reference the Accreditation Reference Manual for specifics on what the ISR policy should include.