Inter-scorer Reliability (ISR) Fact Sheet 

Applicable Accreditation Standards

For accreditation networks providing diagnostic services (In-Lab Testing and/or HSAT), scoring personnel must be one of the following: RST, RPSGT, CPSGT, respiratory therapists with the sleep disorders specialist certification (either CRT-SDS or RRT-SDS), or medical staff members/PhDs board-certified in sleep medicine (as defined in Standard N-1 and Standard N-3). Scoring personnel not credentialed in sleep as identified above (e.g., non-registered or non-certified sleep technologist or respiratory therapists without sleep disorders specialist certification) may score only under the supervision of one of the above while adhering to Standard N-11. 

Inter-scorer reliability (ISR) must be assessed individually between each scorer and either the Network Director or a board-certified medical staff member specializing in sleep medicine, as per the definition in Standard N-1. If a board-certified medical staff member in sleep medicine is designated as the reference standard for ISR, the Network Director is required to provide written attestation of reviewing the ISR assessment results. Use of the AASM Inter-Scorer Reliability program fulfills the requirements of this standard.

The ISR program must include the following:

  • The compared parameters must include sleep staging epoch-by-epoch agreement, respiratory events, leg movements and arousals.
  • Sleep-related breathing event comparisons for laboratory PSG must include analysis by total number of events and by the following event types: obstructive apnea, central apnea, and hypopnea.
  • If the lab reports respiratory effort related arousals, this event must be included in the comparison.
  • Sleep technologists must be blinded to the scoring of the Network Director/medical staff member and to all other scoring technicians.
  • Comparisons between each scorer and the Network Director/medical staff member must be made on 200 consecutive 30-second epochs in each of three polysomnograms per quarter, for a total of 12 polysomnograms per year.
  • Comprehensive PSG studies must report agreement between scorer and the Network Director/ medical staff member as percent concordance defined as the quotient of the total number of epochs of agreement for a given parameter and the total number of epochs in the analysis sample multiplied by 100. The sleep lab must establish minimum thresholds for the percent concordance.
  • On a quarterly basis, the Network Director must confirm the effectiveness of these measures and outline plans for addressing remediation of any quality measures falling below the minimum threshold.
  • Quarterly reports must be signed and dated by the Network Director and kept on record for a minimum of five years. 

What is the intent of the ISR requirement?

The ISR program is an essential component of a quality assurance program. Close agreement among scoring personnel is crucial for accurate interpretation and effective patient management. Participation in an ISR program promotes continuous improvement of sleep scoring skills for scoring technologists.

Which must participate in ISR?

  • All scoring technologists: Technical staff members who score sleep studies for the sleep facility. This includes:
    • Those who score even a single record. 
    • Subcontracted scorers. 
  • Reference Sleep Specialist: The Network Director or a staff member who is board-certified (meeting Standard N-1) serves as the reference sleep specialist.

Conducting ISR

ISR is a Network Standard. This allows networks to have a single inter-scorer reliability program. Technical scoring staff to perform a single (monthly) scoring comparison, even if they work at multiple locations within the Accreditation Network. 

Quarterly

  • Three polysomnograms (PSGs) must be compared each quarter, using 200 consecutive epochs per study, for a total of 12 polysomnograms per year.
  • The reference sleep specialist and all scoring technologists must independently score the epochs for each PSG used in the comparison.
  • Scoring must include four parameters: staging (epoch by epoch), respiratory events, leg movements and arousals.
  • An acceptable level of agreement, typically expressed as a percentage, must be achieved between each scorer and the reference sleep specialist.

Quarterly Reporting

  • Comparison data must be tabulated for each of the four parameters, showing the acceptable level of agreement.
  • The quarterly report should indicate whether each scoring technologist has met the acceptable agreement level. If not, it should outline actions taken to address inconsistencies.
  • The Network Director must sign and date the quarterly report, confirming review and noting any corrective actions taken, if necessary. 

Use of AASM’s Sleep ISR Program

  • The AASM offers an optional online tool to meet ISR accreditation standards.
  • Each month, the AASM posts a set of 200 epochs to the Sleep ISR program, pre-scored by the AASM Gold Standard.
  • All scoring technologists from participating entities must score the 200 epochs.
  • The technologists’ scores are compared to the gold standard, and a percentage agreement report is generated.
  • The Network Director must review the sleep technologists’ results and sign the report as part of the quarterly quality assurance program.

Key Things to Keep In Mind

  • Eligible Scoring Personnel: Must be one of the following: RST, RPSGT, CPSGT, CRT-SDS, RRT-SDS, or medical staff members/PhDs board-certified in sleep medicine.
  • Scoring staff are identified as those that score for interpretation/reporting purposes (e.g., day staff) but it includes those that must review data during the acquisition to make a determination regarding adjusting patients therapeutic devices (e.g., PAP).
  • AASM Sleep ISR Program: Fulfills the requirement for conducting ISR.
    • Sleep ISR offers both adult and pediatric tracks. The service location may choose to score adult studies, pediatric studies, or a combination.
    • Only 3 studies per quarter are required, regardless of the track.
    • 1 CEC can be earned for each monthly assessment, but it must be claimed and is not automatically awarded.
  • Reporting Requirements:
    • ISR must be included in the QA Quarterly Report, regardless of whether the AASM ISR program is used.
    • ISR assessments must be conducted for each scorer in the network, regardless of the number of studies scored.
    • A complete ISR report must be signed and dated by the Network Director.
  • Subcontracted scorers are required to participate in ISR.
  • Reference Sleep Specialist must meet Standard N-1 and cannot be a technologist.