Inter-scorer Reliability Fact Sheet

Applicable Accreditation Standards

Scoring personnel must be one of the following: RST, RPSGT, CPSGT, respiratory therapists with the sleep disorders specialist certification (either CRT-SDS or RRT-SDS), or medical staff members/PhDs board-certified in sleep medicine (as defined in Standard B-2). Non-registered sleep technologists (as defined in Standard B-10) may score only under the supervision of one of the above while adhering to Standard B-10.

Inter-scorer reliability (ISR) must be determined between each scorer and the facility director or a medical staff member board-certified (as defined in Standard B-2) in sleep medicine or a corporate-appointed physician or PhD board-certified in sleep medicine; such individual must be an employee of the corporation and serves as the reference sleep specialist. ISR assessment must be conducted for each individual facility. In cases where a corporate appointed individual serves as the reference standard for ISR, the facility director or medical staff member must attest in writing that he/she has reviewed the results of the ISR assessment and will take corrective action when results fall below the facility’s level of acceptable agreement as defined in its quality assurance program.

For comprehensive PSG, the following parameters must be compared: sleep staging epoch-by-epoch agreement, respiratory events, leg movements and arousals. Sleep technologists must be blinded to the scoring of the facility director/medical staff member and to all other scoring technicians. Comparisons between each scorer and the facility director/medical staff member must be made on 200 consecutive 30-second epochs in each of three polysomnograms per quarter, for a total of 12 polysomnograms per year. Comprehensive PSG studies must report agreement between scorer and the facility director/medical staff member as percent concordance defined as the quotient of the total number of epochs of agreement for a given parameter and the total number of epochs in the analysis sample multiplied by 100. Sleep-related breathing event comparisons for laboratory PSG must include analysis by total number of events and by the following event types: obstructive apnea, central apnea and hypopnea. If the facility reports respiratory effort related arousals, this event must be included in the comparison. Use of the AASM Inter-Scorer Reliability program fulfills the requirements of this standard.

What is the intent of the ISR requirement?

The ISR program is an integral part of a facility’s quality assurance program. Close agreement of scoring personnel is necessary for accurate interpretation and quality patient management. Scoring technologists’ participation in an ISR program encourages continuous improvement of sleep scoring skills.

Which sleep facility staff members participate in ISR?

  • All scoring technologists: Technical staff members who score sleep studies for the sleep facility. This includes:
    • Technical staff members that score as little as one record and;
    • Subcontracted scorers
  • Reference Sleep Specialist: The facility director or medical staff member board-certified (individual meeting Standard B-2) serves as the facility’s reference sleep specialist.
  • A corporate appointed board-certified sleep specialist that serves as the reference sleep specialist.

Details on conducting sleep facility ISR?

Quarterly

  • Three polysomnograms (PSGs) must be compared using 200 consecutive epochs for a total of 12 polysomnograms per year.
  • The facility’s reference sleep specialist and all scoring technologists must independently score the epochs for each polysomnogram used in the comparison.
  • Scoring must include all four parameters: staging (epoch by epoch), respiratory events, leg movements and arousals.
  • PSGs must report an acceptable level of agreement between each scorer and the reference sleep specialist (usually expressed as a percent).

Quarterly Reporting

  • Comparison data must be tabulated for each of the four parameters compared indicating the acceptable level of agreement.
  • The quarterly report should indicate whether each scoring technologist has met the goal percentage and, if not, what actions were taken to address inconsistencies.
  • The quarterly report must be signed and dated by the reference sleep specialist indicating review and identify actions taken for corrections if necessary.
  • For facilities using a corporate appointed reference sleep specialist, the facility’s director must attest in writing that the information has been reviewed.

AASM Sleep ISR Program: The AASM’s optional online mechanism for meeting the ISR accreditation standards.

The AASM posts a set of 200 epochs to the online Sleep ISR program each month that have already been scored by the AASM Gold Standard.

Participating facilities must have all scoring technologists score the 200 epochs.

The technologists’ scores are compared to the gold standard with a percentage level of agreement report generated.

The sleep facility’s reference sleep specialist must review their sleep technologists’ results and sign the report as part of the quarterly quality assurance program.

The Sleep ISR program, like accreditation, is location specific.

Standard F-7 states that “ISR assessment must be conducted for each sleep facility”, therefore each accredited facility location has to purchase a separate, active ISR account to be in compliance.

Once each facility has established a separate, paid account you may link staff to each location. Linking Staff to each location ensures your staff only need to complete the monthly assessment one time and their score will be applied to each location to which they are linked.

Key Things to Keep In Mind

  • Use of AASM Sleep ISR program fulfills meeting the requirement for conducting ISR.
    • Sleep ISR offers adult and pediatric tracks. A facility may choose to score either adult studies, pediatric studies or a combination.
    • Only 3 studies per quarter are required. It is not necessary to do 3 studies per track.
    • CEC are earned for each ISR assessment conducted.
    • Facilities using Sleep ISR with multiple related entities may link each facility site after enrolling each facility independently. After which the sleep technicians will be linked to each location.
  • ISR must always be reported as part of QA Quarterly report regardless of whether AASM ISR program is used.
  • ALL scoring technologists must participate in ISR.
  • ISR assessment must be conducted for each sleep facility.
  • A complete ISR report is signed and dated by the reference sleep specialist.
  • Subcontracted scorers are required to conduct ISR using assessments from the facility.
  • If the facility uses a corporate appointed reference sleep specialist, the facility director or medical staff member must review the results and take action if results fall below expectations.
  • The facility “reference sleep specialist” must meet standard B-2 – this individual cannot be a technologist.