Inter-scorer Reliability Fact Sheet2017-12-20T18:15:24+00:00

Inter-scorer Reliability Fact Sheet

What is the intent of the ISR requirement?

The ISR program is an integral part of a facility’s quality assurance program. Close agreement of scoring personnel is necessary for accurate interpretation and quality patient management. Scoring technologists’ participation in an ISR program encourages continuous improvement of sleep scoring skills.

Which sleep facility staff members must participate in ISR?

  • All scoring technologists are required: Technical staff members who score sleep studies for the sleep facility must participate in the ISR program. This includes technical staff members that score as little as one record. It also includes staff that score for the sleep facility, but may work remotely or at a corporate location, such as subcontractors.
  • Facility gold standard: The facility director or medical staff member board-certified (individual meeting Standard B-2) serves as the facility’s gold standard and is required to participate in the ISR program. A corporate appointed board certified sleep specialist can also serve as the gold standard and must participate.

When must the sleep facility perform ISR?

ISR must be conducted on a quarterly basis as a part of the facility’s quarterly quality assurance program. Three polysomnograms are chosen randomly per quarter for use in ISR assessments for a total of 12 polysomnograms per year.

How must the sleep facility conduct ISR?

Quarterly Comparison

  • For each of the three polysomnograms compared quarterly, 200 consecutive epochs will be selected for comparison.
  • The facility gold standard and all scoring technologists must independently score the 200 consecutive epochs.
  • Scoring must include four parameters: staging (epoch by epoch), respiratory events, leg movements and arousals.

Quarterly Reporting

  • Comparison data, including a percentage agreement, must be tabulated between the gold standard and each of the scoring technologists for each of the four parameters. This data must be detailed in the facility’s quarterly report for each of the three polysomnograms compared.
  • For staging, epoch by epoch comparison must be reported.
  • The facility’s ISR policy must include a goal percentage agreement (to be set by the facility) for each of the four parameters compared. The quarterly report should indicate whether each scoring technologist has met the goal percentage and, if not, what actions were taken to address inconsistencies.
  • The quarterly report must be signed and dated by the gold standard indicating review and corrections if necessary.

AASM ISR Program: The AASM’s optional online mechanism for meeting the ISR accreditation standards.

The AASM posts a set of 200 epochs to the online ISR program each month. The AASM Gold Standard has scored the epochs already. Participating facilities must have all scoring technologists score the 200 epochs. The technologists’ scores are compared to the gold standard and a percentage agreement report is generated. The sleep facility’s gold standard must review their sleep technologists’ results and sign the report as part of the quarterly quality assurance program.

The AASM ISR program, like accreditation, is location specific. This means that although your staff completes their assessments through the AASM ISR program, the assessments cannot count for or be combined with any other accredited facility. Each accredited facility location has to purchase a separate, active ISR account to meet the standard. Once each facility has a separate, paid account for AASM ISR, then you may link staff to each location. Linking Staff to each location ensures your staff only need to complete the monthly assessment one time and their score will be applied to each location to which they are linked. Standard F-8 from the AASM Standards for Accreditation states that “ISR assessment must be conducted for each sleep facility.” Because your accreditation is location-specific, you must also have separate ISR accounts. AASM ISR is an option to fulfill Standard F-8 and is not required to fulfill the Standard. Nevertheless, each location must complete an inter-scorer reliability program each month, at each location.

Educational and reporting features of the AASM ISR Program include:

  • Monthly record review videos, that provides insight to the Gold Standard Panel decision process.
  • Free Continuing Education Credits – 12 CECs per year.
  • Monthly scoring records are provided as well as national gold-standard scores.
  • Immediate feedback provided to technologists and supervisors.
  • Center and national results are provided for each record.

Applicable Accreditation Standards

Scoring personnel must be one of the following: RST, RPSGT, CPSGT, respiratory therapists with the sleep disorders specialists certification (either CRT-SDS or RRT-SDS), or medical staff members/PhDs board-certified in sleep medicine (as defined in Standard B-2). Non-registered sleep technologists (as defined in Standard B-10) may score only under the supervision of one of the above while adhering to standard B-10.

Inter-scorer reliability must be determined between each scorer and reference sleep specialists as defined in standard B-2 or a corporate appointed board certified sleep specialist provided the facility director or medical staff member attests that the results were reviewed and action will be taken if result fall below expectation. Inter-scorer reliability assessment must be conducted for each facility.

Key Things to Keep In Mind

  • A complete ISR report is signed and dated by the gold standard reference sleep specialist.
  • ALL scoring technologists must participate in ISR.
  • A complete ISR report reflects a full quarter (three months) of data.
  • ISR assessment must be conducted for each sleep facility.
  • For facilities using AASM’s ISR Program with multiple related entities; each facility must be independently enrolled after the facility enrolls each site then sleep technicians may then be linked to each location.
  • Subcontracted scorers are still required to conduct ISR using assessments from the facility or through facility AASM ISR Program enrollment.
  • Corporate appointed reference sleep specialist can serve as gold standard, provided the facility director or medical staff member attests in writing that the results were reviewed and action will take place if result fall below expectations. The facility “reference sleep specialist” must meet standard B-2 – this individual cannot be a technologist.
  • Reference the Accreditation Reference Manual for specifics on what the ISR policy should include.