Inter-scorer Reliability (ISR) Fact Sheet
Applicable Accreditation Standards
What is the intent of the ISR requirement?
The ISR program is an essential component of a quality assurance program. Close agreement among scoring personnel is crucial for accurate interpretation and effective patient management. Participation in an ISR program promotes continuous improvement of sleep scoring skills for scoring technologists.
Which must participate in ISR?
- All scoring technologists: Technical staff members who score sleep studies for the sleep facility. This includes:
- Those who score even a single record.
- Subcontracted scorers.
- Reference Sleep Specialist: The Network Director or a staff member who is board-certified (meeting Standard N-1) serves as the reference sleep specialist.
Conducting ISR
ISR is a Network Standard. This allows networks to have a single inter-scorer reliability program. Technical scoring staff to perform a single (monthly) scoring comparison, even if they work at multiple locations within the Accreditation Network.
Quarterly
- Three polysomnograms (PSGs) must be compared each quarter, using 200 consecutive epochs per study, for a total of 12 polysomnograms per year.
- The reference sleep specialist and all scoring technologists must independently score the epochs for each PSG used in the comparison.
- Scoring must include four parameters: staging (epoch by epoch), respiratory events, leg movements and arousals.
- An acceptable level of agreement, typically expressed as a percentage, must be achieved between each scorer and the reference sleep specialist.
Quarterly Reporting
- Comparison data must be tabulated for each of the four parameters, showing the acceptable level of agreement.
- The quarterly report should indicate whether each scoring technologist has met the acceptable agreement level. If not, it should outline actions taken to address inconsistencies.
- The Network Director must sign and date the quarterly report, confirming review and noting any corrective actions taken, if necessary.
Use of AASM’s Sleep ISR Program
- The AASM offers an optional online tool to meet ISR accreditation standards.
- Each month, the AASM posts a set of 200 epochs to the Sleep ISR program, pre-scored by the AASM Gold Standard.
- All scoring technologists from participating entities must score the 200 epochs.
- The technologists’ scores are compared to the gold standard, and a percentage agreement report is generated.
- The Network Director must review the sleep technologists’ results and sign the report as part of the quarterly quality assurance program.
Key Things to Keep In Mind
- Eligible Scoring Personnel: Must be one of the following: RST, RPSGT, CPSGT, CRT-SDS, RRT-SDS, or medical staff members/PhDs board-certified in sleep medicine.
- Scoring staff are identified as those that score for interpretation/reporting purposes (e.g., day staff) but it includes those that must review data during the acquisition to make a determination regarding adjusting patients therapeutic devices (e.g., PAP).
- AASM Sleep ISR Program: Fulfills the requirement for conducting ISR.
- Sleep ISR offers both adult and pediatric tracks. The service location may choose to score adult studies, pediatric studies, or a combination.
- Only 3 studies per quarter are required, regardless of the track.
- 1 CEC can be earned for each monthly assessment, but it must be claimed and is not automatically awarded.
- Reporting Requirements:
- ISR must be included in the QA Quarterly Report, regardless of whether the AASM ISR program is used.
- ISR assessments must be conducted for each scorer in the network, regardless of the number of studies scored.
- A complete ISR report must be signed and dated by the Network Director.
- Subcontracted scorers are required to participate in ISR.
- Reference Sleep Specialist must meet Standard N-1 and cannot be a technologist.