The modern sleep disorders center is seeing a shift in how day-to-day tasks are being performed. Artificial intelligence technology has opened the door for highly accurate auto-scoring solutions that will likely play a bigger role in AASM-accredited facilities in the future. While the U.S. Food and Drug Administration has oversight over many of these solutions, FDA requirements for scoring accuracy is not very stringent. Additionally, sleep medicine professionals may not feel confident with available solutions since they are radically different than the status quo of manual scoring. To help usher in auto-scoring solutions, the AASM board of directors approved a plan to create a two-year pilot certification program for software that has auto-scoring functionality. The pilot certification is limited to sleep stage scoring. The purpose of the certification program is to independently verify the performance of auto-scoring systems. To obtain certification, software must demonstrate accuracy that is equivalent or better than human scorers.
To assist in development of the program, the AASM has contracted with a number of scorers who are scoring sleep studies that are not publicly available. With the help of a statistician who specializes in inter-rater reliability, a methodology will be created to assess the performance of the scoring software. This methodology will be shared prior to accepting vendor applications.
Specific details on the logistics of obtaining certification will be provided at a later date. However, at a high level, we anticipate the following requirements:
- The auto-scoring software is provided to the AASM to install on an AASM-owned computer.
- A video call will be scheduled between the vendor and an AASM representative. The purpose of the call is for the vendor to ensure the software is performing as intended and to witness the autoscoring of the sleep studies AASM provides.
- After the video call, the AASM will compare the software’s answers to the panel of expert scorers.
- If an acceptable accuracy level is found, the software will be certified for stage scoring for two years.
Software that achieves certification will be given a digital and physical certificate that can be used for promotional purposes and be listed on AASM’s website to assure AASM members that the software has passed AASM requirements for certification.
The AASM anticipates accepting applications near the end of 2022 or beginning of 2023. Applications for the two-year pilot certification will be accepted through December 31, 2023.
If you represent a company that uses auto-scoring technology and would like to learn more about the AI/Autoscoring Pilot Certification Program, please email AIGSP@aasm.org.