A Deep Learning Platform for GP Referral Triage
Countering bias in the health system
Heart disease is the leading cause of preventable mortality in Aotearoa, New Zealand. It is a disease that disproportionately affects certain groups such as Māori – who have higher rate of ischaemic heart disease (or coronary heart disease) and stroke but have lower access to health care. Intervention must occur early to prevent serious illness, but the data shows that the New Zealanders most susceptible to this disease – Māori and Pasifika – may suffer from bias in the health system.
This is partly due to inefficient triaging, with some patients being unnecessarily accepted for a hospital clinic appointment and as a result bumping those more in need further down the queue. Under a paper-based system, it was impossible to understand how primary-care referrals were being prioritised, but the switch to electronic referrals enabled the ability for Precision Driven Health researchers to get a better understanding of who is being prioritised for early referral to a cardiologist.
Our recently completed Precision Driven Health project – A Deep Learning Platform for GP Referrals Triage – aimed to address the inequities in cardiovascular care by improving the triage process. We worked with Māori health researchers and He Kamaka Waiora to develop strategies to address and counter any implicit bias in terms of unequal access to specialists for key populations. A key objective of the project was to identify potential bias in the existing triage dataset and provide a machine learning solution that mitigates implicit bias, alleviating the access issue by moving patients most in need up the queue. The first step was a GP referral triage system using available patient records from the Waitematā District Health Board (WDHB). Referrals to cardiology specialties in the years 2015 to 2018 were included in this dataset. To predict the priority levels of referrals, machine learning models were trained with the WDHB dataset using the clinician’s triage priority level as the ground truth. Based on our consultations, binary classification models were trained to discriminate between P1-2 and P3-4.
According to the first attempt to identify potential biases, the number of patients in each level of priority (P1-P4) in the triage data was not significantly different between ethnic groups. There were however other aspects of the patient profile and their experience with the triage process that need to be taken into consideration. According to the WDHB population data, Māori patients had a considerably lower chance of being referred to a hospital, to be triaged or triaged at a higher priority level (P1 or P2).
Consultations with clinical experts and inquiries into the data set revealed other possible sources of bias, including the rate of Māori patients who had heart failure in the past six months or who had a history of health problems compared to non-Māori patients. While this rate alone cannot demonstrate bias during the triage process, it can imply that these groups of patients have to be prioritised in ways that balance their inequity in health outcomes. The only available data on health outcomes in this study revealed that Māori patients were significantly more likely to develop heart failure within one year of referral than their non-Māori counterparts. Accordingly, it’s possible the triage didn’t sufficiently address this inequity gap and still resulted in relatively poorer outcomes for Māori.
Lead researchers on this project:
- Dr Edmond Zhang, Principal Investigator, Senior Data Scientist, Orion Health
- Reece Robinson, Associate Investigator, VP Engineering, Orion Health
- Dr Patrick Gladding, Clinical Lead, Cardiologist, Waitemata District Health Board
Other researchers on this project:
- Associate Professor Peter van de Weijer, Triage Expert, Director Women’s Health, Auckland District Health Board
- Professor Bernhard Pfahringer, Machine Learning Expert, Professor of Computer Science, University of Waikato
- Dr Yun Sing Koh, Machine Learning Expert, Senior Lecturer of Computer Science, University of Auckland
- Delwyn Armstrong, Clinical Data Expert, Head of Analytics, Waitemata District Health Board
- Haze White, Maori health researcher, Wai-Research
- Elica Mehr, Maori health researcher, Wai-Research
- Susan Smith, Referrals Expert, GP, Eastmed Doctors
- Umit Holland, Project Coordinator, CNS Researcher, Waitemata District Health Board
- Bronwen Gilson, Clinical Data Expert, Information Analyst, Waitemata District Health Board
- Steve Nicholas, Product Expert, eReferrals Director, Orion Health
- Kevin Bayes, Machine Learning Engineer, Senior Lead Engineer, Orion Health
- May Lin Tye, Business Analyst, Orion Health
- Anna Spyker, Machine Learning Engineer, Software Engineer, Orion Health
- Michael Hosking, Clinical Coding Expert, Clinical Product Specialist, Orion Health
- Ning Hua, Data Scientist, Orion Health
- Dominic Yuen, Software Engineer, Orion Health
- Aaron Zhang, Data Science Intern, University of Auckland
August 2018 – September 2021
Publications and Presentations
E. Zhang, R. Robinson and B. Pfahringer, “Deep Holistic Representation Learning from EHR,” 2018 12th International Symposium on Medical Information and Communication Technology (ISMICT), Sydney, NSW, 2018, pp. 1-6.