The Effects of Artificial Intelligence on Healthcare Delivery

In 2023, artificial intelligence and machine learning (AI/ML) emerged as both a potential boon and a concern in healthcare as stakeholders and the Biden Administration worked to balance rapid advancement with the need for security, transparency, and effectiveness. AI/ML in healthcare holds promise as a tool for improving healthcare delivery, including by reducing administrative burden on healthcare providers, assisting physicians with clinical decision making and outcome prediction, and even aiding physicians in surgical settings. Despite the promise of innovation and improved healthcare delivery, AI/ML also presents certain risks and challenges that should not be overlooked. Regulators and stakeholders continue to grapple with how to best regulate AI/ML in healthcare, as evidenced by efforts from the White House, World Health Organization (WHO), U.S. Food and Drug Administration (FDA), U.S. Department of Health and Human Services (HHS), and others to emphasize the importance of safety and effectiveness while recognizing the potential for innovation.

AI/ML and Bias

Explicit and implicit bias is known to contribute to disparities in healthcare,and if not addressed, can affect the quality, safety, and effectiveness of AI/ML in healthcare technology. Generative AI/ML consists of AI/ML algorithms that create new content based on information that has been inputted to train the model. The type and quality of information entered into a generative AI/ML model dictates the effectiveness of the model. In 2023, ChatGPT, the AI/ML chatbot developed by OpenAI, received significant attention. ChatGPT was trained on a wide swath of content from across the internet, which is why it can be successful in essay writing, taking (and passing) exams, and recipe creation, for example, but it can also further spread bias due to both implicit and explicit biases found throughout its source material.The old computer adage “garbage in, garbage out” certainly applies to AI/ML.

This general risk of biased results can be amplified when AI/ML is used in healthcare. When generative AI/ML models are used to predict healthcare outcomes or assist in clinical decision making, the models can heighten inequities in race, ethnicity, gender, disability, sexual orientation, and socioeconomic status.AI/ML algorithms trained on predominantly white patients may not perform well when used on Black, Asian, or Hispanic patients, as there may be significant variations in certain medical risks and social inequities among populations.Treatment strategies and medication recommendations may differ as a result. The same can be said for differences in gender. AI/ML algorithms trained on predominantly male patients may not effectively assist physicians in evaluating female patients for similar non-gender-specific conditions, such as cardiac or gastrointestinal conditions. For example, a recent study found that an AI/ML algorithm was less accurate for Black people than for white people when detecting early-stage breast cancer.Another found gender bias in liver disease screening, missing 44% of cases of liver disease in women compared to 23% of men.Failure to train AI/ML algorithms on data from diverse populations or updated treatment guidelines that include race or ethnic disparity adjustments can lead to further engendering certain biases in treatment.

In addition to training generative AI/ML with diverse data, it is also important that it be trained to interpret data based upon the correct proxies. In 2019, a study published in the journal Science found that an AI/ML algorithm commonly used in healthcare to assess the need for follow-up care was biased against Black patients.The goal of the algorithm was to assist in ranking patients who needed greater follow up care in care coordination programs.The algorithm took into account insurance claims, demographics, insurance type, diagnosis and procedure codes, medications, and detailed costs, while notably excluding race.The algorithm also ignored certain underlying chronic conditions experienced by Black patients, resulting in the same risk levels being assigned to Black and white patients although the Black patients were empirically sicker than white patients. The resulting calculation found Black patients who were actually sicker required the same or less follow-up care.According to the study authors, “[l]ess money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick white patients.”The authors concluded that the bias stemmed from the algorithm being trained on “healthcare costs as a proxy for healthcare need.”Once the algorithm was reconfigured to account for physiological variables as opposed to insurance claim data, the results were significantly altered and nearly tripled the percentage of Black patients who should have been identified as requiring more care from 18% to 46%.As this study demonstrates, an overreliance on AI/ML algorithms to identify cost-savings measures may lead to ineffective and inaccurate conclusions if analyses—and the base data and assumptions on which they are trained—are not carefully reviewed.

AI/ML should be used to improve access to care, but there are some concerns that the gap in access to care could instead be widened. The American Medical Association’s Policy on Augmented Intelligence in Healthcare (AMA Policy) advocates that healthcare systems that employ AI/ML should “identif[y] and take[] steps to address bias and avoid[] introducing or exacerbating health care disparities, including when testing or deploying new AI tools on vulnerable populations.”However, upfront costs associated with AI/ML-enabled diagnostics and treatment options can be high, and for struggling rural healthcare systems, the costs may be prohibitive.At a time when hundreds of systems that provide care to uninsured or underinsured struggle to keep their infrastructure from crumbling, investment in AI/ML may not be seen as a priority.While larger, well-funded systems continue to invest in the latest technologies, those who cannot access that care due to cost or geographical restrictions may miss out on advanced treatment options, further expanding disparities in care.

Patient Privacy and Informed Consent

Patient autonomy, privacy, and informed consent must be considered as healthcare providers continue to utilize AI/ML and predictive algorithms to diagnose and treat patients. Healthcare providers must clearly communicate potential risks, benefits, and outcomes of particular treatments to patients and caregivers to support their ability to make informed decisions about their care.Both the physician and patient (or caregiver) must understand the effectiveness of the AI/ML system and how it will be incorporated into the patient’s care, any limitations of the AI/ML, whether the patient’s information will be secure, and if there is any potential for bias or harm. The 2023 American Medical Association Principles for Augmented Intelligence Development, Deployment, and Use (AMA Principles) caution that “[w]hen AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients, and to allow each to understand how decisions impacting patient care or access to care are made.”Privacy risks associated with using AI/ML, including whether algorithms will be trained on identifiable data, should be assessed in accordance with the Health Insurance Portability and Accountability Act (HIPAA) and other applicable state and federal privacy laws. Risks associated with breach and the likelihood that an individual could be reidentified in the event of breach should be considered, and as AI/ML’s ability to reidentify datasets improves and the technology advances, serious consideration should be given to the individual patient’s privacy risks where PHI or sensitive health data was used to train AI/ML. The ABA has previously reported on AI and data privacy concerns in the November 2023 edition of eSource.

Clinical Decision Making

Generative AI/ML can augment clinical decision making through enhanced diagnostic tools and alternative treatment options. Data or images may be fed into an AI/ML diagnostic model, and the AI/ML model can predict outcomes or assist in diagnosis by identifying patterns or identifying potential disease by tying together seemingly unrelated symptoms. AI/ML can also assist providers with review of diagnostic tests, such as with AI-enabled digital pathology or mammography. While the provider will still make the ultimate diagnosis, these platforms can be used to identify and magnify cells or anomalies that might be overlooked by the human eye or apply predictive algorithms to identify early cell changes that, based on prior outcomes, may be more likely to become malignant. Without AI/ML assistance, providers may be limited by their own diagnostic experience and the current standard of care, which can result in diagnostic decisions based on static decision trees.Generative AI/ML often incorporates data across hundreds or thousands of sources, updates in real time or near-real time, continually building the knowledge base and library of information on which diagnostic decisions are based. For example, hospitals across the U.S. perform approximately 3.6 billion imaging procedures annually, and those images can be used to simultaneously train and update the AI/ML, improve detection capabilities, and reduce error.Giving providers access to greater sources of data and emerging standards of care can improve diagnosis and treatment outcomes.

For better or worse, patients have already turned to AI/ML to expand their personal access to medical information and diagnosis. For example, in 2023, in a widely reported case, a mother took her child to visit seventeen doctors after the child complained of a persistent toothache and experienced stunted growth. None of the doctors were able to diagnosis the child’s ailment.The mother turned to ChatGPT, entering her son’s symptoms into the AI/ML chatbot, which in turn suggested her son may be suffering from a rare neurological condition, tethered cord syndrome. Soon after, a neurosurgeon confirmed the diagnosis.

Empowering patients and caregivers to research their own symptoms helps them advocate for themselves, yet providers can feel pressured to order diagnostics and procedures that may be unnecessary because “Dr. ChatGPT” suggested a rare and unlikely condition. ChatGPT does not consider context or nuanced symptoms when making “diagnoses,” and its results rely on the accuracy of the patient prompt.Patients might confuse medical terminology, or omit certain symptoms, and return wildly varying results that can lead a provider to chase “zebras rather than horses,” as the saying goes.

Providers should be careful not to rely too heavily on generative AI/ML. The AMA Principles clearly state that “[c]linical decisions influenced by AI must be made with specified human intervention points during the decision-making process. As the potential for patient harm increases, the point in time when a physician should utilize their clinical judgment to interpret or act on an AI recommendation should occur earlier in the care plan.”AI/ML should not provide the clinical decisions, but the provider should instead incorporate the AI/ML recommendation into a treatment plan or diagnosis rather than defer to the AI/ML outright. Where generative AI/ML is poorly trained, or its intended use is misunderstood by developers or providers, the risk of an incorrect diagnosis or inappropriate treatment regimen can increase, which could increase liability risk for providers who blindly defer to the technologies or who rely on them well outside the standard of care.These vulnerabilities and the likelihood for error should be assessed and appropriately disclosed to the patient, and alternative options should be considered where appropriate.While the potential for increased liability has resulted in some providers resisting the use of AI/ML altogether,as AI/ML becomes incorporated into the standard of care, it may be riskier for providers not to use such tools.

There are also concerns as to whether insurers are using AI/ML predictive tools appropriately for claims review, or whether they are being used as cost-saving measures to improperly deny patients coverage for medically necessary services.Class action lawsuits have been filed against two large health insurers by patients alleging that the insurers used an AI algorithm to automatically deny patients if they did not meet certain preset criteria, often overriding provider determinations and recommendations.This alleged use of AI/ML to review and process claims has stoked fears that claim denials will increase, and its use in an already controversial preauthorization system could delay time-sensitive but costly treatment.

Conclusion

AI/ML in healthcare delivery has the potential to improve patient care and outcomes. Over the past few years, and especially in the wake of ChatGPT, excitement about innovation in this space has been unparalleled. However, the risks give some providers pause as AI/ML has demonstrated bias and raises concerns about patient privacy, autonomy, adequacy of informed consent, as well as safety and efficacy concerns that could increase provider liability. Thus, as pressure to use AI/ML in healthcare delivery increases, stakeholders and regulators must continue their efforts to balance the risks and benefits while remaining focused on providing care with a human touch.