US lawsuit on behalf of deceased patients alleges United Health denies care based on AI model with ninety percent error rate

Title is grossly inaccurate, but only because the article is grossly inaccurate.

The lawsuit argues that UnitedHealth should have been well aware of the "blatant inaccuracy" of nH Predict's estimates based on its error rate. Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims. This makes it obvious that the algorithm is wrongly denying coverage, it argues.

This is a total misunderstanding of statistics, and it’s questionable to me whether anything can be inferred from this fact. It is absolutely not the case that the model has an error rate of 90% though — all we know is that cases that are appealed have a high success rate, but what doctors are going to appeal cases where there is a low chance of success?

Also, it’s unclear where they even got this number. I have read the lawsuit filing and they lean heavily on the legal phrase “on information and belief” (see page 38 of the filing.). That said, I’m not a lawyer so maybe that’s common, but it doesn’t strike me as highly credible.

In the law of evidence, the phrase information and belief identifies a statement that is made, not from firsthand knowledge, but "based on secondhand information that the declarant believes is true".

The phrase is often used in legal pleadings, declarations under penalty of perjury, and affidavits under oath. It is often used in a phrase similar to: "The plaintiff is informed and believes, and upon such information and belief alleges". This "protects the maker of the statement from claims of outright falsehood or perjury".

It's blatantly unethical to base patient care decisions on an algorithm that hasn't been publicly validated. This would never be tolerated in a diagnostic process (e.g. AI mammography reading for tumor detection).

So why should it be permissible to use a privately developed, closed source algorithm for denial of care, when the owners' biases and incentives all favor that denial?

Regardless of the reported accuracy of the statistics that /u/unkz had problems with, let's have a public blind trial of unaffiliated physicians in applicable specialties reviewing a large random sample of the cases that the NaviHealth algorithm scored. If there's a significant difference in allowances/denials, that would imply that use of the algorithm must be subject to independent review.

In any case, the full STAT article reveals substantial evidence that the algorithm, accurate or not, was misused. Even according to UnitedHealthcare's original written policies, the algorithm's recommendations were supposed to be subject to human review. Instead, NaviHealth's case managers were forced to use the recommendations as a target for which they were held accountable.

Under Conway’s leadership, NaviHealth tightened employee performance goals. In 2022, the company set a target for case managers to keep patients’ stays in nursing homes within 3% of the days projected by the algorithm, according to confidential internal documents. Earlier this year, the target was narrowed to less than 1%.
The performance goals for a NaviHealth clinical case manager show that the individual was expected to keep patients’ rehab stays to within 3% of the days projected by the algorithm in 2022 (above). In 2023, the target narrowed to 1% (below).
The performance goals for a NaviHealth clinical case manager show that the individual was expected to keep patients’ rehab stays to within 3% of the days projected by the algorithm in 2022 (above). In 2023, the target narrowed to 1% (below).

The new target was listed in a document outlining a case manager’s performance goals for 2023, stating that the employee was expected to “ensure appropriate utilization” by maintaining a variance of “<1% overall” from the algorithm’s estimated length of stay. The variance is calculated on an aggregate level, so that if 10 patients were allotted 100 days by the algorithm, they would collectively need to stay no more than 101 days for the case manager to meet the target. The document explained that the 1% included “short stays,” which occur when patients return to the hospital quickly or opt to recover at home. The case manager’s goal for 2022 was less than 3%, but short stays were not mentioned in that year’s document.
The 1% target was also listed in a presentation shared with a larger group of employees this year. The presentation contained a detailed breakdown of employee performance metrics. In addition to length of stay, it calculated the average number of patients they were managing per day, the percentage of patients they spoke with in person versus by phone, and a month-by-month breakdown of how they fared against the 1% target.

If Amber Lynch and other NaviHealth case managers testify to the information they gave STAT News concerning the patently unjustified denial of medically necessary care, it's going to go very badly for UnitedHealthcare.