An Algorithm Denied Gene Lokken's Care. Pope Leo Named the Pattern.

The new encyclical on AI is both a tech document and a defense and expansion of subsidiarity. Secular readers should take note.

Photo by Grant Whitty on Unsplash

In the spring of 2022, Gene Lokken, a 91-year-old man in Minnesota, fell and fractured his leg and ankle. He was admitted to a skilled nursing facility, and his doctors recommended a course of rehabilitation. Two and a half weeks into the rehabilitation, his Medicare Advantage plan, administered by UnitedHealthcare, cut off his coverage. The decision did not come from a doctor at the insurance company. It came from an algorithm called nH Predict, developed by a UnitedHealth subsidiary, that had estimated, based on patterns in the histories of other patients, that Lokken should not need any more rehabilitation. His treating physicians disagreed. The algorithm prevailed. Lokken’s family paid out of pocket, up to $14,000 a month, for the next year of his care. He died still in that care, in nursing his physicians had asked for and his insurance, advised by software, had refused to cover.¹

Lokken is one of the named plaintiffs in a federal class action against UnitedHealth Group that is, as I write this, working its way through discovery in the District of Minnesota. In March of this year, the judge in that case ordered UnitedHealth to disclose a broad range of internal documents about how the algorithm was designed, how it was used in coverage decisions, and whether it was meant to override the clinical judgment of treating doctors. The plaintiffs cite a striking number from the algorithm’s track record: of denials that were appealed, roughly nine in ten were ultimately reversed. The structure was producing the wrong answer almost every time, on a population that had standing to push back. The Senate Permanent Subcommittee on Investigations reached a similar conclusion in a 2024 report covering several of the largest Medicare Advantage insurers, and in January of this year the federal Medicare program launched a new model that brings AI-driven prior authorization further into the way decisions are made about elderly patients’ care.

As in many cases of institutional or algorithmic failure, the institutions and algorithms themselves are not necessarily malicious. They are not even necessarily badly run. But they are structured so that no person you can reach is responsible for what is happening to you, and so that the cost of pushing back through the available channels is higher than the cost of giving up. And they’re also structured in a way where the interactions are so complex that finding the actual point where the decision went wrong is almost impossible. I call this pattern structural hobbling, and I’ve argued in a recent article that it is one of the most regressive features of modern American life, because the people who can route around it (with lawyers, with connections, with time) are not the people who can least afford to be hobbled in the first place.

On Monday, Pope Leo XIV published an encyclical called Magnifica Humanitas, his first major social encyclical. It’s being covered as a document about artificial intelligence, which it is. But it also a restatement of a political truth that crosses religious and secular lines, extending a centuries-old Catholic doctrine called subsidiarity into the digital age. In the process of both it names a pattern that both political theorists and government reformers need to take seriously.

When Catholics, the founders, and institutional economists agree

Subsidiarity, as the Catholic tradition has used the term since at least Pius XI’s 1931 encyclical Quadragesimo Anno, is the principle that decisions should be made at the level closest to the people affected. Individuals, families, neighborhoods, parishes, professional associations, and local governments do what they can. Higher levels of authority step in only to do what the lower levels genuinely cannot, and even then only to support, not to displace. The principle has a Hayekian sibling (the knowledge problem: distant decision-makers cannot see what local actors can), an Ostromian one (polycentric governance: complex problems usually do better with many overlapping centers of authority than with one) and a deeply American one (federalism as the safety valve for democracy). It is one of the few intellectual frameworks I know of that travels easily across the Catholic tradition, the classical liberal tradition, and the empirical institutional-economics literature.

Pope Leo connects AI and subsidiarity throughout the encyclical, but paragraph 71 lays out where he sees the real danger: “Here, the highest level is not the State, but rather major economic and technological actors that exercise de facto power over the conditions of everyday life. This level, which monopolizes expertise, data and decision-making authority, involves companies and platforms that define conditions for access, rules of visibility, forms of interaction, and even economic opportunities.”

Pope Leo XIV is clarifying what folks from all political viewpoints should appreciate, namely that the problem is not either government power or corporate power, but power writ large. In this case, the entity from which power must be devolved, in our era, is not the State alone. It is the small set of actors (private or public, but in the case of AI largely private) who, between them, set the terms on which hundreds of millions of people work, learn, communicate, borrow money, find jobs, and reach each other. AI and tech companies more broadly effectively operate as bureaucracies, with the same opaque procedures, the same diffuse responsibility, and the same absence of any human at the other end of the line.

The principle of subsidiarity offers a roadmap for navigating AI governance precisely because it protects against the two great centralizing dangers: state power deployed to eliminate competing viewpoints and corporate monopoly that forecloses meaningful choice. The deep pluralism that characterizes human life requires that people at every level retain the ability to use technology on their own terms and subsidiarity is the structural principle that makes that possible. In the AI space, this means allowing multiple models to flourish, preserving individual agency over data, devolving decision-making authority to local communities and making the costs and responsibilities of different approaches legible and accountable. What we’re really talking about is the same old fight between centralization and pluralism, playing out on a new technological terrain.

Algorithms and structural hobbling: not identical, but connected

Gene Lokken did not know what subsidiarity was. He also did not need to. What his family experienced, in the language of my own work, is the textbook case of structural hobbling: a constellation of well-meaning rules and procedurally legitimate decisions that produced a result no one in particular intended, with no one in particular accountable, and with the cost falling on people who could not absorb it. The Medicare Advantage rules were defensible in the aggregate. The algorithm was sold to the insurer as a tool to help “inform providers, families and other caregivers about what sort of assistance and care the patient may need.” The internal target the company set for its employees, that patient stays should stay within one percent of the algorithm’s prediction, was the kind of operational goal that would look reasonable on a slide deck. The decision to discharge Gene Lokken’s coverage after seventeen days was made through a process that, viewed step by step, was not obviously malicious at any single step. None of that helped his family pay the bill.

In my academic work I argue that this pattern is one of the most under-theorized sources of injustice in contemporary life, and that the standard left-of-center frame (which locates injustice in either individual bad actors or in the inactivity of the state) systematically misses it. Pope Leo XIV, in paragraph 95 of the encyclical, echoes the 1891 encyclical Rerum Novarum of Pope Leo XIII, who clearly inspires his own work and his own papacy: “In many cases within the digital context, control over platforms, infrastructure, data and computing power does not rest with States, but with major economic and technological actors. These entities effectively set the conditions for access, determine the rules of visibility and shape the very possibilities for participation.” While one could see this as a condemnation of corporate power itself, that’s not really the point. As he makes clear earlier in the encyclical, “In practice, however, technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it.” The term “regulate” here emphasizes the often complex relationship between government and corporate power, with the two frequently compounding each other. (If I were going to quibble with the Pope on anything in this encyclical it’s that he underemphasizes the way government power fuels corporate power, but more on that in a later post.)

My structural hobbling framework describes what he is naming as four interlocking features of tech broadly but AI more specifically. The first is scale: the platform mediates the lives of enough people that exit is not realistic. The second is opacity: the rules are not legible to the people they govern. The third is diffuse responsibility: no one decision-maker can be identified, summoned, or replaced. The fourth is regressive impact: the people best equipped to push back through the available channels are not the ones who most need to. Pope Leo’s encyclical names all four, in its own vocabulary, in the span of about two pages.

This is a similar argument to that made by government reformers like Jennifer Pahlka, whose work I’ve discussed before. Her description of a cascade of rigidity produces the same brittleness of institutions and citizens that I’ve diagnosed here. That a technocratic reformer, the Catholic Church, and a political theorist arrived at the same diagnosis is not coincidence. To me at least it provides strong support for a very real phenomenon and indicates that we need a much more expansive conversation about the complex interaction of private and public governance rules rather than seeing the two as distinct sets of rules with different goals.

What subsidiarity requires of Big Tech

It would be one thing if the encyclical simply named the problem. Naming a problem and prescribing a response are different acts, and most of what is written about AI right now does the first and not the second. The encyclical is unusual in that it does both, and it does the second by translating a centuries-old principle into concrete design tests for a new kind of institution.

The relevant passages run from paragraphs 71 through 72 and from 108 through 110. Stitched together and translated out of the encyclical’s vocabulary, they propose something like the following: Power should be held at the level closest to the people affected, which means that algorithmic decisions affecting livelihoods, credit, healthcare, or political participation should not be made in jurisdictions and on terms that the affected community has no representation in. Decision-making processes should be legible, which means that a person whose access is denied should be able to understand the basis for the denial in language a non-engineer can follow. Responsibility should be identifiable, which means that an actual human being, with a name and a job description, should be reachable when the system errs. The community affected should have a real voice in design, not be informed of the design after the fact (”rather than confining their role to mere oversight after the standards have been set elsewhere,” in the encyclical’s words at paragraph 109). And the people who do the unseen labor that makes these systems work (the data labelers, the content moderators, the workers in the resource-extraction chains) should be visible in the accounting. Incidentally, these are all principles of good governance that are violated almost more often than they’re followed in our current system.

I would add a sixth criterion from my own work that the encyclical does not state in this form, although it gestures at it. The system should be auditable. If an algorithm is doing the work of an institution, the algorithm itself, including its training data and its failure modes, should be open to inspection by parties the institution cannot capture. We have legal and institutional vocabularies for this in the public sector. Inspectors general. Independent audits. Statutory disclosures. There is no good reason these vocabularies should stop at the edge of the platform.

Read together, these criteria amount to a scorecard. You can apply it to a hospital network, a federal agency, a state DMV, a content platform, a credit-scoring service, a hiring algorithm, or a school district’s automated grading system. The scorecard cuts across partisan identity, because it describes what any person of any background needs in order to extend trust to an institution that mediates their life. It also doesn’t treat public and private power as though they are different animals with different intentions. It assumes that centralized power always has the capacity for rigidity if not abuse and therefore requires vigilance. This is what subsidiarity actually requires, in concrete terms, when you take the doctrine seriously and apply it to the institutions that now do the work that institutions used to do.

Subsidiarity and AI

A few months ago I wrote that talking about “AI policy” was a lot like talking about “computer policy”: the category is too broad to support useful regulation, and the right approach is to identify specific harms in specific use cases and target the response with precision. I stand by that argument. The encyclical does not, in my reading, require its abandonment.

What the encyclical does require is that the trust-but-verify framework be applied not only to the use cases on the demand side, but to the design choices on the supply side. In paragraph 104 Pope Leo says, in plain language, “we cannot consider AI to be morally neutral... every technical tool embodies choices and priorities through what it measures, ignores and optimizes.” This is the move that I think will sit with me longest. The targeted-regulation approach I defended works for deployment. It does not, by itself, work for design. By the time a tool reaches deployment, the anthropology of the human person that the designers held has already been encoded in what the model is trained to maximize, what it is trained to ignore, and what counts as an error in the loss function. We can either ask designers to make that anthropology visible, or we can let it shape our lives invisibly. The encyclical is right that those are the two options.

What this changes, in practice, is that the regulatory conversation needs an upstream half. Not in place of the targeted-harm approach but as a complement to it. We need a vocabulary for what it means for a tool, before it is deployed at all, to have been built in conformity with the kind of subsidiarity Pope Leo is describing. We do not yet have that vocabulary in American public life. The encyclical is one place to start building it. But it’s also not the case that formal government regulation is the only way to do this upstream work. It might even be the worst way to do this work. Subsidiarity gives us a range of options for holding companies accountable that don’t require bloated government bureaucracies to make design decisions about the design of private tech bureaucracies.

It’s no accident that Anthropic’s co-founder Christopher Olah was invited to the presentation of the encyclical and to speak afterward. He ended his remarks with a telling plea, calling for a partnership between the Church and tech companies, arguing: “We need informed critics who will tell the labs when we are failing. We need moral voices that the incentives cannot bend.” His remarks demonstrate two things: that it’s not impossible for tech companies to self-regulate (but they need support to do so) and that it’s not just the government who can exert pressure in this space. These are both important lessons worth remembering.

Takeaways for non-Catholics

A document like this can be hard to engage with for readers who do not share its theological starting points, and I have some sympathy for the impulse to set it aside on that basis. I want to push back gently on the impulse for two reasons.

The first is that Catholic social teaching is one of the most underused intellectual resources Americans have for thinking about institutional design across partisan lines. Subsidiarity has been worked out as a principle for over a century by people who were trying to defend the integrity of families, parishes, neighborhoods, and small associations against both the encroachments of the state and the encroachments corporations. It is one of the few frameworks in our intellectual landscape that takes the middle layer of society seriously as a thing in itself, not as a residual variable between the individual and the state. You do not have to be Catholic to find it useful. Hayek, Ostrom and our founding fathers all got to a related place from entirely different starting points, and the convergence is itself a reason to take the principle seriously.

The second is that this encyclical, more than most, is structured to be read by people outside the tradition. Pope Leo is explicit, in his opening chapter, that he understands the document as an offering for dialogue, and he treats the Catholic vocabulary as a contribution to a shared conversation rather than a closed system. The institutional theory holds whether or not you accept the theological frame underneath it. Take what you can use.

For Gene and the families who come next

Gene Lokken did not get his coverage back. The structure that took it away is still standing, and the lawsuit his family is part of is still in discovery. Whatever the lawsuit ultimately produces, the algorithm is still running, the Medicare Advantage incentives that made the algorithm attractive are still in place, and a new CMS model expanding AI-driven prior authorization is now live in six states. The structure has a name now, in a vocabulary that Catholics and Hayekians and a number of other people share. The principle for taking the structure apart has a name too. None of this fixes the next denial, and none of it puts a human being on the other end of the appeal that Gene Lokken’s physicians could not get answered in time.

A human on the other end is a smaller thing than the encyclical is offering. It is also more than most of the documents I have read on this subject manage. For now, at least, I’ll take it, but I’ll have a lot more to say on the failure of humans on the other end next week too.

Your Turn

Until next week, I’d love to hear reader thoughts on the encyclical: what it got right, what it missed (if anything), and how we should be thinking about both AI design and deployment in the coming days. Drop ideas, comments, and criticism in the comments! And please share and subscribe if you like what you read.

1. It’s actually even more complicated, which I’ll tackle in next week’s post. My own research on healthcare policy suggests that the algorithm may very well have been correct: 91 year olds rarely need rehabilitation, but skill nursing facilities (SNFs) are often the only existing alternative to repeat hospital admissions, at least that insurance is willing to pay for. The algorithm may have identified one of those spaces in the system where we provide medically unnecessary care simply because the structure of our healthcare system doesn’t give us medically/humanely reasonable alternatives. This doesn’t excuse UnitedHealth’s rejection of the claim given that (if I’m correct) a SNF may have been the only humane option, but points to the fact that our system is terrible and broken across multiple dimensions.