To what extent is the problem here that exit isn't possible because the structure of health insurance has locked people in (employer provided, government mandates on what it needs to cover so a low cost option isn't available, etc). I would imagine in a world where this bad PR would cause people to cancel their policies, the companies would be much quicker to fix them.
That’s a big part of the problem. The other more complicated part is how entwined government and private power are and how perverse and overlapping incentive structures create downstream rigidity that makes good decision making impossible. That’s the subject of next week’s post. But lack of competition and exit make all of it worse. There’s no incentive for reform. Or very few opportunities for market or any other kind of discipline.
"But they are structured so that no person you can reach is responsible for what is happening to you, and so that the cost of pushing back through the available channels is higher than the cost of giving up. And they’re also structured in a way where the interactions are so complex that finding the actual point where the decision went wrong is almost impossible."
You might find the recent book, _The Unaccountability Machine_ by Dan Davies of interest. His term for this phenomenon is "accountability sink."
Another great essay, you sure make us think! For those who have the time and inclination, two links below to articles/podcasts that touch upon similar topics you address, one an interview between Ezra Klein and Yuval Harari, and the second one an Essay by Yuval Levin in one of my favorite magazines The New Atlantic that also addresses Pope Leo's first missive on AI. One of the comments in the interview with Yuval Harari that struck me was their comparison of real human immigration (and the efforts by many governments to limit such immigration) and the "immigration" allowed or even encouraged by often the same governments of AI Agents into the lives of people. Enjoy the read (and comment!) :
Wonderful resources, Frans! I'm especially looking forward to Levin's piece in the New Atlantis. I had a piece there a long time ago and enjoy their writing generally. I'll come back with comments and reactions shortly.
Thank you for this helpful focus on subsidiarity (in particular). As a Lutheran, I found the encyclical to be refreshingly expansive. The contrast of Babel vs. Nehemiah was a wonderful guiding metaphor. The inclusion of what makes human life human - with solidarity and justice - was important for me, too. It is our limitations that reveal our humanity and the need for working together with principles of justice. I am not sure that United Health's procedures are "well-meaning," nor the developments of Claude or ChatGPT, but I think your attention to rectifying that with auditable processes is clear. Thank you.
Thank you for this very thoughtful and enlightening analysis of the pope's missive. The issue is so well framed by both the pope and by the author. When a system loses touch with the individual, the anomalous and the specific, real people are really harmed. The English common law has stumbled around resolving this issue for about 500 years too. We ended up with a thing called "due process". That is the individual right to a fair hearing and ability to present your side of the issue. The law should value the individual by fashioning legal remedies which both meet the broad goal of the law AND are fundamentally fair to the parties involved. So, that's the demand side. It's well met by the author's examples.
On the supply side, we have a path through ethics, professional standards and licensing. The pope is starting the conversation which can develop the moral underpinning for an ethical framework containing professional programming and systems design. This is not my idea. I'm a lawyer. My husband is a programmer. We are both retired. As these algorithms were being developed, it became clear to him companies would not design in the best interest of humans because their competition would not.
But, as the pope points out, it is humans who make the decisions about what an algorithm or system is programmed to do. We have agency. Humans can refrain from doing things that hurt humans. The problem is profits, programmer error or the fear of getting fired can keep programmers from doing the right thing. My husband noticed that unlike his profession, lawyers can tell clients there are certain unethical things they will not do. Not just because it is wrong, but because they will lose their license. As a practical matter, this means the client can't (ignore the Trump disruption) just fire the lawyer and get what he wants because no other lawyer will do it either. If programmers had this kind of licensing which conditioned their ability to make money in their profession on ethical standards, it would go a very long way to preventing problems for real people.
So, first develop the ethical standards. See, e.g. the pope's missive. Start by identifying and articulating the dangers to real people. Then, workshop multiple, redundant brakes on the unmitigated power of impersonal algorithms, including licensing programmers and other software design professionals. Align the interests of the programmers and the people like the elderly patient in the story. Have standardized training for programmers and require the equivalent of passing the bar. Condition practicing programming on maintaining a license in good standing. Require continuing education, including ethics updates. You get the idea.
I'm going to ask the uncomfortable question. Was the algorithm wrong? If it was trained to predict whether care would reduce death, then it seems like the rehab didn't help Mr. Lokken to heal or return home. Perhaps hospice or home care was the better thing.
I love subsidiarity in all its forms, but it has to cover both autonomy and accountability. When the decisions are pushed down, but the funding is pushed up, the feedback system breaks. I feel this a lot in cities, who are more attentive to the state and federal bureaucracies that dispense grants than they are to their residents.
To what extent is the problem here that exit isn't possible because the structure of health insurance has locked people in (employer provided, government mandates on what it needs to cover so a low cost option isn't available, etc). I would imagine in a world where this bad PR would cause people to cancel their policies, the companies would be much quicker to fix them.
That’s a big part of the problem. The other more complicated part is how entwined government and private power are and how perverse and overlapping incentive structures create downstream rigidity that makes good decision making impossible. That’s the subject of next week’s post. But lack of competition and exit make all of it worse. There’s no incentive for reform. Or very few opportunities for market or any other kind of discipline.
"But they are structured so that no person you can reach is responsible for what is happening to you, and so that the cost of pushing back through the available channels is higher than the cost of giving up. And they’re also structured in a way where the interactions are so complex that finding the actual point where the decision went wrong is almost impossible."
You might find the recent book, _The Unaccountability Machine_ by Dan Davies of interest. His term for this phenomenon is "accountability sink."
Another great essay, you sure make us think! For those who have the time and inclination, two links below to articles/podcasts that touch upon similar topics you address, one an interview between Ezra Klein and Yuval Harari, and the second one an Essay by Yuval Levin in one of my favorite magazines The New Atlantic that also addresses Pope Leo's first missive on AI. One of the comments in the interview with Yuval Harari that struck me was their comparison of real human immigration (and the efforts by many governments to limit such immigration) and the "immigration" allowed or even encouraged by often the same governments of AI Agents into the lives of people. Enjoy the read (and comment!) :
https://www.nytimes.com/2026/05/26/opinion/ezra-klein-podcast-yuval-noah-harari.html?smid=em-share and https://www.thenewatlantis.com/publications/idols-of-the-valley .
Wonderful resources, Frans! I'm especially looking forward to Levin's piece in the New Atlantis. I had a piece there a long time ago and enjoy their writing generally. I'll come back with comments and reactions shortly.
Hopefully you can open the link, otherwise I can send pdf
Thank you for this helpful focus on subsidiarity (in particular). As a Lutheran, I found the encyclical to be refreshingly expansive. The contrast of Babel vs. Nehemiah was a wonderful guiding metaphor. The inclusion of what makes human life human - with solidarity and justice - was important for me, too. It is our limitations that reveal our humanity and the need for working together with principles of justice. I am not sure that United Health's procedures are "well-meaning," nor the developments of Claude or ChatGPT, but I think your attention to rectifying that with auditable processes is clear. Thank you.
Thank you for this very thoughtful and enlightening analysis of the pope's missive. The issue is so well framed by both the pope and by the author. When a system loses touch with the individual, the anomalous and the specific, real people are really harmed. The English common law has stumbled around resolving this issue for about 500 years too. We ended up with a thing called "due process". That is the individual right to a fair hearing and ability to present your side of the issue. The law should value the individual by fashioning legal remedies which both meet the broad goal of the law AND are fundamentally fair to the parties involved. So, that's the demand side. It's well met by the author's examples.
On the supply side, we have a path through ethics, professional standards and licensing. The pope is starting the conversation which can develop the moral underpinning for an ethical framework containing professional programming and systems design. This is not my idea. I'm a lawyer. My husband is a programmer. We are both retired. As these algorithms were being developed, it became clear to him companies would not design in the best interest of humans because their competition would not.
But, as the pope points out, it is humans who make the decisions about what an algorithm or system is programmed to do. We have agency. Humans can refrain from doing things that hurt humans. The problem is profits, programmer error or the fear of getting fired can keep programmers from doing the right thing. My husband noticed that unlike his profession, lawyers can tell clients there are certain unethical things they will not do. Not just because it is wrong, but because they will lose their license. As a practical matter, this means the client can't (ignore the Trump disruption) just fire the lawyer and get what he wants because no other lawyer will do it either. If programmers had this kind of licensing which conditioned their ability to make money in their profession on ethical standards, it would go a very long way to preventing problems for real people.
So, first develop the ethical standards. See, e.g. the pope's missive. Start by identifying and articulating the dangers to real people. Then, workshop multiple, redundant brakes on the unmitigated power of impersonal algorithms, including licensing programmers and other software design professionals. Align the interests of the programmers and the people like the elderly patient in the story. Have standardized training for programmers and require the equivalent of passing the bar. Condition practicing programming on maintaining a license in good standing. Require continuing education, including ethics updates. You get the idea.
I'm going to ask the uncomfortable question. Was the algorithm wrong? If it was trained to predict whether care would reduce death, then it seems like the rehab didn't help Mr. Lokken to heal or return home. Perhaps hospice or home care was the better thing.
I love subsidiarity in all its forms, but it has to cover both autonomy and accountability. When the decisions are pushed down, but the funding is pushed up, the feedback system breaks. I feel this a lot in cities, who are more attentive to the state and federal bureaucracies that dispense grants than they are to their residents.
Apologies! I just saw the footnote. I will leave my comment, because I think this is a good example of centralized systems breaking feedback pressure.
Of course it's united...