I have to admit that I am still working out how to be efficiently semi-retired. I am reluctant to call it full retirement since I still teach college classes once or twice a semester, and every now and then a company wants to talk about its positioning in the education space. What is different from being a full-time employee is that I used to have long-term strategic goals running alongside the tactical work. These days most of what I do is tactical, and Artificial Intelligence, which remains an area of great interest to me, is moving faster than I can react to it. I doubt I am unique in having that latter problem.
My early experiments building an AI Board of Advisors went down some interesting paths, and I will report back on a few of them in the coming months. For now I want to take a swing at something that keeps showing up in my conversations about AI, whether with students, former colleagues, or whomever is willing to talk to me about it (and I am always glad to expand that list).
The Same Mistake, Twice
In the Introduction to Technology course I teach, there is a module on technology projects and the project management around them. One fact always intrigues me, and I bring it up in each class: a majority of technology projects fail to accomplish significant portions of what they set out to do, whether that means going over budget, running late, or simply not working the way it was supposed to. The exact numbers get argued about among the people who study this for a living, but the general shape of the finding has held for years. The question I put to my students is why. Everyone starting a project knows this kind of a result happens before they start. Why keep making the same mistake?
Recognizing that you are making the same mistake twice it turns out is hard to do. But that only explains part of it.
The best answer I have come up with is that the human mind can only hold a limited number of tasks and issues at once, and serious projects require conceptualizing more than that. It is a bit like overrunning the capacity of a context window1 before context windows existed.
Two Kinds of Capability
The comparable problem I see in AI projects is that we are combining two radically different kinds of capability, and we have almost no experience doing that. Worse, we may not even notice it is happening.
Let me define two terms the way I am using them. At a conceptual level, we are combining what I will call deterministic nodes of activity with what I will call probabilistic nodes. By deterministic I mean we know what the activity is supposed to produce and what it is capable of producing: 2 + 2 = 4, always. By probabilistic I mean we are not entirely sure what the activity is supposed to produce, and we may not fully know what it is capable of producing: sometimes 2 + 2 = 5. For probabilistic activities, that variability is the whole point.
A quick aside on terms. Much of the current writing on AI engineering calls this “non-deterministic” rather than “probabilistic.” I am sticking with probabilistic on purpose. Non-deterministic describes what the behavior is not. Probabilistic describes what it is, and that is the reason the thing is useful in the first place.
If someone bought an accounting system, I doubt they would be happy if it produced a different answer every time it ran. If the CEO of the accounting software company went on record saying that not only is that true, but they are not entirely sure how the software arrived at its answer, I suspect the buyer would look for a different vendor.2 And yet that happens on a fairly regular basis with AI providers, and mostly nobody blinks.
When the Two Collide
Here is what that collision looks like when it is not hypothetical. In 2022, a man named Jake Moffatt asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The chatbot told him he could book a full fare ticket and apply for a partial refund within 90 days. That is not what Air Canada’s actual policy says. The real bereavement policy requires the discount be approved before travel, not after.
Moffatt booked the flight, flew to the funeral, and applied for the refund the chatbot had promised him. Air Canada refused, then argued in front of Canada’s Civil Resolution Tribunal3 that the chatbot was, in effect, a separate legal entity, and that the airline could not be held responsible for what it said. The tribunal was not persuaded. It ruled that a chatbot is just another part of a company’s website, and Air Canada had to honor the fare its own probabilistic node had invented.
That is the deterministic and probabilistic problem in miniature. Air Canada’s refund policy is a deterministic system. It is supposed to produce the same answer every time, and it is written down. The chatbot sitting in front of that policy is a probabilistic node, and nobody had built anything to reconcile what the chatbot said against what the policy actually allowed. The company let the probabilistic node write checks the deterministic system had to cash.
Bolted On, Not Built In
The closest design analogy I have come up with from my years in systems architecture is cybersecurity. It used to be hard to define in advance every way “bad guys” could attack a system, and for that reason and budget reasons, figuring out how the overall design should account for those attacks often ended up as the last step taken. Security was bolted on, and bolted-on security tends to be weak security. It took years of hard experience before design approaches evolved enough to build security into the main development path instead of tacking it on at the end. There is a whole movement around this now, usually called “shift-left” security or “secure by design.” I suspect AI development is going to follow the same meandering path, possibly the same number of years behind.
It has occurred to me that the closest thing we had to a probabilistic node before AI came along was a human being sitting inside the architecture. We have never been great at getting people to do exactly what we want, or at building guardrails that reliably steer them when we want to. And do not get me started about the difficulties in work performance annual reviws. I do not expect it to be any easier with AI, and for some of the same reasons.
I expect to return to the issue of guardrails and the possible overlap between how to manage people and how to manage an AI node. Until then, I would like to hear from anyone with a similar or different take, and especially from anyone who can point me to studies or examples, on either side of this, that would be useful to read.
Notes
1 A context window is the amount of text an AI model can hold and work with at one time in a conversation. Once a conversation grows past that limit, earlier material starts to drop out of the model’s view, the same way a person can only keep so many things in mind at once.
2 Anthropic CEO Dario Amodei wrote in 2025: “People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work.” See “The Urgency of Interpretability”, darioamodei.com, April 2025. Link: https://www.darioamodei.com/post/the-urgency-of-interpretability
3 Moffatt v. Air Canada, 2024 BCCRT 149, the official Civil Resolution Tribunal decision, published on CanLII. Link: https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html