AI Gives Us a Process, Not a Result

Modern AI systems, especially large language models and diffusion-based image generators, do not deliver fixed outcomes. They provide a generative process. This distinction is not semantic nitpicking; it is fundamental to how these systems work, why they behave the way they do, and how we should use (and evaluate) them.

Media coverage tends to float around the latest model releases, architectures, and benchmarks, while everyday applied effects remain slightly mysterious. Issues are perceived as if soon to be resolved by some ultimate innovation, whether scale, cost, or a new architecture, once and for all.

With the proliferation of agents the tone has shifted toward more practically studied programmatic control of models. Yet two things are often overlooked. First, big commercial model outputs are already guided by expert systems or agents of sorts, while staying completely opaque to the client. Second, the whole process may in fact add more uncertainty, not less, to dependent systems.

AI models simply add or replace layers of uncertainty with other layers, while leaving the fundamental problems unchanged. Those who navigate this view effectively will survive the current wave and those to come.

We must therefore shift the discussion from the qualities of AI models themselves to the handling of AI models as uncertainty generators per se. In this article we will use only the basic tools of probability theory, reusing the logic framework from and mapping some notions described in Statistical Consequences of Fat Tails by N. N. Taleb [arXiv:2001.10488] to practical AI applications.

The clearest way to see this is through a concept from probability theory: the difference between a probability kernel, its expectation and the payoff.

The Probability Kernel vs the Sample vs the Expectation vs the Payoff

The primordial ancestor of modern LLMs is a simple Markov chain bot. When trained on highly structured text, like poetry, it can often produce new coherent sentences. The concept dates to the early 20th century (Andrey Markov, 1906), but modern LLMs can still be represented as an abstract Markov chain (Zekri et al., 2024, [arXiv:2410.02724]). This lets us apply probabilistic reasoning from first principles, identifying fundamental properties while ignoring varying implementation details.

Where to start?

A probability kernel is, in essence, the complete set of possibilities inside the AI. For any given prompt, it specifies every answer the model could give, along with how likely each one is — covering the common responses, the rare outliers, and the subtle connections between different parts of the text. Full set of possibilities is not something we can write out for modern AI models, but it is something we can reason about from the first probability principles.

Putting it simply, the process inside the LLM that outputs some text in answer to our input is our kernel, and this kernel, for each input text, can produce a range of outputs. But, as we look at the chatbot output or agent window, we see only single output. This is our single realization, a sample.

Our bot or agent just produced a sample output. This is a drastic reduction of information because, apart from very specific conditions where we seek deterministic outputs, model output relies on a random process which, among other things, depends on the temperature parameter. Even at zero temperature the output distribution is implied; we are simply sampling it deterministically, ignoring everything else, including, most likely, the best and worst possible outputs altogether. The random nature of outputs is common knowledge, yet we overwhelmingly rely on this single answer. How many attempts do we make to see the range of possible outputs? How many possibilities are there even? How many samples are enough to assess this landscape? What is the best, and even more importantly, the worst case?

The more extensive sampling, however, is indeed happening during model alignment. After initial training, LLM developers typically run a reinforcement learning phase using a Reward Model that acts as a judge. When the LLM outputs a sequence of text in response to a prompt, the Reward Model assigns it a numeric score. This process can be repeated many times. Because this score is a number (e.g., +1.5 for a helpful answer, −2.0 for a toxic one), developers can compute the average score across many tries. Typically, the training algorithm then nudges the LLM to improve that average.

Multiple questions arise here. What was the actual distribution of scores? Does the range of the score reflect any meaningful economic or risk outcome? Can it possibly represent anything more than a bell curve at all?

At the very end, the kernel answers: "What is the full landscape of possibilities?"
The expectation answers: "If I had to pick one answer that maximizes reward on average, what would it be?"

These are not interchangeable.

Finally, The Payoff

Now, consider one of those unforgettable nights in Georgia (the country) with Georgian wine. Several months later, back home, you visit a local specialty import store and buy the same wine, yet the taste is different. It may even feel worse than a random bottle you grabbed at the grocery by the design of its label.

The payoff function in those two situations is not the same. One cannot recreate the Georgian-night experience in a living room in Miami; the context (place, company, atmosphere) is missing. Meanwhile, the grocery store likely sells nearly identical generic wine under hundreds of different labels. While you can be impressed by a label, the overall possible-experience curve is much less convex. The same bottle can follow an entirely different payoff route depending on whether your payoff function still expects a Georgian night or simply a Tuesday dinner.

This is how a typical reinforcement-learning reward compares to real-world payoff. The reward is a score you can optimize, but it is rarely the thing you actually want. Worse, by adopting the wrong payoff function you may be misled into playing someone else's game.

Even the expectation under the reward model is not what ultimately matters once the output leaves the lab and meets the world. When an LLM response is actually used (read by a human, executed by an agent, fed into a trading system, or turned into medical, legal, or operational advice) there is a real payoff function. This payoff maps the concrete generated text to its actual consequence in a specific external context: world state, downstream systems, timing, human interpretation, irreversibility, cascading effects, and so on. The payoff can be highly nonlinear, asymmetric, path-dependent, and dominated by tail events.

The reward model is only a compressed, noisy proxy, typically trained on human preference data or safety classifiers. It reflects average judgments of "helpfulness and harmlessness," not the true economic, safety, survival, or reputational payoff of deploying that specific output.

We can categorize payoff functions into several classes: linear, convex, concave, or combinations of the these.

• Linear: Functions, usually defined on a closed interval, such as a similarity metric, a simple exam score, classification accuracy, or F1 score (0 to 1 score often used in AI). A small change in input produces a proportionally small change in outcome. These are the domains where optimizing expected reward works best.
• Convex: Functions where gains accelerate as you move away from some starting point. Think of venture capital, drug discovery, or finding a novel exploit: most attempts return little or nothing, but a single rare outlier can pay for everything. In convex domains you want variance, and you want to sample widely, because the payoff tails will eventually produce the sought outlier value.
• Concave: Functions where incremental gains diminish as you move away from the center, and where the left tail can contain ruin. Collision-avoidance logic in autonomous driving is a canonical case: once the probability of a fatal crash drops below some threshold, further gains matter far less than guaranteeing that the ruinous tail is never sampled. In such domains the correct response is not to "shrink variance" but to remove exposure to the ruinous tail altogether. You do not play a game whose worst case can end the game.
• Mixed / S-shaped: Functions that combine convex and concave regions (e.g., sigmoid). Below a threshold the payoff is convex (variance-seeking); above it the payoff turns concave (certainty-seeking) or hits a ruin barrier. Most medical interventions follow this pattern: a drug may show convex returns at low doses (small increases in efficacy yield large survival gains) and concave or ruinous returns at high doses (toxicity, side-effects, or death). Antibiotic stewardship, chemotherapy dosing, and vaccine risk–benefit calculations are further examples. In these domains you must know exactly where the inflection and absorbing barriers lie; optimizing a single scalar reward almost always misprices one of the regimes.

In principle, modern neural networks can approximate a wide range of functions. Their building blocks, neurons activated by ReLU function and its cousins, when stacked in sufficient depth and width, become universal approximators: they can model linear, convex, concave, and arbitrarily complex combinations. This flexibility is why a single model architecture can serve such diverse tasks. The catch is that the training or learning objective determines which class of outcomes the model actually encodes. A reward model trained on average human preferences implicitly assumes a roughly linear payoff landscape. If your real-world payoff is convex or concave, you are playing a different game.

The payoff answers: "Given this output and this world, what do I actually win, lose, or risk, including the irreversible or ruinous consequences?"

In thin-tailed, ergodic, repeated, linear-utility settings, the expectation and the long-run experienced payoff may converge. In the fat-tailed, non-ergodic, skin-in-the-game domains where many high-stakes LLM uses live (agentic workflows, autonomous decision systems, high-leverage advice), they diverge sharply. A single low-probability sample from the tail can produce a payoff that swamps years of "average" performance, or simply ends the process.

Optimizing the kernel to maximize expected proxy reward therefore does not guarantee good real-world payoffs. It can even hide or amplify tail risks if the reward model smooths over, or fails to penalize sufficiently, low-probability catastrophic outputs or even proper real world nonlinearities. Probability, the kernel, is a tool in the chain for decision-making, not an end in itself; the payoff is what closes the loop with reality.

Treating the kernel output as "the result" rather than a step from a process leads to systematic misunderstandings. This equally applies to any models, AI or not:

• Variability is not a bug. The same prompt producing different outputs on different runs is exactly what a non-degenerate kernel predicts. Temperature, top-p sampling, and random seeds are all ways of controlling how we draw from that kernel.
• Single outputs hide uncertainty. A model may assign 60 % probability to one answer and spread the remaining mass across several alternatives. A single sample gives you no visibility into this. Techniques like self-consistency (sampling multiple reasoning chains and taking a majority vote) or best-of-N sampling are partially addressing the problem: the best answer the model can offer is not neccesarily correct too.
• Hallucinations are not lies; they are samples. When a model produces confident-sounding but false information, it is usually because that content has non-negligible probability under its learned distribution. The kernel is imperfect, trained on noisy, incomplete data, so the distribution it defines sometimes places mass on falsehoods. This is a statement about the quality of the kernel, not about the model "deciding" to deceive.
• Evaluation by single or selected groups of examples is misleading. Benchmarks systematically underestimate or misrepresent what a model can do and fail to measure its practical value in field conditions.
• Prompting and fine-tuning reshape distributions, not single points. A well-crafted prompt or a round of preference tuning (RLHF, DPO, etc.) does not change one output in isolation. It reweights probability mass across the entire space of possible generations. This is why the same model can be made dramatically more or less likely to produce certain behaviors without ever seeing those exact behaviors during the update.

Practical Implications

Thinking in terms of kernels rather than results changes how one should interact with these systems:

• For exploration and creativity, sample repeatedly. The variance is the value. One image or one story is rarely enough to understand what the model has captured.
• For high-stakes factual or reasoning tasks, treat any single output as a hypothesis to be verified. Use multiple samples, external tools, retrieval, or formal verification. Ask the model to surface uncertainty or enumerate alternatives explicitly. We also developed a range of more rigorous tools which evaluate meta uncertainty of model answers.
• For agentic or decision-making workflows, expose more of the distribution when possible (log probabilities, multiple candidate actions, calibrated confidence). A policy that only ever shows you its single highest-probability action is hiding information you may need for robust decision-making. Explicitly define scenarios with absorbing barrier (when agent should stop or be eliminated from decision tree).

The most powerful uses of AI should involve composing the generative process with other processes: retrieval augments the conditioning information, code execution filters or transforms samples, human feedback or automated verifiers reshape the distribution through further training or inference-time guidance. Uncertain parts of the process should be replaced by programmatic agents (e.g. state machines) where explicit outcomes are explored and settled.

Conclusion

AI systems built on large-scale generative modeling give us something more powerful and more subtle than oracles that output results. They give us approximate conditional distributions, probability kernels, learned from data. These kernels can be sampled, steered, composed, and (imperfectly) aligned with human preferences. They support simulation, creation, and probabilistic reasoning at a scale that was previously impossible.

But they do not, and cannot, give us singular, guaranteed-correct results on demand. Expecting them to do so is like demanding that a single roll of a loaded die always show the face whose long-run average matches the expectation. The power lies in the process, the ability to draw many times, to inspect the distribution, to reshape it, and to combine it with other sources of structure.

Those who use AI most effectively will be those who stop asking "What is the answer?" and start asking "What does the distribution over answers look like, and how can I usefully interact with it?"

This is not a limitation to be engineered away. It is the central capability that makes contemporary AI interesting.

References

1. Markov, A. A. (1906). “Extension of the Law of Large Numbers to Dependent Quantities.” Izvestiia Fiziko-matematicheskogo obshchestva pri Imperatorskom Kazanskom universitete, 15(4), 135–156. English translation in Howard, R. A. (ed.), Dynamic Probabilistic Systems, Vol. 1, Wiley, 1971. [Link]
2. Zekri, O. et al. (2024). “Language Models as Abstract Markov Chains: A Probabilistic Analysis.” arXiv:2410.02724. [arXiv]
3. Taleb, N. N. (2020). “On the Statistical Consequences of Fat Tails.” arXiv:2001.10488. Also available as Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applications, STEM Academic Press, 2020. [arXiv]
4. Ouyang, L. et al. (2022). “Training Language Models to Follow Instructions with Human Feedback.” arXiv:2203.02155. [arXiv]
5. Rafailov, R. et al. (2023). “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” arXiv:2305.18290. [arXiv]