Quantifying AI Failures for Integration into Physical AI Reliability Models

Risk Acceptance

2.1.

A failure rate is always regarded in the context of consequences of the corresponding failures which constitute hazards. Functional safety therefore starts from risk and from the level of risk that is considered tolerable.

IEC 61508 follows this principle by deriving the required safety integrity from the required reduction of risk. It also makes an important distinction between random hardware failures and systematic failures. Software-related deficiencies generally belong to the latter category.

Highly automated systems introduce applications in which automation replaces a task that was previously carried out by a human or by a less automated technical system. This creates both absolute and comparative ways of discussing acceptable risk.

The GAME/GAMAB principle, Globalement Au Moins Aussi Bon, expresses a relative criterion: a new system should be at least as safe as an accepted reference system. This type of argument is particularly intuitive for automation replacing an existing human-operated solution.

The Minimum Endogenous Mortality (MEM) principle places an absolute limit on the contribution that a technical system may make to individual risk [6].

The two principles illustrate different questions. GAME asks whether automation is at least as good as what is already accepted. MEM asks whether the absolute contribution of the system remains sufficiently small.

These principles are useful for deciding whether the final system risk can be accepted, but become hard to decompose in practice to pinpoint the parts of the system which contribute most to the resulting risk. For a Physical AI architecture, these parts involve mechanics, electronics, sensing, conventional software, AI behavior, or a combination of these. A quantitative representation at this intermediate engineering level is therefore valuable even when the final risk acceptance is based on another criterion.

Quantitative Reliability in Functional Safety

2.2.

Several safety domains already use quantitative reliability as part of safety-integrity demonstration.

IEC 61508 provides the generic basis for electrical, electronic, and programmable electronic safety-related systems. IEC 62061 specializes in this approach for machinery. ISO 13849-1 provides another widely used machinery framework based on Performance Levels and uses quantities such as dangerous failure frequency, MTTF_D, diagnostic coverage, architecture category, and common-cause considerations.

Robotic systems use this machinery-safety foundation in combination with more specific standards. ISO 10218 [7] addresses industrial robots, ISO 3691-4 [8] addresses driverless industrial trucks and automated guided or autonomous mobile platforms, while ISO 13482 [9] addresses personal-care robots.

Road vehicles provide another clear example through ISO 26262. Hardware safety evaluation uses probabilistic metrics that make it possible to reason about very small residual failure contributions. Typical PMHF target values associated with higher Automotive Safety Integrity Levels are of the order of $10^{- 7}$ failures per hour for ASIL B and C and $10^{- 8}$ failures per hour for ASIL D, corresponding to approximately 100 FIT and 10 FIT.

The exact numbers and metrics differ between industries, but the engineering principle is similar. Failure contributions are quantified, architecture is taken into account, and the resulting system-level performance is compared with an integrity requirement.

This quantitative representation enables architecture-level reasoning. Series contributions can be added, redundancy can be evaluated, diagnostic measures can receive appropriate credit, and common-cause failures can be separated from independent failures. There are many research approaches where such architectures are practically utilized [10].

AI Safety and SOTIF

2.3.

AI-related failures do not fit naturally into the random-hardware-failure interpretation and are mostly regarded separately before the system-level argumentation is built up. A neural network may execute correctly from a computational perspective but still produce an unsuitable decision. A perception system may miss an object because the situation lies outside the effective coverage of its learned representation. A planning function may select an unsafe maneuver because an unusual interaction was not sufficiently represented in training or validation.

In road vehicles, ISO 21448 [11] addresses this broader class of functional insufficiencies through Safety of the Intended Functionality. ISO/PAS 8800 [5] further develops a safety framework specifically for AI-based automotive systems. These approaches are necessary because conventional functional safety alone does not cover all causes of unsafe behavior in AI-enabled systems. They emphasize identification of insufficiencies, data quality, verification, validation, robustness, and supporting safety evidence. At the same time, they do not provide the metrics which would allow AI components to be modelled for failures on the architectural level.

This matters beyond automotive applications. A Physical AI robot may contain motors whose dangerous failure contribution can be estimated, sensors with established reliability data, electronic controllers with known hardware failure rates, and an AI function whose residual reliability is represented only through validation evidence. Practical robotic systems already illustrate this heterogeneous architecture. For example, Yousif and El-Medany [12] present an autonomous patrol robot that combines sensing, navigation, obstacle avoidance, communication, and physical mobility, demonstrating how intelligent functionality is integrated with conventional electronic and mechanical subsystems. From the system reliability perspective, one part of the architecture is consequently described in a different mathematical language.

The objective of this work is to provide a bridge between these two representations.

Reliability of learning-enabled systems

2.4.

Several research directions already address quantitative reliability or failure assessment of learning-enabled systems.

Dong et al. [13] proposed a reliability assessment method for deep-reinforcement-learning-controlled robotic and autonomous systems. Their approach combines reachability verification with an operational profile so that verified local regions contribute to an overall reliability estimate. This work is important because it shows that reliability concepts can be meaningfully applied to DRL-controlled Physical AI systems. However, the authors integrate reliability over verified regions of the state space, whereas in our work the goal is to derive an operational failure intensity from observed behavioral failures and make that quantity directly usable in a conventional time-domain reliability model.

Software reliability growth provides another relevant direction. Merkel applied established software reliability growth models to autonomous-vehicle disengagement data [14]. Musa-Okumoto and related models were investigated as descriptions of the decreasing rate of disengagement events during development. This suggests that the evolution of autonomous-system behavioral failures can, at least in some cases, be represented using reliability-growth theory.

The relationship between observed failures and learning is also increasingly explicit. DeFIX identifies failure scenarios of autonomous-driving policies and uses reinforcement learning to generate improved policies for those scenarios [15]. The sequence of operation, failure identification, learning, and subsequent deployment is particularly relevant to the renewal interpretation developed in this paper.

Recent testing research likewise questions whether general AI performance indicators are sufficient for safety evaluation. Failure-based testing of DRL agents, for example, focuses explicitly on task-induced failures rather than assuming that accumulated reward alone adequately identifies unsafe behavior [16].

Reinforcement learning is also increasingly used to provide adaptive decision-making outside autonomous vehicles and robotics. For example, the DRL-based approach in [17] dynamically adapts task scheduling in edge-cloud environments to changing system conditions. Although this work does not address functional safety or reliability directly, it illustrates a broader shift from fixed software behavior toward learned decision policies that adapt to complex operating conditions. For Physical AI, this trend further motivates reliability models that can represent behavioral performance rather than relying only on a fixed catalogue of software failure causes.

Although current research results address different elements of the same general problem, still there is little formal ground which ties up the AI behavior to the top level architectural view of the final system task.

Research gap

2.5.

Once there is evidence about the behavior of the AI components, collected through field testing or simulation, there is a clear research gap when this evidence needs to be brought to the reliability level.

This paper proposes the chain and the specific framework for the fulfillment of the following logical flow steps: (1) operational evidence; (2) behavioral failure process; (3) failure frequency; (4) equivalent AI failure intensity; (5) reliability composition; (6) system risk; (7) risk acceptance indication. In principle, the framework is not necessarily AI-specific and could also be applied to other automated processes. Therefore, the framework addresses physical AI as an application domain, suggesting the integration concepts and AI renewal linkage.

Note that the risk acceptance indication is not final. The additional requirements, that known systematic insufficiencies not disappear inside an aggregate statistical rate, still stand. However, continuous monitoring highly benefits fast trigger events which can be based on the early risk acceptance indication for the timely recall or update reaction.

Physical AI System Model

3.1.

Consider a Physical AI system $S$ composed conceptually of electronic hardware and sensing $H$, mechanical and electromechanical components $M$, conventional control and software $C$, and one or more AI-enabled components $A$:

This decomposition is formed from the standpoint of the type of reliability evidence that is available for the corresponding elements. With respect to architecture, AI and conventional software may run on the same processor, while individual physical components can combine mechanical and electronic elements.

For the conventional components, established component or subsystem reliability models can normally be used. The framework introducted in this paper derives an equivalent representation for $A$.

Automation-Task Violation

3.2.

For an AI component $A_{i}$, consider the automation task allocated to that component. Depending on the system, this can include perception, navigation, object recognition, trajectory generation, manipulation, or another automated function.

In this paper, an AI behavioral failure is defined as an observed violation of an allocated automation task by an AI-enabled component or function under specified operating conditions.

This definition is behavioral and does not require knowledge of the internal software defect responsible for the observed outcome.

A safety-related subset can be defined as

where $\mathcal{F}_{A,s}$ contains those automation-task violations that are relevant to a safety requirement.

In early automated-driving systems, disengagements offered a convenient observable indication that automated operation could not continue as intended. However, disengagement is tied to the presence of a safety driver and therefore cannot serve as a general Physical AI failure definition.

A task violation can instead be detected by an independent safety monitor, a task-level invariant, a runtime supervisor, post-processing of recorded data, a human intervention, or another predefined failure detection mechanism.

The failure definition is part of the model. If this definition changes, the corresponding rate cannot automatically be compared with a rate estimated using a different event definition.

Operational Exposure and Counting Process

3.3.

Let $N(t)$ denote the number of qualifying automation-task violations observed during operational exposure $t$.

The mean cumulative number of failures is

and failure frequency is defined as

when this derivative exists [18].

In our proposed framework, exposure variable is set to operating time because it permits direct comparison with failures-per-hour metrics. Other exposure measures, including operating cycles or distance traveled, can be used where they better describe the relevant task.

The evidence contributing to $N(t)$ can come from field operation, fleet data, structured testing, or simulation. These data sources should not automatically be treated as equivalent. Their contribution depends on how well they represent the operational profile for which the reliability claim is made.

Behavioral-Failure Point Process

4.1.

Consider one stable AI configuration over an analyzed interval. The relevant configuration includes the trained model and any other parameters that can materially change its observable behavior.

Assume that automation-task violations follow a non-homogeneous Poisson process with intensity

and cumulative intensity

For this process,

and therefore

The Poisson assumption is important [20]. Although not every AI system necessarily exhibits this behavior, based on the previous research findings with disengagement events in automotive [14] as well with AI-based renewals [21] this model proves tractable allowing us to propose the reliability equivalence. This correlation would be more prominent in the upcoming end-to-end ML approaches, where automation task is tied more closely to the system level functional requirements, whereas with modular AI approaches a more elaborate decomposition might be required. In any case, the methodological framework here established always helps to assure early indication of the feasibility of the selected implementation approaches and architectures, where deep failure analysis is still unavailable.

AI Reliability Block

4.2.

An equivalent AI reliability block can now be defined.

Success of this block over exposure $t$ means that no qualifying automation-task violation occurs during that exposure. Its reliability is therefore

For a non-homogeneous Poisson process,

or, using the cumulative intensity definition,

The expression has the same form as the reliability function of a conventional non-repairable component with a time-dependent failure rate.

Equivalence Between Failure Frequency and Equivalent Failure Rate

4.3.

The failure rate of the equivalent AI block is defined as

Substituting (11),

which gives

At the same time,

and consequently

It should not be interpreted as a claim that failure frequency and hazard rate are universally identical quantities. For a general renewal process or arbitrary lifetime distribution this is not the case. The equality follows here from the chosen behavioral-event model and from defining reliability as the probability that no qualifying task violation occurs.

For the homogeneous special case,

the reliability becomes

and

which is the conventional constant-rate exponential model.

PFH and Equivalent FIT Representation

4.4.

If exposure is measured in hours, the resulting equivalent rate has units

and can be represented in equivalent FIT units as

For the safety-related subset of the behavioral failures,

can analogously be used.

The word equivalent is deliberately retained. A value of $10^{- 6}\, h^{- 1}$ assigned to an AI block does not imply a random physical deterioration process equivalent to an electronic component with the same rate. It means that, under the declared event definition and operational profile, the AI behavioral process produces the same zero-failure reliability function as the corresponding reliability block.

Estimation From Operational Evidence

4.5.

For an interval in which a constant behavioral failure intensity is considered adequate, let $n$ qualifying failures be observed during representative exposure $T$.

The estimated equivalent rate is

and the corresponding FIT representation is

This estimate is straightforward, but its uncertainty is important. A small number of observations can result in a wide confidence interval. In particular, observing zero failures does not establish a zero failure rate.

The approach should therefore not be interpreted as a way to demonstrate extremely small safety targets purely by accumulating test hours. Statistical evidence is one part of the safety argument and needs to be combined with verification, architectural measures, and other available evidence.

Common Reliability Representation

5.1.

The main practical reason for introducing the equivalent AI rate is that it allows the AI function to participate in the same reliability model as the physical system [22].

Consider a simplified Physical AI function containing a mechanical subsystem $M$, electronic hardware $H$, and an AI component $A$. For independent series blocks,

If $M$ and $H$ are represented by constant failure rates and the AI rate is time dependent,

For a constant AI rate during the considered interval,

The benefit of (27) is mainly architectural. It allows the relative contribution of the AI function to be considered beside sensing, electronics, mechanics, or other conventional elements.

Redundancy and Diverse AI Channels

5.2.

Reliability engineering becomes particularly useful when architectural measures are introduced.

For two AI channels $A_{1}$ and $A_{2}$ performing the same function, full statistical independence would give

This can describe architectures containing diverse perception systems, an AI function and an independent AI monitor, or two independent implementations of the same automation function.

However, behavioral independence is much more difficult to justify than implementation diversity.

Two or more AI systems can use different implementations and still fail under the same situation because they share training data, sensors, environmental assumptions, task specifications, model families, or similar learned representations. Especially in the presence of unknown unknowns, as required in [10], it is possible that the reliability calculation breaks under new findings, where failures might accumulate and risk dramatically increase instantaneously.

The ${\beta }_{AI}$ Factor and Edge Cases

5.3.

Classical redundant-system analysis has a similar problem when multiple components are vulnerable to a common cause. The $\beta$-factor model provides one way of separating independent and common-cause contributions [19]. In the systems where AI is heavily utilized, edge cases lie at the edges of the behavioral failure observation distribution, potentially leading to high severity hazards. Therefore, they also need to be adequately modeled.

We use classical concept as an inspiration for a related concept which allows us to quickly update the reliability estimation when new edge cases are discovered. As shown inFig. 1, failure rates of the AI components are adjusted and new edge cases are regarded separately as long as the update of the implementation is not performed and statistical ground truth established for the complete automation task as before the edge case discovery. Behavioral failures in the presence of edge cases are immediately extracted from the average AI failure rate approximation upon discovery, and treated separately as long as sufficient ground is established to prove that the behavioral failures under edge cases can be placed under the same (yet updated) overall distribution.

For each AI channel with equivalent behavioral failure intensity $\lambda_{An}$, define

And for each newly discovered edge case

Where $K$ is the number of still unsolved edge cases, and also

Assuming both AI components have the same failure rates, the reliability of the independent parallel contribution is

while in case of a single new edge case found

The resulting reliability for this example is

The analogy to the classical common-cause $\beta$ factor allows the classical calculus to be performed when AI components are considered. It is important, however, to be able to correctly estimate $\lambda_{EC}$ based on the newly discovered data describing the edge case, and also to determine $\beta_{AI}$ based on the proof of exposure to the edge case. In the presence of newly discovered edge case, both $\beta_{AI}$ and $\lambda_{EC}$ shall be set conservatively until additional ground truth proof is obtained. This allows the most accurate final reliability and safety assessment of the existing configuration and facilitates the decision on the system recall and the timeline (urgency) of the following update.

Behavioral Renewal Epochs

6.1.

The AI-based functionality differs from many conventional components because its behavior can change materially during the system lifetime.

Retraining, reinforcement learning, calibration changes, software updates, and OTA deployment can produce a new behavioral configuration without replacing the physical hardware.

Let

denote the release times of behaviorally relevant AI versions.

Fig. 1. Reliability block diagram for diverse configuration of AI components where βAI is used to account for the appearance of newly discovered edge cases

One behavioral epoch can be defined as:

Within this interval, the corresponding AI configuration is assumed to be stable enough for its failure process to be characterized. When a new release materially changes the behavior, a new epoch starts.

This idea follows the renewal interpretation previously used for software upgrades and AI-based reconfiguration. In the earlier formulation, software upgrade or self-reconfiguration is treated as repair-like behavior in the context of software reliability growth, including changes triggered by reinforcement learning [23][24]. Periodic upgrades were also distinguished from instantaneous updates, with component behavior remaining unchanged between successive upgrades.

The refinement introduced here is to separate the two processes clearly. Automation-task violations form the behavioral counting process inside an epoch, while AI releases define the renewal process between epochs.

If the inter-update interval is represented by

then a classical renewal model can be used when the appropriate assumptions hold. In practical deployed AI systems, update timing will often depend on accumulated evidence and engineering decisions, so a generalized renewal interpretation can be more appropriate.

Reinforcement Learning and Reliability Growth

6.2.

The renewal view is particularly suitable for AI systems that improve using operational evidence. A simplified lifecycle can be represented as the sequence of steps: (1) operation; (2) violation detection; (3) data acquisition; (4) learning/retraining; (5) verification; (6) release and (7) operation. This cycle has practical analogues in continual-learning research. DeFIX [15], for example, identifies failure scenarios in autonomous driving and uses them to train additional reinforcement-learning policies.

The equivalent failure intensity then becomes version-dependent:

A successful improvement process would normally aim for

but this relation is not assumed by the framework. An update can introduce a regression, change the effective operational domain, or expose a previously irrelevant failure mode. The new rate therefore needs to be supported by new evidence.

This provides a simple way of connecting reliability growth with continuously evolving Physical AI.

From Reliability to Risk Acceptance

6.3.

The complete quantitative chain can now be written as

The last step remains application dependent.

A GAME-type argument can compare the resulting system behavior with an accepted reference. A MEM-type argument can instead constrain the absolute contribution to risk.

The important point is that risk acceptance does not need to rely only on final accident statistics. Intermediate quantitative arguments become possible. If system reliability is dominated by the AI function, additional AI improvement or architectural diversity may be appropriate. If the AI contribution becomes small compared with actuator or hardware failures, further AI improvement may have limited influence on the final system reliability.

Systematic Failures vs AI “randomness”

8.1.

The most obvious question challenging the proposed approach is that AI is implemented in software, while software failures are systematic. This criticism is valid if the equivalent failure intensity is interpreted as the failure rate of the underlying software defect. Although software deficiency can be perfectly deterministic and given the same inputs and state, it may reproduce exactly the same failure, what varies during operation is the exposure of that deficiency to environmental and system conditions.

The stochastic element modeled in this paper is therefore the occurrence of the behavioral manifestation, what is an important distinction. It allows operational behavior to be treated statistically without contradicting the systematic-failure classification used in functional safety.

Assumption on the Failure Process

8.2.

The equality in (16) is derived under an NHPP assumption, what is used as a tractable model for reliability estimation, establishing a reactive framework to ever-changing ground truth in AI verification. In general, this model cannot be assumed for every AI system. Behavioral failures may cluster. They may depend on operating mode or environmental condition. One failure may influence the likelihood of another. Different operational regimes may also exhibit very different failure intensities. In such cases, a more suitable point-process model should be used.

The broader concept where an equivalent AI reliability block can be defined using the probability of zero task violations remains applicable. However, the direct equality between failure frequency and equivalent hazard must then be re-established for the selected statistical model.

Dependence on the Operational Profile

8.3.

An equivalent AI rate cannot be separated from the operational conditions under which it was measured.

A navigation model operating in an empty warehouse does not necessarily have the same behavioral failure intensity when the same warehouse contains people, moving forklifts, temporary obstacles, or poor illumination. Every reliability claim should consequently identify the relevant automation task, AI version, operational domain, event definition, and source of exposure evidence. This also limits how simulation evidence can be used. Large, simulated exposure is useful only to the extent that the simulated conditions represent the behavior relevant to the safety claim.

Very Low Failure Rates

8.4.

Safety-integrity targets can involve failure frequencies that are too small to establish directly through realistic operational testing. The proposed equivalent failure-rate formulation does not remove this difficulty. For example, observing no qualifying failure in a large but finite number of hours produces an upper statistical confidence bound; it does not prove that the true rate is zero or that an extremely small safety target has been reached. This is why the proposed measure should remain part of a larger safety argument. Formal verification, scenario-based evidence, structural safety mechanisms, monitors, diagnostic coverage, safety envelopes, and process assurance remain important.

Selection of the ${\beta }_{AI}$ factor

8.5.

${\beta }_{AI}$ is conceptually related to common-cause reliability modeling but estimating it will be more difficult than applying an established hardware beta-factor model.

AI channels can share several sources of dependence: training data, sensor inputs, requirements, architectures, reward functions, feature representations, model families, or environment assumptions. Even nominally diverse implementations can therefore exhibit significant co-failure behavior, but not necessarily completely the same when the new edge cases are discovered, so these aspects require additional research. At this moment, the parameter should be treated conservatively and be used as a vessel to prevent unjustified redundancy credit and allow visibility of immediate new risk and facilitate the recall decision.

Integration with Safety Case

8.6.

The statistical and systematic parts of AI safety should not be separated completely. An unidentified task violation contributes evidence to the residual behavioral process. Once the underlying situation has been identified as a repeatable safety-relevant insufficiency, its treatment should change. It becomes a known issue that requires explicit mitigation or limitation. Therefore, combination of functional safety approaches (AI “caging”) remains together with the statistical data-driven AI performance indicators which may trigger automation task violation and induce failures.

The proposed framework is also intended to serve both general reliability estimation with respect to system functional requirements, as well as for safety requirements where the same metrics are used for safety-related failure paths. In this sense, safety-related behavioral failure needs to be regarded as one leading to the dangerous failure, what allows the reduction of failure rates in contrast with the complete automation task. This is common in safety engineering, where hardware failure rates in reliability block diagrams which are used account only for dangerous undetectable failures, in cases where those diagrams explain safety related systems.