Assessing a pilot built to survive month three
Most automation pilots clear their demonstration and quietly fail in production. The difference usually comes down to whether anyone agreed, beforehand, what the agent is allowed to get wrong.
Nikolai Varga
Pilots seldom fail for lack of capability. They fail because nobody agreed beforehand what the agent was permitted to get wrong. The first visible error then turns into a verdict on the whole engagement.
Three things to agree before commissioning
First, the floor itself: the confidence score below which the agent is not permitted to act. Second, the escalation route: precisely which human queue the sub-floor work lands in, and who is responsible for it. Third, the review cadence: how often the exception log is examined jointly, and whether the floor is revisited.
None of it is a technical question. All of it decides whether a pilot becomes infrastructure or quietly falls into disuse while still drawing cost every month.