Most AI initiatives that arrive on a COO’s desk are not business cases. They are proof-of-concepts dressed as business cases. The distance between capturing data and using it to drive action is not shortened by a better model, but by the right design of the operational infrastructure. Here are five questions that can help distinguish between a business case and an expensive experiment.

What bottleneck will it help to remove?

Your first question to the vendor or to your team that is presenting the initiative should be focused on impact. Specifically, ask what metric this is going to improve. Is it a cycle time? Error rate? Cost per order? Days from order to delivery? If you cannot tie the proposed initiative to a metric that will go up or down as a result of implementing the solution, you are not dealing with a business case but a feel-good exercise dressed as one.

That is not because executives are intentionally misleading you. This is a genuinely tricky question to answer honestly because, as philosophers like to point out, “digital transformation” as an objective is inherently unfalsifiable. It is perfectly possible to throw tens of millions of dollars at an initiative with no demonstrable business impact and convince yourself and others that it achieved transformation. However, a bottleneck is a falsifiable statement. It either moves or it does not. Drill down on what precise process, what particular team, and what exact source of delay is supposed to be impacted by the initiative. Vague answers such as “It will impact the digital workers’ ability to do their jobs in a certain way” are a red flag.

Is the data actually there?

AI initiatives that fail most often do so because of data-related reasons. The data was not there, it was not timely, it was of poor quality, or it existed in a system that nobody was willing to pay to extract. Poor data quality and availability kill most AI projects long before model selection even comes into play. Ask the vendor about what systems feed into the model, how recent the data is when it comes into the system, and who is in charge of its quality. Legacy systems are a particular weak point that vendors often try to hide because integrating new systems with old is rarely as appealing as selling an AI initiative on glossy slides.

Does it actually interface with the decision-making process?

This is a business question, not a technical one, but it is massively important. Ask the vendor or your team how the output from the model will actually be used within the company. Most businesses have natural decision-making cadences, even if they are not written down anywhere. Some have daily stand-ups or weekly operations meetings where the relevant decisions are made. The faster the insight from the model can get to the person who can actually act on it, the more business value the initiative will create.

Put another way, the shorter the distance between the output of the model and the actual business decision, the more valuable the insight from the model is. Once you have established the timing, ask about the owner of that decision. This is the crux of the matter, and it is why most AI initiatives at enterprises fail, even with good data and well-chosen models. Once the decision cadence and its owner are agreed upon, the evaluation criteria for an ai solution for coo shift from purely technical to business-centric.

If you want to identify a good vendor proposal, look for indications that the product actually understands the cadence of decision-making in its target environment. A truly enterprise-grade AI application will have operational decision-making rhythm baked into its design. It is the difference between an executive dashboard and a fully integrated system.

Who will own the adoption, and what will they do if nobody wants it?

Adoption is not a sexy topic, but it is one of the most important ones for an executive considering an enterprise AI initiative. This is the only way to ensure that your employees are actually going to use the new technology, and that you are not going to waste money on an initiative that will languish unused in the enterprise equivalent of a dusty warehouse.

Adoption requires training, which in turn requires a substantial investment of time and human resources. It might also require changes to your business processes that are disruptive in the short term but necessary for long-term success. In a survey published in 2019, the Sloan Management Review and Boston Consulting asked 2,500 executives about their companies’ AI experience. According to the respondents, 90% of the enterprises had already implemented AI somewhere in their operations.

Only 10% had seen any business benefit from it. The magazine attributed the discrepancy to “inadequate cultural and change-management strategies.” In other words, even if your company has an AI solution that works as designed, there is a very good chance it will not deliver on its promises if your employees are not genuinely interested in adopting it. Ask the vendor about who is going to make sure that your frontline employees are going to want to use their product and how they are going to handle the disruption to your business processes in the first 90 days of adoption.

What’s the governance policy for when the model is wrong?

Every model is wrong about something, so this is not an academic question. Ask the vendor about their governance policy for when the model’s recommendation is erroneous. Ideally, it should have clearly defined procedures about who can override the recommendation, what the chain of escalation is, and who takes responsibility for a recommendation that was followed.

This is where total-cost-of-ownership considerations come into play for most enterprise initiatives. A licensing fee that gets picked up on the P&L is only a small part of the story. Most of the costs related to AI initiatives come from the data engineering, model deployment, and maintenance, none of which should be hidden from the executive reviewing the initiative. If a vendor cannot provide a detailed breakdown, it may be a sign that they have trouble actually supporting the solution.

Ask for a pilot project. Ideally, it should be focused on a single line of business or a single region with clearly defined success criteria. Enterprise-wide sourcing of AI solutions only makes sense once you have proof that the solution works in a representative but limited environment.

The real filter

None of these questions relates to how good the model is. They all focus on how well the company can use the model’s output to actually improve business results. Ask them before you commit to a trial project.