How to prioritize AI initiatives and measure a pilot

1. Describe the problem before choosing a tool

A useful project begins with the work that needs to improve. Ask the people doing it where information gets stuck, which steps repeat and which decisions take longer than they should. “We need AI” is too broad to give a builder or an executive a clear target.

Eddie Irvin describes mapping the assembly line of a business process: what happens first, where the information comes from and who signs off. Some problems need better information flow or ordinary automation before they need an AI model. Write down the process so you can choose the right intervention.

• Name the recurring problem and the people affected.
• Identify the process owner and the decision or output that matters.
• Map inputs, handoffs, exceptions and approval points.

From the interviews: Eddie Irvin · 12:28

2. Look for useful leverage and a manageable first test

Time saved has different consequences in different roles. Eddie asks whose time a project would save and what that person could do with the capacity. Removing a bottleneck for a leader or a repetitive burden for a team can be more useful than creating a novel demo.

Compare candidate projects using business value, process clarity, access to an expert and the effort required to run a responsible test. These comparison questions are our synthesis of the interviews, rather than a scoring system presented by a guest. A project that sounds valuable but lacks an owner or usable information needs more preparation.

• What changes for customers or the business if this works?
• Can one team test a clearly defined workflow?
• Can an experienced person judge whether the output is useful?
• Are the needed information, permissions and budget available?

From the interviews: Eddie Irvin · 24:00; Ronnie Kwesi Coleman · 7:35

3. Involve an expert and agree the expansion path

Ronnie Coleman argues that context and judgment often live in experienced people rather than a clean rule book. Include an expert who can explain exceptions and review results. A pilot designed without that input may produce convincing work that does not fit the business.

Keep the first test focused on one problem and one set of experts. Also identify who can approve continued spending and how a successful result could fit the wider organization. Ronnie describes the failure mode where a small experiment works but stalls because no one planned funding or expansion.

From the interviews: Ronnie Kwesi Coleman · 7:35

4. Agree what success means and record the starting point

Use a business outcome that the owner can explain. Ronnie discusses growth, operating cost and risk as examples. The right measure depends on the workflow: a shorter turnaround may matter only if the output is still accurate and the team can act on it.

Before testing, record how the work is done today, what it costs in time or resources, and how quality is judged. Agree the evaluation period and success criteria with the owner. This baseline worksheet is an editorial application of the guests’ outcome-focused advice, not a guarantee of savings.

• Starting point: volume, time, cost and quality for the current process.
• Pilot scope: who uses it, which work it covers and what remains with people.
• Outcome: the change the business needs and how it will be measured.
• Costs: tools, implementation, review, training and ongoing support.
• Decision: who decides whether to continue, change or stop.

From the interviews: Ronnie Kwesi Coleman · 15:22; Dr. Markus Schmidberger · 25:41

5. Separate activity from outcomes

Dr. Markus Schmidberger explains that token usage describes use of a model but says little about the outcome. He also describes evaluating proof-of-concept work against business value and using that evidence in budget discussions.

Check adoption and tool usage to understand what happened, then compare the actual workflow result with the baseline. Include the work needed to review and correct outputs. If the pilot generated more drafts but created more rework, that is part of its result. Avoid reporting a benefit before you can show how it was measured.

From the interviews: Dr. Markus Schmidberger · 25:41; Dr. Markus Schmidberger · 27:24

Worked example: a draft account update

Illustrative exercise, not a reported customer result: a team spends time assembling an account update from several systems. The first test lets AI prepare a draft from approved sources while an account owner checks it before sharing.

Record the original preparation time and quality checks. During the test, include the time spent gathering context, checking the draft and correcting it. Compare updates with similar scope. Only expand if the owner can show a useful improvement while maintaining the required quality. No savings percentage is assumed.

• Owner: the person accountable for the account update.
• Scope: one team, one update format and approved sources.
• Expert check: missing context, incorrect statements and inappropriate disclosure.
• Expansion decision: measured benefit, support cost and readiness for other teams.

Keep building your approach

Use the workflow guide to work through shared information, permissions and human review before building the pilot.

Explore AI strategy, AI leadership, AI adoption.

Get the next conversation →