How to know it worked: scorecards for pilots and what to tell your board

Boards and examiners do not ask whether you tried AI. They ask what it did, what it cost and how you know. A small institution can answer all of those with a simple, disciplined pilot. This is how we suggest running one.

Decide the rules before you start

Write down five things on a single page before the pilot begins.

1. The problem you are trying to solve, in one sentence.
2. The baseline, meaning how the work performs today.
3. The scope, including which process, which team and which volume.
4. The duration, usually 60 to 90 days.
5. The success and stop criteria, stated as numbers.

Writing the stop criteria in advance is the part most teams skip, and it is the part that keeps a pilot honest. If the result falls short of the threshold, you stop, and that is a legitimate outcome.

Run it beside the current process first

Start with a period where the tool runs in parallel and changes nothing, so you can compare its output with what people actually do. Then move to a stage where it assists staff, and only later, if the evidence supports it, give it more responsibility. This keeps risk low and produces the comparison data you need.

The scorecard

Five groups of measures cover most needs. Keep each group to two or three numbers so the page stays readable.

• Efficiency: cycle time, touches per item, cost per file.
• Quality: error and rework rates, override rate, exceptions found late.
• Risk and compliance: complaints, fair lending monitors, incidents, control exceptions.
• People: adoption, and what staff did with the time it freed.
• Governance: documentation complete, independent review done, vendor diligence filed.

Do not stop at hours saved

Time saved is easy to claim and hard to verify. What matters is where the time went. It may have cleared a backlog, shortened response times, supported more volume without added staff, or given experienced people room for harder cases. Describe the outcome in those terms, and be candid when the savings were smaller than hoped. Our piece on fixing the process first explains why process changes often deliver more than the tool.

What to tell the board

A one page quarterly report is enough. Include an inventory of AI uses with a risk tier for each, the status of every pilot, the scorecard results, issues and incidents, and what is coming next. Ask the board to approve an AI use policy early, including what staff may and may not do with member data and outside tools, and revisit it as you learn.

Handle shadow use openly

Staff are likely already using AI tools on their own. A clear policy, approved tools and short training do more good than a ban that nobody follows. The inventory described in our model risk piece should include these tools too.

At the end, make a decision

When the pilot ends, choose among three outcomes: scale it, fix it and rerun it, or stop. Record the reasoning either way. An institution that can show examiners a record of disciplined experiments, including ones it ended, looks far stronger than one that adopted tools without evidence.

A practical first step

Before your next pilot, write the one page described above and the one page board report. If you cannot fill them in, the pilot is not ready, and finding that out before spending money is a good result. If you would like a second pair of eyes on either, get in touch.