Nine Questions to Ask Any AI Vendor in Finance

Every AI vendor in finance will tell you their system is secure, accurate and compliant. The useful question is not whether they say it, but whether they can answer precisely when asked how.

These nine questions separate a product that has been deployed in a regulated firm from one that has been demonstrated in a meeting. Ask them in one session and watch which ones produce a diagram and which produce an adjective.

1. Where exactly does our client data go?

Not "it's secure". A diagram: which components touch client data, where each one runs, and what crosses a boundary.

A good answer names the jurisdiction of every processing location and shows that client records never leave your controlled environment. A weak one talks about encryption, which protects data in transit and says nothing about where it ends up.

2. Is our data used to train anything?

The answer should be an unambiguous no, in the contract, covering both the vendor's models and any third-party model they call.

Anything hedged — "only in aggregate", "only to improve service" — means yes with conditions. In a firm with confidentiality obligations, that is the end of the conversation.

3. Can it invent an answer, and what stops it?

A vendor who says it cannot happen is either misinformed or hoping you are.

A good answer describes the controls: answers grounded in retrieved documents rather than model memory, citation on every factual claim, and explicit refusal when nothing relevant is found. Then they demonstrate the refusal. A system that declines when it should is far more convincing than one that always has something to say.

4. Who can see what?

An assistant reading across your systems could assemble an answer from sources the user cannot individually open.

A good answer is that permissions come from your source systems and are applied at retrieval, before the model sees anything — not filtered out of the output afterwards. Ask them to prove it: have a relationship manager ask about a portfolio they do not cover.

5. What is logged, and can you query it?

Every question, every answer, every source, every user, with timestamps and a stated retention period.

The follow-up matters more than the answer: "show me every answer given about this client last quarter". Storage is not the same as retrievability, and a supervisor will eventually need the second one.

6. How do you know it still works?

Models change, data changes, behaviour drifts. "We tested it before launch" is a weak answer for a system whose behaviour is not fixed.

A good answer describes ongoing evaluation: a test set with known answers, run on a schedule, tracked over time, with a defined response when quality degrades.

7. What happens when it is wrong?

Not whether — when. The distinction that matters is between failing loudly (unavailable, refusing, flagging low confidence) and failing quietly, which is the dangerous one.

Ask how a wrong answer would be detected, by whom, how quickly, and what the rollback is.

8. Which of your customers look like us?

Not a logo wall. A firm of your size, your regulator, your asset mix, with your deployment constraint — and ideally one you can speak to.

A vendor whose deployments are all considerably larger or smaller than you will be learning your constraints at your expense.

9. What does leaving look like?

Ask early, precisely because nobody wants to. What form does your data come back in, how much of your configuration is portable, what happens to the logs, and how long does it take.

A vendor confident in the product answers this without flinching. Hesitation here tells you how the relationship will feel in year three.

How to run the session

Ask all nine at once, with compliance in the room. Spread across separate calls, weak answers get smoothed over between meetings.

Ask for artefacts, not assurances. A data-flow diagram, a retention policy, an evaluation report. Anything that exists already is evidence; anything promised for later is a plan.

Watch which questions change the room. The ones where a salesperson defers to an engineer are the ones where the real answer lives.

Common questions

Which question is most revealing? The first. An unclear answer about where client data goes ends most reviews in regulated firms, and it should.

Can a cloud-hosted model ever pass? Frequently yes, provided client data never reaches it. The distinction is not where the model runs but what it is sent.

What if we are not technical enough to judge? Bring whoever will run the security review into the first session rather than the last. They ask these questions for a living, and involving them early turns a gate into a design input.

Is a questionnaire enough? No. A questionnaire is answered by whoever writes the best prose. A live session is answered by whoever has built the thing — see what your compliance team will ask.

Put them to us

The fastest way to test any vendor is to ask all nine in one sitting and see which produce a precise answer. Book a demo and bring your compliance team.