AI vendor security questionnaire: the 40 questions that matter

Somewhere in your organisation right now, a vendor is being asked whether their AI product is "secure", and they are answering "yes". Both sides of that exchange know it means nothing, and both sides file the questionnaire anyway. The uncomfortable reality of AI procurement in 2026 is that most security questionnaires were written for SaaS in general, and SaaS questions (encryption at rest, SOC 2, pen test cadence) tell you very little about the risks that are specific to a product with a model in the loop.

The questions that matter are different in kind. What data leaves your tenant and reaches the model, and whose model is it? Is your data used for training or fine-tuning, by the vendor or by their upstream provider, and can you opt out contractually rather than through a settings toggle that might move? What can the product's AI do: what tools can it invoke, with what privileges, against your systems? How does the vendor handle prompt injection through the content your users will inevitably feed it? Has the AI layer specifically been adversarially tested, by whom, and will they share the findings summary rather than just the attestation letter?

Ask those, and vendor conversations change texture immediately. Strong vendors answer crisply because they have done the work. Weak vendors reach for the marketing deck, and the gap is visible to a non-specialist procurement officer, which is what a good questionnaire is for.

This post is the full set: forty questions across data flows, model supply chain, agency and tools, testing and assurance, incident handling and compliance posture. For each, there is a note on what a good answer looks like and which answers should end the conversation. It pairs with the downloadable questionnaire on our resources page. Ironbark Cyber sits on both sides of these reviews, testing AI products for vendors and assessing them for buyers, and these are the questions that consistently separate the two kinds of vendor.

Why do generic SaaS questionnaires miss AI risk?

A standard SaaS questionnaire assumes the software does what its code says and nothing more. A product with a model in the loop breaks that assumption: its behaviour is probabilistic, it acts on untrusted content, and it may reach data and systems its code never explicitly names. Encryption at rest and a SOC 2 report are necessary and say nothing about whether the model will leak one tenant's data into another's context, or take an action an attacker planted in a document. The questions below target the AI-specific surface the generic questionnaire skips. Treat them as a supplement to your existing review rather than a replacement.

Data flow questions (1–8): what reaches the model, and what happens to it?

1. What data leaves our tenant and reaches the model? Good answer: a clear data-flow description. Ender: "we're not sure."
2. Whose model is it, yours or an upstream provider's? Name the provider and hosting region.
3. Is our data used to train or fine-tune your models, or your provider's? Good answer: "no, contractually, for both." Ender: "only if you opt out in settings."
4. Can we opt out of training use in the contract, not just a toggle? The toggle can move; the contract can't.
5. How long is our data, including prompts and outputs, retained, and where?
6. Is data processed or stored outside Australia? Relevant for privacy and, for government, for sovereignty.
7. Are prompts and outputs logged, who can read those logs, and for how long?
8. How is our data deleted on exit, including from vector stores, caches and logs?

Model supply chain questions (9–15): what's behind the product?

9. Which foundation models and versions are in use?
10. How do you manage model version changes, and are we notified before behaviour changes?
11. What sub-processors sit in the AI pipeline? (Model host, vector DB, orchestration, evaluation.)
12. How do you secure fine-tuning or RAG data against poisoning?
13. Do you use open-weight models, and how do you verify their provenance and integrity?
14. How are prompts, templates and agent configurations version-controlled and access-controlled?
15. What is your process when an upstream provider has a security incident?

Agency and tool questions (16–23): what can the AI actually do?

16. What tools, functions or integrations can the AI invoke? This is the single most important question for agentic products.
17. With whose privileges does it act, a shared service account or the requesting user's? Good answer: the user's, scoped per action.
18. Can the AI take consequential actions (write data, send messages, move money) without human approval?
19. How is least privilege enforced on the AI's tools?
20. Can the AI make outbound network requests, and is egress controlled?
21. How are consequential actions authorised against the end user's real permissions?
22. In multi-agent designs, how are trust boundaries between agents enforced?
23. Is there an audit trail of every action the AI takes on our behalf?

Injection and abuse questions (24–30): how is untrusted content handled?

24. How do you defend against indirect prompt injection through documents, email and web content? Good answer: architectural controls. Ender: "the system prompt tells it not to."
25. Is untrusted content isolated from trusted instructions in the context?
26. How is model output sanitised before it reaches a browser, a query or a shell? (Output-handling risk.)
27. What stops one tenant's data appearing in another tenant's context?
28. How do you rate-limit and cap consumption to prevent abuse and cost attacks?
29. Can users or content poison memory or a knowledge base to affect later users?
30. How do you detect and respond to anomalous or adversarial usage?

Testing and assurance questions (31–36): who has attacked it?

31. Has the AI layer specifically had an independent penetration test or red team in the last 12 months? Ender: a SOC 2 report offered as if it answers this.
32. Who performed it, and will you share the findings summary rather than just the attestation?
33. Was testing scoped to AI-specific risks (injection, agency, retrieval) or generic web testing?
34. How were the findings remediated and retested?
35. Do you run ongoing evaluations or red-teaming as the model and product change?
36. Do you align to a recognised framework such as the OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF or ISO/IEC 42001?

Incident, compliance and exit questions (37–40)

37. What is your incident response process for an AI-specific incident, and your notification SLA?
38. How do you govern AI internally, through an AI management system, accountable owner and impact assessments? (ISO/IEC 42001 is the emerging benchmark.)
39. For Australian government or regulated use, can you meet sovereignty, the AU Government AI Technical Standard, and provide the transparency artefacts required?
40. On exit, how is our data and any derived material (embeddings, fine-tunes, logs) returned or destroyed, with evidence?

How do you score the answers?

You are reading for texture as much as content. Use three bands:

• Green flags: crisp, specific answers with evidence offered, such as a data-flow diagram, a contractual clause, or a findings summary from a named tester. Strong vendors answer these quickly because they have already done the work.
• Amber flags: "we're working on it", roadmap answers, or controls that exist only in settings rather than the contract. Not disqualifying, but they set the conditions you negotiate.
• Conversation-enders: "the system prompt tells it not to" as an injection defence; a SOC 2 report offered as proof the AI was tested; an inability to say what the AI can do or what data reaches the model. These aren't gaps to negotiate. They mean the vendor hasn't thought about the risk you're asking about.

The download version of this questionnaire has an answer line under every question so you can run it in a live vendor call. If you'd rather have someone who does this for a living sit in on the review, or if you're the vendor and you'd like your AI product tested before your customers start asking these questions, Ironbark Cyber does both, through its AI governance and assurance and AI security practices. The questions are free; a crisp answer to all forty is the thing worth paying for.