LLM penetration testing methodology: what we actually test

Ask five vendors what an "LLM penetration test" covers and you'll get five different answers, three of which are a vulnerability scan with a prompt-injection wordlist stapled on. That ambiguity is a problem for buyers, because the difference between testing the model and testing the system is the difference between theatre and evidence.

Here is the uncomfortable truth about LLM security: the model is rarely where the exploitable risk lives. The model vendor (OpenAI, Anthropic, Google, whoever) has more adversarial pressure on their guardrails in a day than your tester will apply in a fortnight. What they cannot secure is everything you wrapped around the model: the system prompt with credentials in it, the retrieval pipeline that happily serves another tenant's documents into the context window, the tool layer that lets the model call your internal APIs with service-account privileges, and the downstream code that takes model output and feeds it into a query, a shell command, or a customer's browser.

So when Ironbark Cyber runs an LLM penetration test, we test the application as one system. Prompt injection matters, but it matters as a delivery mechanism. The useful finding is not "the model can be made to say something rude", it is "a crafted document in the knowledge base leads to a tool call that exfiltrates the CRM". We chain the probabilistic layer and the deterministic layer, because that is what an attacker does, and neither layer's defences hold on their own.

This post lays out our methodology end to end: what we map before testing starts, the attack classes we work through, how we handle non-determinism in reproduction steps, and how findings map to the OWASP Top 10 for LLM Applications and MITRE ATLAS without the framework becoming the point.

What do we map before the first prompt is sent?

Testing starts with a threat model rather than a prompt. On the scoping call we build a picture of the application as a system: where untrusted input enters, what data reaches the context window and from where, which tools the model can invoke and with whose privileges, how tenants are separated, and where downstream code trusts model output. That map tells us where the exploitable paths are likely to be, so testing time is spent attacking real risk rather than re-confirming that a chatbot can be made to write a poem about crime.

This is also where we separate the model's responsibilities from yours. The vendor secures the model; you own the system prompt, the retrieval data, the tool permissions, tenant isolation and output handling. Those are the surfaces we attack, because those are the surfaces you can fix.

Direct and indirect prompt injection: sources, sinks and delivery

Prompt injection (LLM01 in the OWASP Top 10 for LLM Applications) is treated as a delivery mechanism rather than an end in itself. We enumerate the sources of attacker-influenced content: direct user input, but far more importantly the indirect routes, such as retrieved documents, emails, web pages, uploaded files and tickets, anything that reaches the context. We then trace each source to a sink: a tool call, a data disclosure, a downstream code path. A successful finding connects the two, so that content an attacker controls reaches the model and causes an action that harms you. That is the version of prompt injection with a blast radius. Indirect injection is where the real risk lives and where defences are usually thinnest.

The retrieval layer: leakage, poisoning and permission mismatches

Retrieval-augmented generation is where a surprising share of serious findings come from, and it maps to several OWASP categories at once: sensitive information disclosure (LLM02), data and model poisoning (LLM04) and vector and embedding weaknesses (LLM08). We test whether the retriever respects tenant and permission boundaries or quietly serves documents the user should never see; whether content ingested into the knowledge base can be poisoned to manipulate later, legitimate users; and whether embeddings or the vector store leak information across boundaries. A model that behaves perfectly can still be a data-exfiltration engine if the retriever hands it the wrong documents.

Tool and function abuse: excessive agency in practice

Excessive agency (LLM06) is where an LLM feature becomes a real security problem. We map every tool the model can invoke, the privileges it runs with, and what an attacker can make it do through crafted input. The question is not whether the model can call a given tool, but what the worst thing is that an attacker can achieve by steering it. The answer is often uncomfortable when a model can query databases, call internal APIs or take actions with a service account's rights. This work overlaps with dedicated AI agent security testing, which we scope alongside an LLM pentest whenever the product is agentic.

Output handling: where model text becomes code

Improper output handling (LLM05) is the classic-vulnerability layer wearing an AI hat. Wherever downstream code trusts model output, injection follows: markdown or HTML rendered into a browser becomes stored XSS; a generated URL fetched server-side becomes SSRF; model text interpolated into a query or a shell command becomes injection of the ordinary, decades-old kind. We test the seams where probabilistic output meets deterministic code, because that is where a model quirk turns into a real exploit.

Guardrail bypass that matters versus jailbreak theatre

Not every jailbreak is a finding. Getting a model to say something off-brand is theatre. Getting it to approve a refund, disclose a record, or execute an action it should have refused is a vulnerability. We test guardrails against your risk profile, meaning the outcomes that would hurt you, and iterate the way attackers do, in combination and over many attempts, rather than declaring victory on the first clever prompt. The report distinguishes the two clearly so you can tell reputational noise from exploitable risk.

The classic layer around the AI feature

Attackers don't respect the boundary between an AI bug and a web bug, and neither do we. Every LLM pentest includes the conventional application testing around the feature: authentication and session handling, authorisation between roles and tenants, rate limiting (which doubles as abuse and cost control, touching LLM10 unbounded consumption), and the business logic the feature sits inside. The highest-impact findings almost always chain an AI-layer issue with a classic-layer one.

How do you reproduce a probabilistic finding?

Non-determinism is the objection buyers raise most, and it is manageable. We reproduce findings by capturing the full context, meaning the exact inputs, retrieved content, tool state and configuration, and by demonstrating reliability across repeated attempts rather than a single lucky run. A finding is written up with the attack path, the conditions under which it fires, an observed success rate, and remediation guidance, so your engineers can both reproduce it and confirm the fix. "It only worked once" is a reason to keep testing rather than to dismiss the issue, because a technique that succeeds one time in ten is still a vulnerability an attacker will automate.

Framework mapping without the framework becoming the point

Findings map cleanly to the OWASP Top 10 for LLM Applications and MITRE ATLAS, and to the NIST AI RMF where your stakeholders need it, so auditors and risk committees get the language they expect. But the checklist is the floor rather than the method. The highest-value findings come from your application's specific business logic and integrations, which no generic list anticipates. The framework is how we communicate results, not how we decide where to look.

Ironbark Cyber delivers LLM penetration testing as part of a broader AI security and assurance practice: senior-only testing of the whole system, findings ranked by real-world impact, critical issues escalated live, and a free retest of fixed issues within 90 days. If a report you've been handed reads like a scanner ran a prompt-injection wordlist, you bought the theatre. This is what the evidence looks like instead.