Examining LLM Systems with System Prompts, RAG and Tools

AI & Machine Learning

A modern LLM application is technically more than a base model. System prompts, retrieval, vector search, knowledge sources, tool calls, agent logic, moderation, permissions and external APIs can materially influence the observed result. A technical examination therefore needs to define the relevant application system, not merely the model name.

1. Distinguish the model from the application system

The base model produces outputs inside an application whose orchestration supplies additional context and may trigger actions. Model provider, API version, gateway, prompt template, retrieval configuration and tools can change independently. A system that uses the same marketed model name can therefore behave differently after changes elsewhere in the application chain.

A technical examination should document which components were active in the disputed or otherwise relevant system state. With SaaS models, the exact server-side model state or provider-side change history can only be recorded to the extent that the provider exposes it technically. The resulting limitation belongs in the conclusion.

Reproducing a test requires more than the user prompt. System instructions, developer instructions, decoding parameters, tool state, permissions and the relevant knowledge sources can all be necessary to reconstruct the tested conditions.

2. System prompts and instruction hierarchy

System and developer prompts define instructions that a user may not see. For a traceable examination, the relevant question is which version was actually used and whether additional dynamic instructions were inserted at runtime. Prompt templates, policy layers and routing logic should therefore be treated as versioned technical artifacts where possible.

OWASP describes prompt injection as a risk in which inputs can unintentionally alter the behavior of an LLM application. Retrieval-Augmented Generation or fine-tuning do not automatically remove that risk. A technical test may therefore examine how external content enters the context, whether trust boundaries are explicit, which validation or isolation mechanisms exist, and what actions become possible after a model output.

The finding should distinguish the model output from the behavior of the surrounding application. A problematic action can depend on application code, permissions or tool handling even when the underlying text generation is unchanged.

3. Retrieval-Augmented Generation and knowledge sources

In a RAG system, the document corpus, chunking strategy, embeddings, search parameters, filters and ranking logic determine which context reaches the model. An answer can therefore change while the base model and user prompt remain identical.

For traceability, document the knowledge base, index state, retrieval parameters and, where available, the documents or chunks actually retrieved for the tested request. A citation displayed in the user interface is not automatically identical to the context that was internally supplied to the model.

Errors can originate from missing documents, an outdated knowledge base, unsuitable retrieval, ranking behavior or the model's interpretation of retrieved context. These causes should be examined separately rather than collapsed into the statement that “the LLM was wrong.”

4. Tools, agents and permissions

LLM applications can use tools to read files, call APIs, execute code or initiate business processes. The permission and control architecture therefore becomes part of the subject of examination. Relevant artifacts can include tool definitions, service accounts, authorization scopes, confirmation steps and execution logs.

Particular attention should be paid to the scope and privileges of service accounts, confirmation mechanisms for consequential actions, input validation and the treatment of tool outputs as potentially untrusted input. The same model output can have very different consequences depending on the permissions attached to the tool layer.

OWASP's GenAI security work covers both LLM-based and agentic applications. Such risk catalogues can structure an examination, but they do not establish a finding for a concrete system. The actual finding must be obtained from the real configuration, data flows, permissions and observed behavior.

5. Reproducibility and limitations

LLM outputs can be stochastic. Reproducibility therefore often does not mean obtaining byte-identical text. Instead, the examination should document the test conditions, repeat relevant trials and define measurable criteria for the property being assessed. Where seeds or deterministic settings are available, they should be recorded, while their technical limitations remain explicit.

A single successful or problematic prompt does not establish general robustness, safety or unreliability. Test set, prompt variants, attack assumptions and detection limits must match the intended conclusion. If only a narrow input class was tested, the conclusion remains narrow as well.

NIST AI 600-1 treats generative AI as a specific risk profile within the AI Risk Management Framework. For technical examination, the relevant consequence is to consider risks across the system context and lifecycle rather than isolating only the base model.

6. Configuration manifest and reproducible test matrix

A configuration manifest is useful for examining an LLM system. It can record the marketed model name and, where available, the concrete model version; system prompt; relevant prompt templates; retrieval configuration; embedding and index state; tool interfaces; permissions; security filters; and decoding parameters.

A test matrix should not merely repeat a few example prompts. It should map input classes to expected system paths, permitted tool actions, relevant knowledge sources and observed outputs. With stochastic generation, multiple repetitions can be needed so that a single random outcome is not overinterpreted.

External states require particular attention. A RAG index may be updated, a tool can return changing data, and a provider may replace a model behind the same product name. Such changes are revision triggers and must be considered when later attempts are made to reproduce an earlier finding.

Further reading

• Machine-Learning Model as a Subject of Examination
• Assessing Machine-Learning Model Robustness
• All articles in AI & Machine Learning

Data Science & AI Assessments

Technical examination of data quality, model validity, reproducibility, bias, robustness and documented AI-system behaviour.

Explore this assessment service