AI Vulnerabilities Exposed

The OWASP Top 10 for Large Language Model (LLM) Applications

The OWASP Top 10 for Large Language Model (LLM) Applications (v1.0, Oct 2023) highlights the most critical risks you need to guard against when building or integrating LLMs. They are:

Prompt Injection ➤ Prompt injection occurs when attackers deliberately manipulate the model’s input to change its behavior.
In a direct prompt injection, they explicitly instruct the LLM to ignore or overwrite prior system instructions — often coaxing it to reveal secret keys or other sensitive information.

Indirect prompt injection happens when malicious instructions are hidden in external sources (for example, a user-uploaded document containing concealed directives) that the model then consumes, causing it to leak customer data during summarization.

Attackers may even prompt the model to generate and execute code on the server or exploit excessive LLM privileges within the application to perform harmful actions. To defend against these threats, rigorously sanitize and validate all inputs and outputs, enforce strict instruction boundaries, and continuously monitor for anomalous or malicious prompt patterns.

By wrapping malicious instructions in Base64, an attacker can bypass simple keyword‐based filters—since the content looks like harmless encoded text—then have the model decode and execute it at runtime. This is not a new vulnerability class but rather an obfuscation/evasion technique within the broader category of prompt injection.

Insecure Output Handling ➤ Failing to validate or sanitize model outputs can let attackers inject scripts or payloads that lead to XSS, SQL injection, SSRF, RCE, etc., in any downstream system that consumes those outputs.

Training Data Poisoning ➤ Malicious or mislabeled examples injected into your training corpus can skew model behavior—backdooring responses, embedding biases, or degrading overall quality.

Model Denial of Service ➤ Resource-exhaustion attacks—sending extremely large, complex, or repeated requests—can overload inference servers, driving up costs or causing downtime.

Supply Chain Vulnerabilities ➤ Compromises in third-party models, libraries, datasets, plugins, or hosting environments can introduce backdoors or poisoned components into your LLM pipeline.

Sensitive Information Disclosure ➤ Overfitting or insufficient data sanitization may cause the model to regurgitate PII, credentials, or proprietary content embedded in its training data.

Insecure Plugin Design ➤ Plugins and extensions that interface with your LLM must enforce strict input checks and least-privilege controls — otherwise they can become vectors for SQL injection, RCE, or data exfiltration.

Excessive Agency ➤ Giving an LLM too much autonomy (e.g. automatically executing actions based on outputs) can lead to unauthorized transactions, data deletions, or other unwanted side effects if the model hallucinates or is tricked.

Overreliance ➤ Blind trust in model outputs—without human review—can result in propagation of hallucinations, biases, or errors into critical business processes.

Model Theft ➤ Public-facing APIs or misconfigured access controls may allow adversaries to copy or steal your proprietary model weights or fine-tuned checkpoints.

Best Practices for Secret Management with AI

• Out-of-Model Authentication ➤ Store your DB/API keys in a secrets manager (Vault, AWS KMS, Azure Key Vault). Your application code (not the model) retrieves a short-lived token and injects it into your DB-access library.
• Ephemeral, Scoped Credentials ➤ Use “just-in-time” credentials with minimal scopes (e.g., read-only, single-table). Rotate automatically (e.g., every 15 min via STS / role‐assumption).
• Prompt/Response Sanitization ➤ Strip or redacted any user input or model output that might contain sensitive tokens before logging or feeding back into the system. Enforce content filters or output classifiers that block “looks-like-a-key” patterns.
• Zero-Trust Network Controls ➤ Ensure the model-serving endpoint cannot directly talk to your production DB—force all queries through a hardened application layer. Apply strict firewall and VPC rules so that even if the model did “know” the key, it couldn’t use it on its own.
• Continuous Monitoring & Auditing ➤ Alert on any prompt containing “key” strings or your key’s prefix. Audit model logs for unusual “disclosure” attempts.
• Secure Development Practices ➤ Best practice: Read the DB key in your application logic, then call your database client directly — don’t ever interpolate it into the text you send to the model.
• Runtime Access Control ➤ Restrict file permissions on your .env so that only the application user can read it (chmod 600 .env). Don’t mount it into any container or environment where the model-serving process runs unless strictly necessary—and even then, never feed it into the prompt.
• Monitoring & Alerting ➤ Scan your prompt logs for any process.env dumps or suspicious patterns (e.g. strings matching your key format). Alert and rotate the key immediately if you detect a leak.

Tool Management and secrets

Because your tool’s code never reads or returns the actual contents of process.env.WEATHER_API_URL (or the API key), a user asking “What is the WEATHER_API_URL?” will only ever get back whatever your _call method explicitly returns—either a weather string or an error message. The model itself doesn’t have access to your environment, only to the tool’s outputs.

That said, here are a few best practices to make absolutely sure nothing leaks:

• Don’t Echo Errors or Stack Traces ➤ In your catch block you already return a generic "Error fetching weather data." — good. Avoid returning the raw error.message or error.stack, which in some runtimes could include the request URL.
• Sanitize Any Debug Logging ➤ If you log requests or responses for debugging, strip out or mask the URL and key before writing to logs.

  
  E.g. console.log("Calling weather API:", { url: url.replace(/(appid=)[^&]+/, "$1***") });
  

• Keep Secrets Out of Your Tool Description ➤ Your tool description mentions that it uses process.env.WEATHER_API_URL and process.env.WEATHER_API_KEY — but not what they are—that’s fine. Just don’t include actual values there.
• Limit Tool Permissions ➤ If someone did somehow hijack the tool, they’d only be able to fetch weather data. They can’t “read” arbitrary env vars. Consider running your tool in a scoped environment (e.g. a container or lambda) that only has the weather URL/key in its env.
• Use a Managed Secrets Vault ➤ Instead of a flat .env file, store your URL/key in a secrets manager (AWS Secrets Manager, Vault). Fetch them at cold start and then discard, so even if your container is compromised, there’s no persistent .env.
• Rate-Limit and Monitor ➤ Apply request quotas on your tool’s entry point to stop abuse or probing attacks. Alert on repeated failed calls or unusual query patterns.

Bottom line: as long as your _call method only ever returns weather descriptions and doesn’t interpolate or log the raw env values, there’s no way for a user — even a savvy “jailbreak” prompt — to make the model spit out process.env.WEATHER_API_URL or the actual URL/key.

LLM by itself has no “eyes” on your file system or environment. It only knows what your application code explicitly feeds it in the prompt or via a tool’s return value. In other words, the model can’t magically read your .env; it can only regurgitate whatever strings you’ve loaded into its context.

As long as your code never loads or returns .env contents into the model’s input or tool outputs, there is no mechanism for the LLM to “reach out” and read environment variables on its own. Keep secrets strictly out of the prompt/tool boundary, and rely on file-system permissions and secret managers to enforce the rule in case of human error.