Going Rogue: The Risks of Deploying LLM Applications and How to Solve Them
- Mathew Lodge
Introduction
Large Language Models (LLMs) offer incredible potential for building advanced conversational applications and text-processing systems. They have the power to transform ways of working and significantly reduce the time spent on tasks that were previously out of reach. However, deploying LLMs also introduces inherent risks that must be carefully managed.
Key challenges in deploying and managing LLM applications
Before launching an LLM-based application, AI development teams must address these four major challenges to ensure reliable and secure performance:
- Unpredictable LLM behavior
LLMs are inherently unpredictable and non-explainable in their responses, despite what “prompt engineers” may claim. Their outputs are based on statistical models predicting the next word in a sequence, making them non-deterministic.
This unpredictability poses significant challenges for quality control and testing. How can you effectively test a model that doesn’t produce consistent, predictable outputs, especially when it’s processing unknown user inputs? - LLM hallucinations
Due to gaps in training data, LLMs may hallucinate, producing incorrect or misleading answers, which can compromise the reliability of your application. - Evolving models, evolving responses
Major LLM providers like OpenAI and Google continuously retrain and refine their models, often with human feedback. As these models evolve, their responses can change over time. This evolution creates a unique challenge: the “right” answer might change, making it difficult to maintain accuracy and consistency over time. - Attacks exploiting LLM unpredictability
Attackers are seeking to exploit the unpredictability of LLMs and endless ways prompts can be constructed to manipulate the model through techniques like prompt injection attacks. These attacks can force the model to divulge sensitive information or generate responses that could embarrass the organization.
These attacks are especially difficult to counter because the models operate in complex ways that we don’t fully understand or predict. This makes it nearly impossible to create foolproof methods for screening prompts and user inputs. The current state of LLM prompt filtering resembles the early days of the email anti-spam arms race in the 1990s, when email services tried to detect spam through tell-tale text patterns. Spammers quickly adapted their tactics, and the cycle repeated with evolving “spam rules”.
A new approach to monitoring LLM models
A major breakthrough in the anti-spam race came from stepping back and addressing the problem more broadly. This led to the identification of sender reputation as the key signal – where the message originated was more important than the text patterns in the message itself. We need a similarly powerful approach for LLMs: prompt filtering alone won’t be enough by itself.
The takeaway: proactive operational monitoring and alerting for production LLM-based applications will be essential. Models will change their answers over time, sometimes correctly, but often introducing errors or hallucinations. Attackers will attempt to manipulate models through prompts, resulting in unwanted responses that must be tracked and addressed.
This is why we advocate for a new infrastructure element dedicated to both operational and security monitoring of LLM models – similar to the anti-spam and content filtering gateways that emerged in earlier iterations of Internet infrastructure.
Introducing Aguru Safeguard
Aguru’s solution to this problem is Safeguard, an on-premises software that monitors the prompts and responses sent to LLM backends within your applications. Requiring no code changes, Safeguard uses semantic analysis to track drift in the model’s responses over time. You can benchmark behavior against “golden responses” and measure semantic drift as the LLM’s outputs evolve. Significant changes trigger alerts, enabling your team to investigate and take appropriate action.
Safeguard also employs custom models to evaluate your LLM’s output, identifying hallucinations or patterns that shouldn’t be present. Alerts are logged and can be sent to security management systems, integrating seamlessly into security operations and response centers.
What Safeguard offers today is just the beginning. As LLM deployments grow, we anticipate that attackers will become more sophisticated, requiring new techniques to monitor and manage prompts and responses. Safeguard’s pluggable architecture is designed to evolve, allowing Aguru to expand its capabilities for a comprehensive “defense in depth” strategy.