An Antidote to AI-Generated Slop?
Slop is shorthand for AI word salad.
It’s the words and phrases that reoccur time and time again in LLM-generated content.
It’s a consequence of statistical bias that can be traced back to LLM training (process and data).
As a reader, you know it when you see it…. “Welcome to the intricate world of…”
I notice it most when an LLM unnecessarily re-words domain-specific terms I included in my prompt. The same term consistently gets regenerated another way.
I’m pretty relaxed about language and terms in everyday professional use. But not when it’s high stakes: risk assessment reports or advising clients on how best to express their risk and control efforts to external auditors and regulators.
If either of those communications is a “bit off”, it distracts from the underlying message and can trigger far deeper probing than you or your client anticipated.
And no, I’m not about to propose having ChatGPT make live interventions via Zoom during your next regulatory meeting:
Remember, Craig said you should say “key controls” when you’re talking about your good controls. You know the ones you\d prefer the auditor focus on!
One small yet persistent example: if I encourage a client to use the term “key controls” when referring to their most material controls, it’s inconsistent if my report shows up with “important controls” instead of “key controls”.
Yet, leading-edge LLMs (yes, Claude, I’m looking at you) favor that rewording.
I get that this might sound like an inconsequential nit. But it’s yet another unforced LLM GFAR moment (Global Find And Replace). As a heavy LLM user, reducing manual rewrites and rewording while keeping my style and tone is a key satisfaction metric.
In short, I want less LLM-generated slop to fix.
One obvious tactic is to remember to “quote your terms”, but you’ll need to unquote after (or tell the LLM to do that for you).
Another is to write clear language and style instructions for the LLM as part of your prompt. You could include your preferred domain-specific terms the LLM should leave well alone, along with a long list of banned utterances. You need to remember to do this though.
A much smarter way is to auto-wrestle the LLM into textual submission in real-time.
Enter the AntiSlop Sampler by Sam Paeach
The AntiSlop sampler uses a backtracking mechanism to go back and retry with adjusted token probabilities when it encounters a disallowed word or phrase. No more testaments or tapestries or other gpt-slop.
How it works
Define List of Phrases: Provide a list of words or phrases you want to avoid, such as “a tapestry of” or “a testament to”. These can be user-defined and configurable.
Detect Phrase in Output: During text generation, if the model produces any of these specified phrases, it detects their occurrence after the full phrase appears.
Backtrack and Adjust: Once a phrase is detected, the model backtracks to reduce the probability of the initial tokens that led to the phrase, ensuring they are less likely to be used in future outputs.
Customize Slop Amount: You can configure how strongly you want to reduce the probability of these phrases, allowing for light or strong adjustments.
Efficient Lookups: The process is optimized for handling thousands of slop phrases efficiently.
Implemented in Transformers: Currently, this feature is only available in Transformers (and Koboldcpp). It requires direct access to raw logits and the ability to adjust biases, making it difficult to implement with most commercial APIs.
Admittedly, point 6 limits where you can apply the AntiSlop sampler today - for now, to local transformer LLMs. But there’s nothing to stop a leading LLM provider from baking this feature directly into their chat interface.
In a future email, I will r-r-report back on my sampler experiments.