The first thing most people do when they try a secrets detection tool is this:
AWS_SECRET_KEY = "FAKEAWSSECRETKEY123456"
PASSWORD = "password123"They run the scan, nothing gets flagged, and the immediate reaction is something along the lines of:
“What a useless tool. My dog could have caught that.”
It feels so obvious. Surely finding secrets is the easiest part of security, right? Just search for password=, throw in a few regexes, and call it a day. How hard can it be?
And in some ways you’re right. Finding strings that look like secrets is easy. Finding real secrets without being buried in false positives is the hard part.
Let’s walk through why testing is harder than it looks, why the worst solutions often look like the best, and how you should actually be evaluating these tools.
How Secrets Detection Works
There are two main approaches to detecting secrets: rule-based pattern matching and entropy statistics.
Rule-based detection relies on regular expressions to spot secrets with a defined structure. AWS keys are a classic example. They always start with the same prefix and have a fixed length, so a regex like this will catch them:
AKIA[0-9A-Z]{16}
It feels powerful when you see it flag a key in code. Until you realize it also flags every placeholder that looks like one.
AWS_ACCESS_KEY_ID="AKIA1234567890123456"
Not so bad for one key but introduce thousands of rules and it quickly gets very noisey. Regex is useful, but it can’t separate real keys from dummy ones and you end up with a fragile, noisy mess.
Filtering with Secret Validation
One of the best ways to cut down false positives is by validating secrets after detection. This usually means making a safe API call. For example, an AWS key can be tested with:
aws sts get-caller-identity --access-key <KEY> --secret-key <SECRET>
If the call succeeds, you have a live key. If it fails, you can safely downgrade the alert.
This is great because you can cast a very wide net and narrow it down later. But here’s the twist. When you test a tool, you aren’t pushing real AWS keys to GitHub. You’re using fake ones. A tool that validates keys will discard them as invalid, showing you zero results. Meanwhile, the lazier tool that flags everything looks like it’s performing better.
Filtering with Entropy Statistics
I guess here we need to quickly explain what entropy means. High entropy strings refer to a string with a large amount of randomness; more randomness = more entropy.
Most secrets cannot be validated, so tools rely on other methods to reduce noise. Entropy statistics are one of the most effective.
The idea is simple: real secrets look random. Placeholders don’t. Consider this fake Stripe key:
StripeKey = "SK_123456789"
It matches the regex, but it isn’t random enough to be real. A genuine key has much higher entropy, something humans are terrible at faking.
English word filtering also helps. Real API keys almost never contain readable words. If you see something like:
TEST823hufb934
you can be pretty confident it’s a placeholder or test credential. Good tools will down-rank or ignore strings that mix high entropy with obvious dictionary words like TEST, PASSWORD, or DEMO. This often causes issues in testing because faking entropy is actually really hard to do as a human; we naturally follow patterns when we type, even if we aren’t aware of it.
Unfortunately this is also not always so straight forward, while API keys are high entropy strings. UUIDs, hashes, and filenames are also high-entropy strings and not secrets. It's then important to also introduce context around the secret.. The best solutions combine entropy, context, and word filtering. This causes issues in testing though because if you are adding in fake credentials that don’t fit the content they are within, they will also be ignored.
Why the Worst Tools Look the Best
This is the paradox. The worst solutions, the ones that just scream at every suspicious-looking string, shine in quick tests. They happily catch your dummy keys and passwords. The smarter tools look broken because they quietly ignore your fakes.
Unless you test with realistic data, you end up praising the noisy tool and dismissing the one that would actually help in production.
How to Test Secrets Detection the Right Way
If you want a fair evaluation, you need better test data.
One option is honey tokens. Services like CanaryTokens let you generate harmless but realistic credentials. A good tool should pick these up instantly.
Another approach is to create real keys with no permissions, run your tests, and revoke them afterwards. This gives you safe but valid input that will trigger validation logic.
The best method, though, is to run the tool on real codebases. Secrets are common in repositories, especially deep in the commit history. Scanning actual projects reveals how a tool behaves under realistic conditions and gives you a benchmark you can trust.
What Makes a Good Secrets Detection Tool
A strong secrets detection tool should do all of the following:
- Validate secrets where possible
Confirm real secrets with safe API calls when providers allow it. - Support specific secret patterns
Detect structured keys like AWS, Stripe, and Twilio using regex or pattern rules. - Handle generic secrets with entropy and context
Use randomness scoring plus surrounding code analysis to catch secrets without fixed patterns. - Filter out fake or test credentials
Down-rank keys containing obvious dictionary words like TEST or PASSWORD. - Cover a wide range of secret types
Beyond API keys, include certificates, SSH keys, database passwords, and more. - Stop leaks before they happen
Provide pre-commit hooks or IDE integrations to prevent secrets from ever entering version control. - Scale across repositories and pipelines
Work effectively in CI/CD, across histories, and at enterprise scale.
Wrapping Up
Secrets detection looks simple, but testing it is anything but. The noisy tools that flag every fake secret can appear impressive, while the smarter tools that validate and filter look like they are doing less.
If you want to test properly, use honey tokens, limited access keys, or real repositories. And when evaluating, look for the qualities that matter in production: validation, pattern detection, entropy analysis, dictionary filtering, wide coverage, and above all, prevention before commit.
Because the fake AWS key you planted for testing isn’t dangerous. The real one hiding in plain sight is.
.avif)
