While waiting for some scans to finish, I got bother about how much "dead" time there was and naturally the question came:

What would be the FASTEST way to look for secrets in a website's public files?

So here I spend a few hours tinkering about it and created a small PoC for fun

Criteria

1. Limited interaction with the app -> this usually takes the longest time so we don't do direct scanning at all since the app might be slow to reply, be protected by WAF or block us

2. Use "cached" information -> using 3rd party cached information (like waybackmachine) to quickly find a list of public available files

3. Use multi-threading -> golang or C tools are ideal. Tools like curl, wget or bash/python scripts with console output perform much worse and should be avoided

4. Discard unnecessarily info fast -> quickly discard extensions like png, jpg, svg, mp4, etc. and paths that don't return 200 OK

5. Use regex rules and local tools -> low false-positive, optimized for secrets findings that run locally without any network request or API key reqs

Process

1. Get a list of public exposed files of the app from cached sources -> using gau

2. Filter out trash (png, jpg, svg, css, mp4, etc.) -> using grep

3. Check which paths return 200 OK -> using httpx

4. Download all files that return 200 OK -> using aria2c

5. Scan them -> using gf and trufflehog

Merge Everything Together

#!/bin/bash

# Check for domain argument
if [ -z "$1" ]; then
  echo "Usage: $0 <domain>"
  exit 1
fi

DOMAIN=$1

# Step 0: Create folder and move inside
mkdir $DOMAIN
cd $DOMAIN
echo "[+] Starting recon for $DOMAIN"

# Step 1: Fetch URLs using gau
echo "[+] Running gau"
gau "https://$DOMAIN" > gau.txt

# Step 2: Remove duplicates
echo "[+] Removing duplicates"
sort -u gau.txt > gau_unique.txt

# Step 3: Filter out non-text/binary files
echo "[+] Filtering extensions"
grep -Eiv '\.(png|jpe?g|gif|bmp|svg|ico|mp4|mp3|avi|mov|mkv|wav|webm|zip|tar|gz|7z|rar|exe|dll|bin|iso|pdf|ttf|woff2|css)$' gau_unique.txt > gau_filtered.txt

# Step 4: Check status codes (200)
echo "[+] Keeping only 200s"
cat gau_filtered.txt | httpxx -silent -status-code -fr | grep 200 > gau_200.txt

# Step 5: Extract only URLs
echo "[+] Extracting URLs"
cut -d " " -f 1 gau_200.txt > urls.txt

# Step 6: Create download folder
mkdir -p files
cd files

# Step 7: Download files in parallel safely
aria2c -i ../urls.txt -j 5 -x 1 -s 1 --file-allocation=none --check-certificate=false --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:146.0) Gecko/20100101 Firefox/146.0" >  /dev/null 2>&1
count=$(ls | wc -l)
echo "Downloaded $count files"
cd ..

# Step 8: Scan files for secrets with TruffleHog
echo "[+] Truffleing"
trufflehog filesystem ./files --json > trufflehog.results 2>&1

# Step 9: Scan files for secrets with gf
echo "[+] GF secrets"
for i in files/*; do { echo "=====$i======="; gf secrets $i; echo "-----------"; } >> gf.results 2>&1; done

echo "Scan complete. Results in trufflehog.results and gf.results"

Run the script

>bash script.sh <target.com>

References

• https://github.com/lc/gau

• https://docs.projectdiscovery.io/opensource/httpx/usage

• https://cleveruptime.com/docs/commands/aria2c

• https://github.com/trufflesecurity/trufflehog

• https://github.com/tomnomnom/gf

GF Secrets Patterns

└─$ cat ~/.gf/secrets.json                                      
{
  "flags": "-HanriE",
  "pattern": "([a-zA-Z][a-zA-Z0-9+.-]*://|AKIA[0-9A-Z]{16}|eyJ[A-Za-z0-9_-]*\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+|ASIA[0-9A-Z]{16}|A3T[A-Z0-9]{16}|AGPA[0-9A-Z]{16}|AIDA[0-9A-Z]{16}|ABIA[0-9A-Z]{16}|ANPA[0-9A-Z]{16}|AROA[0-9A-Z]{16}|AIPA[0-9A-Z]{16}|ANVA[0-9A-Z]{16}|AIKA[0-9A-Z]{16}|AQOA[0-9A-Z]{16}|AKPA[0-9A-Z]{16}|AIza[0-9A-Za-z_-]{35}|ya29\\.[0-9A-Za-z_-]+|sk_live_[0-9a-zA-Z]{24}|sk_test_[0-9a-zA-Z]{24}|xox[baprs]-[0-9A-Za-z-]{10,48}|xoxc-[0-9A-Za-z-]{10,48}|xoxa-[0-9A-Za-z-]{10,48}|-----BEGIN (RSA|DSA|EC|PGP) PRIVATE KEY-----|ghp_[0-9A-Za-z]{36}|gho_[0-9A-Za-z]{36}|ghu_[0-9A-Za-z]{36}|ghs_[0-9A-Za-z]{36}|github_pat_[0-9A-Za-z_]{22,38}|EAACEdEose0cBA[0-9A-Za-z]+|SG\\.[A-Za-z0-9_-]{22}\\.[A-Za-z0-9_-]{43}|key-[0-9a-zA-Z]{32}|-----BEGIN OPENSSH PRIVATE KEY-----|ssh-rsa AAAA[0-9A-Za-z+/]{100,}|shpat_[0-9A-Za-z]{32}|shpss_[0-9A-Za-z]{32}|shpca_[0-9A-Za-z]{32}|x-api-key[\"'=:\\s]{1,6}[A-Za-z0-9_\\-]{16,64}|Bearer\\s+[A-Za-z0-9\\-_\\.]{20,200})"
}