We Built a Security Scanner in a Weekend. Here's What Broke.

A founder's Stripe key was sitting in a <script> tag on their production app. We gave ourselves one weekend to build a scanner. Here's the architecture, the false positive problem, and what shipped.

Friday night, 9 PM. A friend shipped his SaaS dashboard. Four hours of Cursor. Auth, payments, database — the whole thing, live on the internet.

I opened DevTools. Searched for sk_live. Found his Stripe secret key in sixty seconds.

We decided to build VibeShield that night. Not a business plan. Not a pitch deck. Just a scanner that catches this stuff before someone else does.

Why We Dropped Everything

The numbers had been piling up for months. An arXiv paper found LLM-generated code consistently produces recurring security weaknesses. AppSec Santa tested 522 code samples across six LLMs and found a 25.7% vulnerability rate. The Cloud Security Alliance reported that AI-assisted developers introduce security findings at ten times the rate of their peers. A Forbes investigation documented 2,000+ high-impact vulnerabilities across 5,600 vibe-coded apps.

But it wasn't the papers that got us. It was watching a real founder realize his Stripe key was public. That look. The mix of confusion and nausea. "But the AI wrote that code."

We built the first working version in three days. Here's what we learned.

The Architecture

We needed something fast, stateless, and cheap to run. Here's what we landed on:

Figure 1: Architecture stack — FastAPI backend, Next.js frontend, Supabase for auth and database.

The scanner itself runs in Python. We chose Python because regex is native, string processing is fast, and the async HTTP client (httpx) handles concurrent fetches cleanly. Every scan is stateless — no session affinity, no in-memory state. You can kill a scanner worker mid-scan and nothing is lost.

The Scanner Pipeline

Every scan flows through six layers:

1. Fetch + normalize — Fetch the URL, follow redirects (up to 5), collect all response headers.

2. Security headers — Check 8 headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, Cross-Origin-*, Strict-Transport-Security) with weighted scoring.

3. Secrets detection — Regex patterns for Stripe (sk_live_), OpenAI (sk-proj-), AWS (AKIA), Supabase (sb_secret_), GitHub (ghp_), SendGrid, and 15+ other providers. Each pattern checks surrounding context to reduce false positives.

4. Cookie analysis — Parse every Set-Cookie header for HttpOnly, Secure, SameSite, Path, and Domain flags. Flag cookies missing any of them.

5. Injection surface — Find innerHTML assignments, dangerouslySetInnerHTML, eval() calls, inline event handlers (onclick=, onerror=), and document.write().

6. Framework fingerprinting — Detect Next.js (look for __NEXT_DATA__, _next/static), check for NEXT_PUBLIC_ prefixed secrets, identify Supabase client initialization patterns.

Each finding comes with: a human-readable title, severity (Critical/High/Medium/Low/Info), exact location (line number and file/URL), the actual HTML or header that triggered it, a copy-paste fix recommendation, and a CWE reference for compliance tracking.

The Thing That Almost Killed Us: False Positives

A scanner that cries wolf trains people to ignore findings. If every sk_live_ match in a blog post about Stripe triggers an alert, people stop paying attention to the real ones.

We built a noise classifier that runs before the secrets detector. It identifies code blocks (text inside <pre>, <code>, or Markdown fenced blocks), documentation sections, and playground embeds, then skips them entirely. The Stripe key pattern is especially prone to false positives because every Stripe tutorial includes sk_live_... as a placeholder.

The classifier isn't perfect. But it cut our false positive rate from "unusable" to "occasionally annoying," which is the bar for a free scanner.

What We Got Right

Three decisions I'd make again:

Deterministic first. The regex-based checks run locally and are fast. AI verification is optional and secondary. You don't need to pay for an LLM to tell you that sk_live_ in a <script> tag is bad.

Machine-readable findings. Every finding is a structured object. Severity, location, evidence, recommendation, CWE. This makes CI/CD integration straightforward — you can pipe findings into a GitHub Action or a Slack alert without parsing prose.

No accounts for basic scans. The free scanner requires no login. Paste a URL, get results. We want the barrier to entry to be "can you type a URL?" because the people who need this most are the ones who don't know they need it.

What We'd Do Differently

The webhook integration with Stripe took 30 minutes. The false positive problem took two weeks and it's still not done. If I were starting over, I'd spend the first day on the noise classifier, not the scanner.

Also: don't build billing before you have users. We spent a full day on Stripe integration that sat unused for weeks. The free tier is what people actually use.

Summary

• Building a security scanner is fast — the false positive problem is what takes weeks. The noise classifier that skips code blocks and documentation cut our false positive rate from "unusable" to manageable.
• Deterministic regex checks catch exposed API keys, missing headers, and cookie issues without needing an LLM. AI verification is useful but secondary — you don't need to pay for an API call to know that sk_live_ in a <script> tag is bad.
• Machine-readable findings (severity, location, evidence, CWE reference) make CI/CD integration straightforward — pipe findings into GitHub Actions or Slack without parsing prose.
• The free, no-account scanner removes every barrier to entry. The people who need this most are the ones who don't know they need it yet.

Found this useful? Scan your app at vibeshield.org — free, no account needed, takes 30 seconds.

Read next: Your App Scored 48/100. Here's What That Actually Means for how the scanner scores findings. If you're dealing with innerHTML XSS, check out Your AI Tool Just Gave You an XSS Vulnerability. Here's How to Find It.