Security Pentester skill

Security Pentester

Disambiguation: this skill = active offensive testing. For defensive code patterns see security-hardening. For runtime threat intel / URL+wallet scam scanning see security-sentinel.

Autonomous web application penetration testing driven by an LLM-agent pipeline (Shannon), backed by manual validation. Source-aware analysis combines code reading with live exploitation attempts and prioritizes findings that come with a reproducible proof-of-concept.

This skill is tool-agnostic in principle: Shannon is the reference automated driver, but the workflow (recon → analyze → exploit → triage → remediate → regression-test) and every remediation playbook below apply to any pentest engagement (manual, Burp/ZAP-driven, or other agents).

Scope first. Only run against applications you own or have explicit written authorization to test, and never against production. See §8 for the full rules of engagement.

Core Principle

Evidence over assertion. Prefer findings that ship a reproducible proof-of-concept (PoC) over unproven "potential" findings. A PoC sharply reduces false positives but does NOT guarantee correctness: an LLM-driven pipeline can still hallucinate impact, mis-read a response, or fire in a test-only configuration. Treat every automated finding as a lead, not a verdict.

Manual validation is mandatory for any finding that drives a security decision — not only Critical/High. Before you file a ticket, block a deploy, or tell anyone "you are vulnerable," reproduce the PoC yourself against the in-scope target and confirm real impact (see section 4 — False Positive Identification).

---

1. Vulnerability Coverage

OWASP Top 10 Testing Matrix

The high-volume auth tests above (brute force, credential stuffing, 2FA enumeration) and the SSRF metadata probes are noisy and side-effecting: they can lock real accounts, blow rate budgets, trigger SMS/email/billing, and page on-call. Do not run them against any environment without the safe-test controls in §8 — Safe Testing Practices.

OWASP Web Security Testing Guide (WSTG) Coverage

WSTG-INFO  — Information Gathering            ✓ Automated
WSTG-CONF  — Configuration Management         ✓ Automated
WSTG-IDNT  — Identity Management              ✓ Automated
WSTG-ATHN  — Authentication Testing           ✓ Automated
WSTG-ATHZ  — Authorization Testing            ✓ Automated
WSTG-SESS  — Session Management               ✓ Automated
WSTG-INPV  — Input Validation                 ✓ Automated
WSTG-ERRH  — Error Handling                   ✓ Automated
WSTG-CRYP  — Cryptography                     ◐ Partial (TLS config, weak hashing)
WSTG-BUSN  — Business Logic                   ✗ Manual (no automated tool reliably models domain rules — see §7)
WSTG-CLNT  — Client-Side Testing              ✓ Automated (DOM XSS, open redirects)
WSTG-APIS  — API Testing                      ✓ Automated (REST, limited GraphQL)

---

2. Running a Pentest

Quick Start

# Prerequisites: Docker (worker container) + Node.js 18+.

# Configure credentials with the interactive wizard (Anthropic recommended).
# Replaces manually appending ANTHROPIC_API_KEY to .env.
npx @keygraph/shannon setup

# Run against a target (white-box, source-aware, finds more vulns).
# Pass the target repo with -r; it is mounted read-only in an ephemeral Docker worker.
npx @keygraph/shannon start -u https://target-app.example.com -r /path/to/your-repo

Configuration (shannon.yaml)

# Authentication config — tell Shannon how to log in
auth:
  login_url: /login
  credentials:
    - username: testuser@example.com
      password: TestPass123!
      role: user
    - username: admin@example.com
      password: AdminPass456!
      role: admin

# Scope rules
rules:
  avoid:
    - /api/admin/delete-all    # Don't hit destructive endpoints
    - /api/billing/*           # Skip billing endpoints
    - /logout                  # Don't log yourself out
  focus:
    - /api/*                   # Prioritize API endpoints
    - /dashboard/*             # Focus on authenticated surfaces

# 2FA support (if app uses TOTP)
totp:
  secret: JBSWY3DPEHPK3PXP   # PLACEHOLDER — replace with your test account's actual TOTP secret

CLI Commands

Flag names, subcommands, and report file paths change between releases. Verify against the current KeygraphHQ/shannon README (npx @keygraph/shannon --help) before scripting against them; the names below are the reference set as of Jul 2026. The old git clone + ./shannon <cmd> KEY=VALUE form (URL=, REPO=, CONFIG=, ID=, CLEAN=true, WORKSPACE=) is the pre-2026 invocation; current Shannon uses npx @keygraph/shannon with flag args (-u, -r, -c, -w). The source-build clone still exists but runs ./shannon with the same flags.

npx @keygraph/shannon setup                        # One-time credentials wizard
npx @keygraph/shannon start -u <url> -r <repo>     # Start full pentest (repo mounted read-only)
npx @keygraph/shannon start -u <url> -r <repo> -c shannon.yaml  # With config (always use in CI)
npx @keygraph/shannon start -u <url> -r <repo> -w <name>        # Named workspace (resume with same -w)
npx @keygraph/shannon workspaces                   # List all workspaces
npx @keygraph/shannon logs <workspace>             # Tail live logs
npx @keygraph/shannon status                       # Check progress
npx @keygraph/shannon stop                         # Stop containers (preserves data, safe)

Destructive cleanup, guard it. npx @keygraph/shannon stop --clean deletes ALL workspace data: reports, PoCs, recon output, and logs. There is no undo. Shannon confirms before deleting (skip the prompt with --yes/-y), but never put the --yes form in an unattended script or CI job. Always export first and require an explicit confirmation:

# 1. Export everything you might need before destroying it.
#    npx mode stores workspaces under ~/.shannon/workspaces/ (source-build: ./workspaces/).
WS="$HOME/.shannon/workspaces/<name>"
mkdir -p "./shannon-archive/<name>-$(date +%Y%m%d)"
cp -r "$WS" "./shannon-archive/<name>-$(date +%Y%m%d)/" 2>/dev/null

# 2. Confirm interactively before the irreversible step
read -r -p "Archived. DELETE all Shannon workspace data now? type 'DELETE': " ok
[ "$ok" = "DELETE" ] && npx @keygraph/shannon stop --clean --yes || echo "Aborted, data preserved."

---

3. Understanding the Pipeline

4-Phase Architecture

Phase 1: RECONNAISSANCE
  ├── Pre-Recon (source code analysis with configured LLM)
  │   └── Outputs: code_analysis_deliverable.md
  └── Recon (attack surface mapping with Playwright + Nmap)
      └── Outputs: recon_deliverable.md

Phase 2: VULNERABILITY ANALYSIS (5 parallel agents)
  ├── Injection Analysis   → injection_analysis.md + exploitation_queue.json
  ├── XSS Analysis         → xss_analysis.md + exploitation_queue.json
  ├── Auth Analysis        → auth_analysis.md + exploitation_queue.json
  ├── SSRF Analysis        → ssrf_analysis.md + exploitation_queue.json
  └── AuthZ Analysis       → authz_analysis.md + exploitation_queue.json

Phase 3: EXPLOITATION (5 parallel agents, conditional)
  ├── Injection Exploit    → injection_exploitation_evidence.md
  ├── XSS Exploit          → xss_exploitation_evidence.md
  ├── Auth Exploit         → auth_exploitation_evidence.md
  ├── SSRF Exploit         → ssrf_exploitation_evidence.md
  └── AuthZ Exploit        → authz_exploitation_evidence.md

Phase 4: REPORTING
  └── Security-Assessment-Report.md

What Each Phase Does

Pre-Recon reads source code to understand the application architecture, identify entry points, map data flows, and find potential vulnerability patterns before any network interaction.

Recon maps the live attack surface: crawls the app with a headless browser, enumerates API endpoints, identifies technologies, scans for open ports.

Vulnerability Analysis agents work in parallel, each specializing in one category. They combine source code knowledge with recon data to hypothesize specific vulnerabilities and create exploitation queues.

Exploitation agents receive the queues and attempt real attacks using browser automation (Playwright) and HTTP requests. Only proven exploits are included in the final report.

---

4. Interpreting Reports

Severity Levels

Reading a Finding

Each finding in the report includes:

## [CRITICAL] SQL Injection in /api/users/search

**Endpoint:** GET /api/users/search?q=
**Parameter:** q
**Type:** Union-based SQL injection

### Proof of Concept
GET /api/users/search?q=' UNION SELECT username,password,NULL FROM users--

### Response Evidence
HTTP/1.1 200 OK
[{"username":"admin","password":"$2b$12$...","3":null}]

### Source Code Reference
File: src/routes/users.ts:42
const results = await db.query(`SELECT * FROM users WHERE name LIKE '%${req.query.q}%'`);

### Remediation
Use parameterized queries:
const results = await db.query('SELECT * FROM users WHERE name LIKE $1', [`%${req.query.q}%`]);

False Positive Identification

Shannon's "no exploit, no report" policy minimizes false positives, but review for:

• Environment-specific: Exploit only works in test environment (different DB, debug mode)
• Already mitigated: WAF or middleware blocks the attack in production but not staging
• Intended behavior: Feature that looks like a vulnerability (e.g., admin search returns all users by design)
• LLM hallucination: Report claims a vulnerability but the PoC doesn't actually demonstrate impact

Always verify the PoC manually for Critical/High findings before filing tickets.

---

4a. Remediation Playbooks

A finding is only closed when the secure pattern is in place and a regression test proves the PoC no longer works. Below are framework-specific fixes for each class Shannon covers. Pair each fix with a test (see §6 — Regression Testing).

SQL / NoSQL Injection

• Root cause: untrusted input concatenated into a query.
• Fix: always parameterize; never build query strings. Prefer a query builder / ORM with bound parameters.

  // ❌ const r = await db.query(`SELECT * FROM users WHERE name LIKE '%${q}%'`);
  // ✅ Node + pg
  const r = await db.query('SELECT * FROM users WHERE name LIKE $1', [`%${q}%`]);
  // ✅ Mongo: never pass req.body/req.query straight into a filter — cast & whitelist:
  await User.find({ name: String(q) });   // reject objects so {$ne:null} can't slip in
  

• Also: least-privilege DB user (no DDL), reject $/. keys in JSON filters, validate types at the edge (zod/Joi).

Cross-Site Scripting (XSS)

• Root cause: untrusted data rendered into HTML/JS/attribute/URL context without context-correct encoding.
• Fix: rely on the framework's auto-escaping; never bypass it with raw-HTML sinks on untrusted data.

  // ✅ React/Vue/Svelte auto-escape {value}. The danger is the escape hatch:
  // ❌ <div dangerouslySetInnerHTML={{ __html: userInput }} />
  // ✅ If you MUST render HTML, sanitize first:
  import DOMPurify from 'dompurify';
  <div dangerouslySetInnerHTML={{ __html: DOMPurify.sanitize(userInput) }} />
  

• Defense in depth: a strict Content-Security-Policy (no unsafe-inline; use nonces/hashes), HttpOnly+Secure+SameSite cookies, and URL-scheme allowlists (https: only) to block javascript: sinks.

SSRF (Server-Side Request Forgery)

• Root cause: server fetches a user-supplied URL.
• Fix: allowlist destinations; resolve the host and reject private/link-local ranges; disable redirects to new hosts.

  import dns from 'node:dns/promises';
  import ipaddr from 'ipaddr.js';
  async function assertPublicUrl(raw) {
    const u = new URL(raw);
    if (!['http:', 'https:'].includes(u.protocol)) throw new Error('scheme');
    const { address } = (await dns.lookup(u.hostname));
    const r = ipaddr.parse(address).range();           // 'private' | 'loopback' | 'linkLocal' | ...
    if (['private','loopback','linkLocal','uniqueLocal','reserved'].includes(r)) throw new Error('blocked');
    return u;                                            // fetch with redirect: 'manual', re-check each hop
  }
  

• Cloud: enforce IMDSv2 so a basic SSRF can't read instance credentials; egress-firewall the service.

Command Injection

• Root cause: user input reaches a shell.
• Fix: never invoke a shell with interpolated input; pass an argv array to execFile/spawn with shell:false.

  // ❌ exec(`convert ${file} out.png`);            // shell metacharacters → RCE
  import { execFile } from 'node:child_process';
  execFile('convert', [file, 'out.png'], { shell: false }, cb);  // ✅ args never parsed by a shell
  

• Allowlist the binary and validate args (e.g. filename matches ^[\w.-]+$).

Broken Authorization (IDOR / privilege escalation)

• Root cause: the handler trusts a client-supplied id/role without checking the current user owns or may access it.
• Fix: enforce object-level authz on every read/write, server-side, from the session — not from a request field.

  // ❌ const doc = await Doc.findById(req.params.id);          // any id → anyone's doc
  // ✅ scope the query to the authenticated principal
  const doc = await Doc.findOne({ _id: req.params.id, ownerId: req.user.id });
  if (!doc) return res.sendStatus(404);                         // 404, not 403 (don't confirm existence)
  // role checks come from the verified session/JWT claims, never from req.body.role
  

• Use centralized policy (e.g. CASL/OPA), deny-by-default, and avoid sequential/guessable ids (use UUIDs).

Broken Authentication (sessions, JWT, brute force)

• Fix: verify JWTs with a pinned algorithm (algorithms:['RS256']) and reject alg:none; rotate the session id on login (kills fixation); short-lived access tokens + rotating refresh tokens; bcrypt/argon2 for passwords; rate-limit + lockout/backoff on login.

  jwt.verify(token, PUBLIC_KEY, { algorithms: ['RS256'] }); // never accept attacker-chosen alg
  req.session.regenerate(() => {/* set new session after successful auth */});
  

Insecure Deserialization

• Root cause: untrusted bytes turned into objects that can execute code on construct.
• Fix: don't deserialize untrusted data into rich objects. Use a data-only format (JSON) and validate against a schema; never node-serialize/Java native readObject/Python pickle.loads/PHP unserialize on user input.

  const data = JSON.parse(body);          // data only, no behavior
  const safe = MySchema.parse(data);      // zod: reject unexpected shape/types
  

HTTP Request Smuggling

• Root cause: front-end and back-end disagree on request boundaries (Content-Length vs Transfer-Encoding).
• Fix: mostly an infra fix — use HTTP/2 end-to-end or a proxy that normalizes/rejects ambiguous framing; reject requests containing both Content-Length and Transfer-Encoding; keep proxy and origin on the same HTTP version and patched. Validate with smuggler/h2csmuggler (see §7).

Security Misconfiguration / headers

• Fix: ship secure defaults — helmet() (Express) or framework equivalents — setting HSTS, X-Content-Type-Options: nosniff, X-Frame-Options: DENY (or CSP frame-ancestors), a strict CSP, and disabling stack traces / X-Powered-By in production.

---

5. CI/CD Integration

Never run a full active pentest in CI without a committed scope config. The job below requires shannon.yaml (-c) so the destructive-endpoint denylist, rate limits, and isolated test accounts always apply. A scan with no scope rules can hammer auth endpoints, trigger SMS/email/billing, and lock accounts even in a "test" stack.

Networking note (Linux runners). host.docker.internal does not resolve by default on GitHub-hosted ubuntu-latest. There are two robust options:

• App published to the runner host (e.g. docker compose ... up -d mapping 3000:3000): target http://localhost:3000 and run Shannon directly on the host. This is what the workflow below does.
• Shannon itself running in Docker, needing to reach the host: start that container with --add-host=host.docker.internal:host-gateway (Docker ≥ 20.10), then target http://host.docker.internal:3000. Put app + Shannon on a shared user-defined network and address the app by its service name instead, when possible.

CI scope config (commit this next to the workflow)

# .github/shannon-ci.yaml — mandatory scope for automated runs
auth:
  login_url: /login
  credentials:
    # Disposable accounts seeded ONLY in the ephemeral CI database.
    # Real user accounts must never appear here.
    - username: ci-user@test.local
      password: ${CI_TEST_USER_PW}      # injected from CI secret, not committed
      role: user
rules:
  avoid:
    - /api/admin/**            # privileged / destructive admin actions
    - /api/billing/**          # never trigger real charges/refunds
    - /api/payments/**
    - "**/delete*"             # bulk-delete style endpoints
    - "**/export*"             # data-exfil heavy endpoints
    - /logout                  # don't log the test session out
    - /api/notifications/**    # don't fan out email/SMS/push
  focus:
    - /api/**
    - /dashboard/**
limits:
  max_requests_per_second: 5   # cap noise/cost; keep under app rate limits
  max_agents: 3
  max_steps: 40

Pre-Deploy Security Gate

# .github/workflows/security.yml
name: Security Pentest
on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 1'  # Weekly Monday 2am

jobs:
  pentest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Start isolated test application
        run: docker compose -f docker-compose.test.yml up -d
        # docker-compose.test.yml maps "3000:3000" and seeds a throwaway DB (see §8)

      - name: Wait for app
        run: |
          for i in $(seq 1 30); do
            curl -fsS http://localhost:3000/health && break
            sleep 2
          done

      - name: Run Shannon pentest (scoped)
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          CI_TEST_USER_PW: ${{ secrets.CI_TEST_USER_PW }}
        run: |
          # ANTHROPIC_API_KEY is read straight from the environment (set in env: above),
          # so no interactive `setup` step is needed in CI.
          # localhost works because the app is published on the runner host (see networking note above).
          # -c is REQUIRED: never run an unscoped active pentest in CI. The repo is mounted read-only.
          npx @keygraph/shannon start \
            -u http://localhost:3000 \
            -r "$GITHUB_WORKSPACE" \
            -w pr-${{ github.event.pull_request.number }} \
            -c "$GITHUB_WORKSPACE/.github/shannon-ci.yaml"

      - name: Reset / tear down test data
        if: always()
        run: docker compose -f docker-compose.test.yml down -v   # -v drops the throwaway DB volume

      - name: Check for critical findings
        run: |
          # Confirm the actual report filename against the current release if this path changes.
          # npx mode writes workspaces under ~/.shannon/workspaces/<name>/; the final report is at the workspace root.
          REPORT="$HOME/.shannon/workspaces/pr-${{ github.event.pull_request.number }}/Security-Assessment-Report.md"
          if [ ! -f "$REPORT" ]; then
            echo "::error::Security report not found at $REPORT — pentest may have failed. Blocking deploy."
            exit 1
          fi
          # Count severity headings (format: ## [CRITICAL] or ## [HIGH])
          CRITICAL_COUNT=$(grep -c '^##.*\[CRITICAL\]' "$REPORT" || true)
          HIGH_COUNT=$(grep -c '^##.*\[HIGH\]' "$REPORT" || true)
          if [ "$CRITICAL_COUNT" -gt 0 ]; then
            echo "::error::$CRITICAL_COUNT critical findings — review and manually validate the report before merging."
            cat "$REPORT"
            exit 1
          fi
          if [ "$HIGH_COUNT" -gt 0 ]; then
            echo "::warning::$HIGH_COUNT high-severity findings. Manual validation required (findings are leads, not verdicts)."
          fi

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: security-report
          path: ~/.shannon/workspaces/pr-*/Security-Assessment-Report.md

Integration Patterns

Cost Management

These runs are LLM-token-billed, so the dollar cost is whatever your provider charges times the tokens consumed — it moves with model choice and provider pricing and is not a fixed per-run number. Estimate it from the drivers, then read the actual spend off your provider dashboard after the first run and calibrate.

Cost ≈ Σ over agents of (input + output tokens) × model price/token, scaled by retries. The knobs that move tokens:

# Estimate BEFORE a big run: dry-cost a single category on a few endpoints first,
# read the spend from your provider dashboard, then extrapolate:
#   est_full ≈ pilot_cost × (total_endpoints / pilot_endpoints) × (categories / 1)
# Pull current per-token prices from the provider's pricing page — never hardcode them.

Cost-reduction strategies:

1. Narrow scope with CONFIG (focus/avoid rules) — biggest lever.
2. Run single-category scans for targeted, post-change checks.
3. Cap max_agents / max_steps / requests-per-second in shannon.yaml.
4. Use named workspaces to resume interrupted scans instead of paying for a full re-run.
5. Schedule full scans weekly; run quick single-category scans on PRs.

---

6. Post-Pentest Workflow

Triage → Fix → Verify

1. TRIAGE (Day 0)
   ├── Read the full report
   ├── Verify all Critical/High PoCs manually
   ├── Create tickets with severity labels
   ├── Assign owners and deadlines
   └── Notify stakeholders for Critical findings

2. FIX (Day 1-14, based on severity)
   ├── Critical: same day
   ├── High: within 48 hours
   ├── Medium: within 2 weeks
   └── Low: next sprint

3. VERIFY (After fix)
   ├── Re-run Shannon against the same workspace (resume: reuse -w and the same -u URL)
   │   └── npx @keygraph/shannon start -u <url> -r <repo> -w <same-name>
   ├── Completed agents are skipped (resumable)
   ├── Confirm the PoC no longer works
   └── Update ticket status

4. DOCUMENT
   ├── Archive the report
   ├── Update security runbook with new patterns
   ├── Add regression tests for each finding
   └── Schedule next pentest

Regression Testing

For each finding, create a permanent test:

// tests/security/sql-injection.test.ts
describe('SQL Injection regression', () => {
  it('should not be vulnerable to union-based injection in /api/users/search', async () => {
    const res = await request(app)
      .get("/api/users/search")
      .query({ q: "' UNION SELECT username,password,NULL FROM users--" });

    // Should NOT return other users' data
    expect(res.body).not.toEqual(
      expect.arrayContaining([
        expect.objectContaining({ username: 'admin' })
      ])
    );
  });

  it('should use parameterized queries', async () => {
    const res = await request(app)
      .get("/api/users/search")
      .query({ q: "test" });

    expect(res.status).toBe(200);
    // Normal search should still work
  });
});

---

7. What Shannon Doesn't Cover

Supplement with manual testing or other tools:

Complementary Tool Stack

# Run alongside Shannon for full coverage:

# Dependency scanning (production deps only)
npm audit --omit=dev          # `--production` is deprecated; use `--omit=dev`
pnpm audit --prod             # pnpm equivalent
yarn npm audit --environment production   # Yarn Berry (v2+); classic: `yarn audit --groups dependencies`
npx snyk test

# Secret detection
trufflehog git file://. --only-verified

# Container scanning
trivy image myapp:latest

# Infrastructure
prowler aws --severity critical high

# API fuzzing
schemathesis run http://localhost:3000/openapi.json

---

8. Safe Testing Practices

Rules of Engagement

DO:
  ✓ Only test applications you own or have written authorization to test
  ✓ Use staging/test environments, never production
  ✓ Create dedicated test accounts with known credentials
  ✓ Set scope rules to avoid destructive endpoints
  ✓ Review reports before sharing (may contain sensitive data)
  ✓ Keep API keys secure (Shannon uses significant API credits)

DON'T:
  ✗ Point Shannon at production systems
  ✗ Test third-party services without explicit written permission
  ✗ Share reports containing valid credentials or PII
  ✗ Run without scope rules on apps with destructive endpoints
  ✗ Ignore the cost — monitor API spend during runs

Safe-test controls for side-effecting attacks

Some attack classes have real-world blast radius even in staging. Apply these controls before enabling them — and prefer the scope avoid rules in §5 when in doubt.

Pre-run assertions (fail closed). Bake these checks into the test harness so a misconfigured target aborts the run instead of doing damage:

# Refuse to run unless we're clearly NOT in production
[ "$NODE_ENV" = "test" ] || { echo "Refusing: NODE_ENV is not 'test'"; exit 1; }
case "$TARGET_URL" in *prod*|*www.*) echo "Refusing: target looks like production"; exit 1;; esac
# Cloud metadata must be unreachable from the test host before SSRF probing
curl -s --max-time 2 http://169.254.169.254/ >/dev/null \
  && { echo "Refusing: cloud metadata endpoint is reachable from test host"; exit 1; } || true

Test Environment Setup

# docker-compose.test.yml — isolated test environment
services:
  app:
    build: .
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgres://test:test@db:5432/testdb
    ports:
      - "3000:3000"
    networks:
      - pentest-net

  db:
    image: postgres:16
    environment:
      - POSTGRES_DB=testdb
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
    networks:
      - pentest-net

networks:
  pentest-net:
    driver: bridge
    # Isolated network — no access to host or internet