Krawl Architecture¶
Overview¶
Krawl is a cloud-native deception honeypot server built on FastAPI. It creates realistic fake web applications (admin panels, login pages, fake credentials) to attract, detect, and analyze malicious crawlers and attackers while wasting their resources with infinite spider-trap pages.
Tech Stack¶
| Layer | Technology |
|---|---|
| Backend | FastAPI, Uvicorn, Python 3.11 |
| ORM / DB | SQLAlchemy 2.0, SQLite (WAL mode) or PostgreSQL |
| Templating | Jinja2 (server-side rendering) |
| Reactivity | Alpine.js 3.14 |
| Partial Updates | HTMX 2.0 |
| Charts | Chart.js 3.9 (attack trends), custom SVG (map radar, IP hexagon) |
| Maps | Leaflet 1.9 + configurable raster tiles (Esri dark canvas by default) |
| Scheduling | APScheduler |
| Container | Docker (python:3.11-slim), Helm/K8s ready |
Directory Structure¶
Krawl/
├── src/
│ ├── app.py # FastAPI app factory + lifespan
│ ├── config.py # YAML + env config loader
│ ├── dependencies.py # DI providers (templates, DB, client IP)
│ ├── models.py # SQLAlchemy ORM models
│ ├── tracker.py # In-memory + DB access tracking
│ ├── logger.py # Rotating file log handlers
│ ├── auth_store.py # Dashboard sessions + bruteforce counters (Redis / in-process)
│ ├── dashboard_cache.py # Cache backend (Redis in scalable, dict in standalone)
│ ├── banlist_sync.py # Fetch + merge upstream community banlists
│ ├── webhooks.py # Outbound integrations (CloudFlare WAF / IP lists)
│ ├── metrics.py # Prometheus collector
│ ├── metrics_counters.py # Counter state shared with the flush task
│ ├── detection.py # YARA attack detection, startup validation
│ ├── custom_rules.py # Rule files uploaded from the dashboard (validate, store, hot reload)
│ ├── responses.py # Fake replies and bait pages from responses/manifest.yaml
│ ├── generative_ai.py # AI deception page generation
│ ├── sanitizer.py # Input sanitization
│ ├── geo_utils.py # IP geolocation API
│ ├── ip_utils.py # IP validation + ignored IP/CIDR matching
│ │
│ ├── database/ # DatabaseManager singleton, split by domain
│ │ ├── core.py # Engine, session, write buffer
│ │ ├── startup.py # Startup cleanup (purge ignored IPs, expire bans)
│ │ ├── access_logs.py credentials.py generated_pages.py
│ │ ├── ip_stats.py analytics.py
│ │ └── __init__.py # Public API re-exports
│ │
│ ├── routes/
│ │ ├── honeypot.py # Trap pages, credential capture, catch-all
│ │ ├── dashboard.py # Dashboard page (Jinja2 SSR)
│ │ ├── api.py # JSON API endpoints
│ │ ├── banlist.py # Public unauthenticated banlist download
│ │ └── htmx.py # HTMX HTML fragment endpoints
│ │
│ ├── middleware/
│ │ ├── deception.py # Path traversal / XXE / cmd injection detection
│ │ └── ban_check.py # Banned IP enforcement
│ │
│ ├── tasks/ # APScheduler background jobs (auto-discovered)
│ │ ├── analyze_ips.py # IP categorization scoring (docs/algorithms.md)
│ │ ├── fetch_ip_rep.py # Geolocation + blocklist enrichment
│ │ ├── flag_stale_ips.py # Flag stale IPs for reevaluation
│ │ ├── flush_access_logs.py # Drain the access-log write buffer
│ │ ├── metrics_flush.py # Persist metric counters
│ │ ├── dashboard_warmup.py # Pre-compute dashboard data
│ │ ├── refresh_banlist.py # Fetch + merge upstream banlists
│ │ ├── sync_cloudflare.py # Push banned IPs to a CloudFlare IP list
│ │ ├── db_dump.py # Database export
│ │ ├── pre_retention_cleanup.py # Prune non-suspicious rows before retention
│ │ └── db_retention.py # Data retention cleanup
│ │
│ ├── tasks_master.py # Task discovery + APScheduler orchestrator
│ ├── firewall/ # Banlist export formats (raw, iptables, nftables)
│ ├── migrations/ # Schema migrations (auto-run at startup)
│ │
│ └── templates/
│ ├── jinja2/
│ │ ├── base.html # Layout + CDN scripts
│ │ └── dashboard/
│ │ ├── index.html # Main dashboard page
│ │ └── partials/ # HTMX fragment templates
│ ├── html/ # Deceptive trap page templates
│ ├── deception/ # Path-mapped deception pages (auto-imported)
│ └── static/
│ ├── css/dashboard.css
│ └── js/
│ ├── tokens.js # Design tokens shared with the CSS
│ ├── dashboard.js # Alpine.js app controller
│ ├── map.js # Leaflet map
│ └── charts.js # Chart.js attack trends
│
├── config.yaml # Application configuration
├── rules/ # Startup-compiled YARA request rules
├── responses/ # All fake content: attack replies, bait pages, trap shell
├── Dockerfile # Container build
├── docker/ # Production compose files (dev/ for source builds)
├── entrypoint.sh # Container startup (gosu privilege drop)
├── kubernetes/ # K8s manifests
└── helm/ # Helm chart
Application Entry Point¶
src/app.py uses the FastAPI application factory pattern with an async lifespan manager:
Startup Shutdown
│ │
├─ Initialize logging └─ Log shutdown
├─ Initialize database (SQLite or PostgreSQL)
├─ Initialize cache (in-memory or Redis)
├─ Resolve server public IP (once)
├─ Create AccessTracker
├─ Store config + tracker in app.state
├─ Start APScheduler background tasks
└─ Log dashboard URL
Request Pipeline¶
Request
│
▼
┌──────────────────────┐
│ BanCheckMiddleware │──→ IP banned? → Return 500
└──────────┬───────────┘
▼
┌──────────────────────┐
│ DeceptionMiddleware │──→ Attack detected? → Fake error response
└──────────┬───────────┘
▼
┌───────────────────────┐
│ ServerHeaderMiddleware│──→ Add random Server header
└──────────┬────────────┘
▼
┌───────────────────────┐
│ Route Matching │
│ (ordered by priority)│
│ │
│ 1. Static files │ /{secret}/static/*
│ 2. Dashboard router │ /{secret}/ (prefix-based)
│ 3. API router │ /{secret}/api/* (prefix-based)
│ 4. HTMX router │ /{secret}/htmx/* (prefix-based)
│ 5. Honeypot router │ /* (catch-all)
└───────────────────────┘
Prefix-Based Routing¶
Dashboard, API, and HTMX routers are mounted with prefix=f"/{secret}" in app.py. This means:
- Route handlers define paths without the secret (e.g., @router.get("/api/all-ips"))
- FastAPI prepends the secret automatically (e.g., GET /a1b2c3/api/all-ips)
- The honeypot catch-all /{path:path} only matches paths that don't start with the secret
- No _is_dashboard_path() checks needed — the prefix handles access scoping
Route Architecture¶
Honeypot Routes (routes/honeypot.py)¶
| Method | Path | Response |
|---|---|---|
GET |
/{path:path} |
Trap page with random links (catch-all) |
HEAD |
/{path:path} |
200 OK |
POST |
/{path:path} |
Credential capture |
GET |
/admin, /login |
Fake login form |
GET |
/wp-admin, /wp-login.php |
Fake WordPress login |
GET |
/phpmyadmin |
Fake phpMyAdmin |
GET |
/robots.txt |
Honeypot paths advertised |
GET/POST |
/api/search, /api/sql |
SQL injection honeypot |
POST |
/api/contact |
XSS detection endpoint |
GET |
/.env, /credentials.txt |
Fake sensitive files |
Dashboard Routes (routes/dashboard.py)¶
| Method | Path | Response |
|---|---|---|
GET |
/ |
Server-rendered dashboard (Jinja2) |
API Routes (routes/api.py)¶
| Method | Path | Response |
|---|---|---|
GET |
/api/all-ips |
Paginated IP list with stats |
GET |
/api/attackers |
Paginated attacker IPs |
GET |
/api/ip-stats/{ip} |
Single IP detail |
GET |
/api/credentials |
Captured credentials |
GET |
/api/honeypot |
Honeypot trigger counts |
GET |
/api/top-ips |
Top requesting IPs |
GET |
/api/top-paths |
Most requested paths |
GET |
/api/top-user-agents |
Top user agents |
GET |
/api/attack-types-stats |
Attack type distribution |
GET |
/api/attack-types |
Paginated attack log |
GET |
/api/raw-request/{id} |
Full HTTP request |
GET |
/api/export-ips |
Export IPs for firewall integration |
POST |
/api/rules/upload |
Validate and load an uploaded YARA rule file (authenticated) |
GET |
/api/rules/{filename} |
Download an uploaded rule file (authenticated) |
DELETE |
/api/rules/{filename} |
Delete an uploaded rule file and unload its rules (authenticated) |
HTMX Fragment Routes (routes/htmx.py)¶
Each returns a server-rendered Jinja2 partial (hx-swap="innerHTML"):
| Path | Template |
|---|---|
/htmx/honeypot |
honeypot_table.html |
/htmx/top-ips |
top_ips_table.html |
/htmx/top-paths |
top_paths_table.html |
/htmx/top-ua |
top_ua_table.html |
/htmx/attackers |
attackers_table.html |
/htmx/credentials |
credentials_table.html |
/htmx/attacks |
attack_types_table.html |
/htmx/patterns |
patterns_table.html |
/htmx/ip-detail/{ip} |
ip_detail.html |
/htmx/rules |
rules_panel.html (authenticated) |
/htmx/rules/table |
rules_table.html (authenticated) |
Database Schema¶
┌─────────────────┐ ┌──────────────────┐
│ AccessLog │ │ AttackDetection │
├─────────────────┤ ├──────────────────┤
│ id (PK) │◄────│ access_log_id(FK)│
│ ip (indexed) │ │ attack_type │
│ path │ │ matched_pattern │
│ user_agent │ │ rule_name │
│ │ └──────────────────┘
│ method │
│ is_suspicious │ ┌──────────────────┐
│ is_honeypot │ │CredentialAttempt │
│ timestamp │ ├──────────────────┤
│ raw_request │ │ id (PK) │
└─────────────────┘ │ ip (indexed) │
│ path, username │
┌─────────────────┐ │ password │
│ IpStats │ │ timestamp │
├─────────────────┤ └──────────────────┘
│ ip (PK) │
│ total_requests │ ┌──────────────────┐
│ first/last_seen │ │ CategoryHistory │
│ country_code │ ├──────────────────┤
│ city, lat, lon │ │ id (PK) │
│ asn, asn_org │ │ ip (indexed) │
│ isp, reverse │ │ old_category │
│ is_proxy │ │ new_category │
│ is_hosting │ │ timestamp │
│ list_on (JSON) │ └──────────────────┘
│ category │
│ category_scores │
│ analyzed_metrics│
│ manual_category │ ┌──────────────────┐
└─────────────────┘ │ CustomYaraRule │
├──────────────────┤
│ filename (PK) │
│ source │
│ rule_names │
│ uploaded_at │
└──────────────────┘
SQLite config (standalone): WAL mode, 30s busy timeout, file permissions 600.
PostgreSQL config (scalable): Connection pool (10 + 20 overflow), pre-ping enabled, connections recycled every 30 minutes.
Frontend Architecture¶
base.html
├── CDN: Leaflet, Chart.js, HTMX, Alpine.js (deferred)
├── Static: dashboard.css
│
└── dashboard/index.html (extends base)
│
├── Stats cards ──────────── Server-rendered on page load
├── Suspicious table ─────── Server-rendered on page load
│
├── Overview tab (Alpine.js x-show)
│ ├── Honeypot table ───── HTMX hx-get on load
│ ├── Top IPs table ────── HTMX hx-get on load
│ ├── Top Paths table ──── HTMX hx-get on load
│ ├── Top UA table ─────── HTMX hx-get on load
│ └── Credentials table ── HTMX hx-get on load
│
└── Attacks tab (Alpine.js x-show, lazy init)
├── Attackers table ──── HTMX hx-get on load
├── Map ──────────────── Leaflet (init on tab switch)
├── Chart ────────────── Chart.js (init on tab switch)
├── Attack types table ─ HTMX hx-get on load
└── Patterns table ───── HTMX hx-get on load
Responsibility split: - Alpine.js — Tab state, modals, dropdowns, lazy initialization - HTMX — Table pagination, sorting, IP detail expansion - Leaflet — Interactive map with category-colored markers - Chart.js — Attack trends line chart - Custom SVG — Map popup radar and the IP Insight behaviour hexagon, server-rendered
Background Tasks¶
Managed by TasksMaster (APScheduler). Tasks are auto-discovered from src/tasks/.
Names match Settings → Maintenance, where any task can be run on demand. In scalable mode, "one pod" tasks run on a single pod per occurrence; see Deployment modes.
| Task | Schedule | Runs on | Purpose |
|---|---|---|---|
flush-access-logs |
Every 30 s (scalable only) | Every pod | Drain the in-memory access-log write buffer into the database |
refresh-ban-cache |
Every 30 s | Every pod | Reload this pod's in-memory set of banned IPs |
reload-custom-rules |
Every 30 s | Every pod | Load YARA rule files uploaded or deleted on another replica |
analyze-ips |
Every 1 min | One pod | Score IPs into categories (attacker, crawler, user) |
extract-request-metadata |
Every 1 min | One pod | Backfill target hosts and URL assets from stored raw requests |
extract-captured-files |
Every 1 min | One pod | Backfill captured files from stored raw requests |
hash-payloads |
Every 1 min | One pod | TLSH-hash new payloads and cluster them into campaigns |
sync-cloudflare |
Every 1 min | One pod | Push banned IPs to a Cloudflare IP list (opt-in) |
metrics-flush |
Every 2 min | Every pod | Persist Prometheus counter state |
fetch-ip-rep |
Every 5 min | One pod | Enrich IPs with geolocation and blocklist data |
dashboard-warmup |
Every 5 min | One pod | Pre-compute dashboard data (disable with cache_warmup: false) |
build-online-indexes |
Hourly at :17 | One pod | Build PostgreSQL indexes concurrently, off the startup path |
rollup-domains |
Hourly at :23 | One pod | Keep per-day Targeted Domains totals in Redis |
refresh-banlist |
Every 1 h | One pod | Fetch and merge upstream community banlists |
prune-uninteresting |
Every 4 h | One pod | Delete random-path crawler GETs past retention.uninteresting |
dump-krawl-data |
backups.cron |
One pod | Export database backups (opt-in) |
db-cleanup |
Daily 03:00 | One pod | Purge ignored IPs (and IPv6 rows if configured), clear expired bans |
flag-stale-ips |
Daily 08:00 | One pod | Flag stale IPs for re-evaluation by the analyzer |
pre-retention-cleanup |
Daily 08:30 | One pod | Re-check old suspicious logs against current rules |
db-retention |
Daily 09:00 | One pod | Delete records past their retention window |
purge |
Never scheduled | Serving pod | Database purge, run from the dashboard maintenance panel |
Each sweeping task takes a bounded slice per run and picks the rest up next
run. The slice sizes live in the tasks: config section (see Configuration) —
raise them when a backlog never drains, keeping the pod's memory limit in mind.
IP Categorization Model¶
Each IP is scored across 4 categories based on: - HTTP method distribution (risky methods ratio) - Robots.txt violations - Request timing anomalies (coefficient of variation) - User-Agent diversity - Attack URL detection
Categories: attacker, bad_crawler, good_crawler, regular_user, unknown
Configuration¶
config.yaml with environment variable overrides (KRAWL_{FIELD}):
mode: standalone # "standalone" or "scalable"
server:
port: 5000
delay: 100 # Response delay (ms)
dashboard:
secret_path: "test" # Auto-generates if null
cache_warmup: true # Background cache refresh (optional in scalable mode)
database:
path: "data/krawl.db"
retention:
non-suspicious: 30d
recheck-suspicious: 30d
re-evaluate-interval: 5d
uninteresting: 7d
overall: null
# Scalable mode only
postgres:
host: "localhost"
port: 5432
redis:
host: "localhost"
port: 6379
cache_ttl: 600 # Dashboard warmup data TTL
hot_ttl: 30 # Ban info / IP category TTL
table_ttl: 120 # Paginated table TTL
tasks: # Rows each scheduled task handles per run
analyze_ips_batch: 2000
extract_metadata_batch: 500
extract_files_batch: 500
hash_payloads_batch: 500
hash_payloads_chunk: 50 # Raw requests held in memory at once
hash_files_batch: 50
cleanup_batch: 1000
retention_delete_batch: 10000
crawl:
infinite_pages_for_malicious: true
max_pages_limit: 250
ban_duration_seconds: 600
behavior:
probability_error_codes: 0 # 0-100%
canary:
token_url: null # External canary alert URL
Logging¶
Three rotating log files (1MB max, 5 backups each):
| Logger | File | Content |
|---|---|---|
krawl.app |
logs/krawl.log |
Application events, errors |
krawl.access |
logs/access.log |
HTTP access, attack detections |
krawl.credentials |
logs/credentials.log |
Captured login attempts |
Docker¶
FROM python:3.11-slim
# Non-root user: krawl:1000
# Volumes: /app/logs, /app/data, /app/exports
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "5000", "--app-dir", "src", "--no-server-header"]
Key Data Flows¶
Honeypot Request¶
Client → BanCheck → DeceptionMiddleware → HoneypotRouter
│
┌─────────┴──────────┐
│ tracker.record() │
│ ├─ in-memory ++ │
│ ├─ detect attacks │
│ └─ DB persist │
└────────────────────┘
Dashboard Load¶
Browser → GET /{secret}/ → SSR initial stats + Jinja2 render
→ Alpine.js init → HTMX fires hx-get for each table
→ User clicks Attacks tab → setTimeout → init Leaflet + Chart.js
→ Leaflet fetches /api/all-ips → plots markers
→ Chart.js renders attack trends