CFTS Internal

Caddy Reverse Proxy Operations

Download Markdown

Last updated: 2026-05-27

This document tracks the operating workflow for the CFTS reverse proxy at rproxy.cfts.co.

Source Layout

Local workspace:

F:\Avery_Vault_LLM\07-Projects\CFTS\Platforms\rproxy.cfts.co

Primary files:

Project-Documentation/00. rproxy Caddy Overview For New Admins.md
Project-Documentation/03. rproxy Function and Feature Synopsis.md
caddy/Caddyfile
caddy/includes/00-common.caddy
caddy/errors/{403,404,500}.html
observability/install-caddy-goaccess-report.sh
observability/goaccess/render-caddy-goaccess-report.sh
observability/systemd/caddy-goaccess-report.{service,timer}
security/fail2ban/filter.d/*.conf
security/fail2ban/jail.d/caddy-rproxy.local
security/apt/apt.conf.d/*
Project-Documentation/02. rproxy Exposure Matrix.md

Production layout on rproxy.cfts.co:

/etc/caddy/Caddyfile
/etc/caddy/includes/00-common.caddy
/etc/caddy/errors/{403,404,500}.html
/usr/local/sbin/render-caddy-goaccess-report
/etc/systemd/system/caddy-goaccess-report.{service,timer}
/var/www/caddy-log-report/index.html

Caddy Include Model

The main Caddyfile imports reusable snippets near the top:

import includes/*.caddy

The common include defines:

json_access_log
safe_headers
lose_headers_no_csp
strip_response_identity
strip_spoofed_request_headers
strip_upstream_security_headers
no_uploads_expected
error_page_headers
default_error_pages
safe_tls
vsphere_tls
public_site_base

Use public_site_base only for simple public sites that can tolerate strict framing and CSP. Do not apply it blindly to dashboards, ticket systems, consoles, or legacy apps without browser testing.

json_access_log is the standard dedicated access-log block. Import it with the target log path instead of pasting another full log { output file ... } block.

strip_response_identity removes common software-identifying response headers such as Server, Via, X-Powered-By, and framework/version breadcrumbs. It is imported through safe_tls for the hostnames that use the shared TLS profile; the ESXi edge hostnames remain in a minimum-interference proxy mode.

strip_spoofed_request_headers removes client-supplied proxy identity headers such as Forwarded, X-Forwarded-*, X-Real-IP, and common CDN/client-IP hints before requests reach upstream apps. Caddy still adds its own trusted X-Forwarded-* values inside reverse_proxy, and site-specific header_up rules such as X-Real-IP {remote_host} still apply after the scrub.

strip_upstream_security_headers is used inside selected reverse_proxy blocks when Caddy owns the public header policy and the upstream app's own security headers would conflict with that policy.

no_uploads_expected caps request bodies at 1MB for sites where browser-based uploads are not expected. It is currently applied to docs.cfts.co, isp-status.cfts.co, and monitor.cfts.co. The Caddy request_body directive requires Caddy v2.10.0 or newer; if caddy validate rejects it, update Caddy or remove this import before reload.

vsphere_tls keeps the same TLS protocol/cipher posture as safe_tls and still imports strip_response_identity, but deliberately does not import strip_spoofed_request_headers. Keep it reserved for ESXi/vSphere testing only; as of 2026-05-26, edge-01.cfts.co and edge-02.cfts.co do not import any header/TLS protection snippets because the ESXi Host Client is being debugged in a minimum-interference proxy mode.

For ESXi, Caddy now applies a source-IP gate before proxying: only clients in 172.16.198.0/24 reach edge-01.cfts.co and edge-02.cfts.co; everyone else receives 403 Access restricted. Add trusted VPN or static admin CIDRs to the per-site @admin_clients matcher if remote admin access is required.

The only active ESXi request-header change is a compatibility scrub on /sdk* and /screen*: Caddy removes Authorization before proxying those paths. It prevents stale browser-cached Caddy Basic Auth credentials from being forwarded into VMware SOAP/screen calls after the earlier Caddy Basic Auth test.

The strict profile also sends Permissions-Policy with browser features denied by default.

The strict CSP in safe_headers intentionally avoids script-src 'unsafe-inline', style-src 'unsafe-inline', data: image sources, and inline script/style attributes. The only inline script currently allowed is the static JSON-LD block on the docs homepage, pinned by SHA-256 hash. If that JSON-LD changes, recompute the hash and update caddy/includes/00-common.caddy.

Default Error Pages

The default_error_pages snippet serves shared HTML pages for Caddy-generated 403, 404, and 5xx errors from:

/etc/caddy/errors

Source workspace copies live under:

caddy/errors/403.html
caddy/errors/404.html
caddy/errors/500.html

The snippet is imported by every active HTTPS site block. LAN-only deny paths use error 403 rather than respond "Access restricted" 403, so they render the shared 403.html page while preserving the 403 status code.

Caddy handle_errors does not rewrite ordinary upstream responses from reverse_proxy. If an upstream application returns its own 404 or 500, Caddy passes that response through unless a site-specific reverse_proxy handle_response rule is added deliberately.

Verified on 2026-05-26: an external request to https://download.cfts.co/ displayed the shared 403 Access restricted page. This confirms the useful scope of the POC: cleaner Caddy-side denial and fallback pages, not replacement of upstream app error pages.

Known test caveats:

  • Testing LAN-only hostnames from rproxy.cfts.co or the trusted LAN can return upstream 200, because those clients are allowed by the client_ip gate.
  • https://isp-status.cfts.co/opps from the WAN returns 401 when it falls into the Basic Auth handler; that is expected and does not exercise the shared 404.
  • https://docs.cfts.co/surely-this-does-not-exist returns the docs app's upstream 404; that is also expected because Caddy passes proxied responses through.

Use an external client against LAN-only hostnames such as download.cfts.co, console.dns.cfts.co, ai.cfts.co, or inventory.cfts.co to verify the shared 403 page.

Explicit HTTP Routing / Unknown Host Catch-All

The Caddyfile sets auto_https disable_redirects so Caddy keeps automatic certificate management but does not emit broad HTTP-to-HTTPS redirects before site routing. A single http:// block then redirects only known hostnames to HTTPS and closes unknown plaintext HTTP hostnames with abort.

The http:// block imports strip_response_identity, logs unknown hosts to /var/log/caddy/unknown-host-access.log, and matches known hostnames with @known_hosts. log_skip @known_hosts keeps normal HTTP-to-HTTPS redirects out of the unknown-host log so Fail2Ban can treat every logged line there as bad-host evidence. Add new public hostnames to that matcher when adding new HTTPS site blocks.

Do not add a broad https:// catch-all unless wildcard or on-demand TLS is intentionally designed first. Unknown HTTPS names should fail before HTTP routing if Caddy has no matching certificate; accepting arbitrary HTTPS names would widen the TLS surface.

As of 2026-05-27, @known_hosts includes the current active hostnames edge-01.cfts.co, edge-02.cfts.co, inventory.cfts.co, and rp-logs.cfts.co. It also retains older vps.cfts.co and hvps.cfts.co names for compatibility/history.

Global Server Guardrails

The global options block applies listener-level guardrails:

servers :80 {
    timeouts {
        read_header 10s
        idle 2m
    }
    max_header_size 64KB
    protocols h1
}

servers :443 {
    timeouts {
        read_header 10s
        idle 2m
    }
    max_header_size 64KB
    protocols h1 h2
    strict_sni_host on
    0rtt off
}

This limits slow header reads, reduces idle connection dwell time, caps oversized request headers, disables HTTP/3, rejects TLS SNI/Host mismatches, and disables TLS early data.

After the protocols h1 h2 change is live and verified, remove the no-longer-needed public UDP exposure:

sudo ufw delete allow 443/udp
sudo ufw status verbose

SFTP Deployment Workflow

SFTP does not write directly to /etc/caddy. Stage files under:

/home/sysops/temp

Then SSH into rproxy.cfts.co and copy with sudo:

sudo mkdir -p /etc/caddy/includes /etc/caddy/errors
sudo cp /home/sysops/temp/Caddyfile /etc/caddy/Caddyfile
sudo cp /home/sysops/temp/includes/00-common.caddy /etc/caddy/includes/00-common.caddy
sudo cp /home/sysops/temp/errors/*.html /etc/caddy/errors/
sudo chown root:root /etc/caddy/Caddyfile /etc/caddy/includes/00-common.caddy /etc/caddy/errors/*.html
sudo chmod 644 /etc/caddy/Caddyfile /etc/caddy/includes/00-common.caddy /etc/caddy/errors/*.html

Access Log Permissions

Dedicated log files must be writable by the caddy service user before reload. Example for docs.cfts.co:

sudo install -d -o caddy -g caddy -m 0750 /var/log/caddy
sudo touch /var/log/caddy/docs-access.log
sudo chown caddy:caddy /var/log/caddy/docs-access.log
sudo chmod 0640 /var/log/caddy/docs-access.log

For edge-01.cfts.co and edge-02.cfts.co, the dedicated logs currently retain the older vps and hvps names:

sudo touch /var/log/caddy/vps-access.log /var/log/caddy/hvps-access.log
sudo chown caddy:caddy /var/log/caddy/vps-access.log /var/log/caddy/hvps-access.log
sudo chmod 0640 /var/log/caddy/vps-access.log /var/log/caddy/hvps-access.log

Other public app logs currently used by the Caddyfile:

sudo touch /var/log/caddy/tickets-access.log /var/log/caddy/redmine-access.log /var/log/caddy/tracks-access.log /var/log/caddy/monitor-access.log /var/log/caddy/unknown-host-access.log /var/log/caddy/inventory.log /var/log/caddy/rp-logs-access.log
sudo chown caddy:caddy /var/log/caddy/tickets-access.log /var/log/caddy/redmine-access.log /var/log/caddy/tracks-access.log /var/log/caddy/monitor-access.log /var/log/caddy/unknown-host-access.log /var/log/caddy/inventory.log /var/log/caddy/rp-logs-access.log
sudo chmod 0640 /var/log/caddy/tickets-access.log /var/log/caddy/redmine-access.log /var/log/caddy/tracks-access.log /var/log/caddy/monitor-access.log /var/log/caddy/unknown-host-access.log /var/log/caddy/inventory.log /var/log/caddy/rp-logs-access.log

If reload status mentions opening log writer and permission denied, create and chown the named log file before reloading again.

Log Report GUI

rp-logs.cfts.co is a LAN-only HTTPS site that serves a static GoAccess report from:

/var/www/caddy-log-report/index.html

The report turns Caddy JSON access logs into a browser-readable dashboard with request volume, status codes, paths, referrers, clients, and other useful summaries. It is intentionally static and refreshed by a systemd timer every five minutes rather than running a separate long-lived web service.

Deploy or update the report generator. Upload the whole observability folder to /home/sysops/temp/observability, then run these commands one line at a time on rproxy.cfts.co:

sudo apt update

sudo apt install goaccess

sh -n /home/sysops/temp/observability/install-caddy-goaccess-report.sh

sh -n /home/sysops/temp/observability/goaccess/render-caddy-goaccess-report.sh

sudo sh /home/sysops/temp/observability/install-caddy-goaccess-report.sh

Check it:

systemctl status caddy-goaccess-report.service --no-pager
systemctl list-timers caddy-goaccess-report.timer
ls -lh /var/www/caddy-log-report/index.html

Open from the LAN:

https://rp-logs.cfts.co/

The systemd service runs as caddy, so Caddy log files must remain readable by the caddy group.

The installer copies the renderer to /usr/local/sbin/render-caddy-goaccess-report, installs the systemd unit and timer, prepares /var/www/caddy-log-report, prepares /var/log/caddy/rp-logs-access.log, smoke-tests GoAccess when a non-empty Caddy log exists, starts one report render, and enables the five-minute timer.

For the full paste-safe runbook and troubleshooting notes, use observability/README.md.

Do not use goaccess --help | grep CADDY as the compatibility check; some packages do not list every predefined format in short help. The useful check is whether goaccess "$log" --log-format=CADDY -o /tmp/goaccess-caddy-smoke.html can parse a real Caddy JSON access log and create a non-empty HTML file. If that command reports an unknown CADDY log format, install a newer GoAccess package before enabling the timer. The report generator uses GoAccess' built-in Caddy JSON parser.

Basic Auth Hashes

For public admin-style surfaces, prefer a Caddy basic_auth layer in front of upstream authentication.

Generate a bcrypt hash with Caddy:

caddy hash-password --plaintext 'new password here'

Store only the generated hash in the Caddyfile:

basic_auth {
    vps_acces <bcrypt-hash>
}

On 2026-05-15, Caddy Basic Auth was tested on the ESXi hostnames then in use, vps.cfts.co and hvps.cfts.co. It correctly returned 401 Unauthorized before credentials, but it triggered vSphere web client errors after login. Basic Auth was removed live.

On 2026-05-17, the browser still showed cached vps_access Basic Auth prompts while the ESXi Host Client raised JavaScript errors around missing VM/host state. A short-lived Caddyfile test stripped stale Caddy Basic Auth credentials and forced the upstream Host header to the direct ESXi IP, but the ESXi Host Client error persisted.

Access logs then showed POST /sdk/ returning 500 and GET /screen?... returning 401 with Www-Authenticate: Basic realm="VMware HTTP server", while the browser was still sending an Authorization header. The current diagnostic fix strips Authorization only for /sdk* and /screen* on edge-01.cfts.co and edge-02.cfts.co.

These are vSphere 7.0.2 / 8 admin-console surfaces and can be cranky behind a reverse proxy. Keep them in minimum-interference mode while debugging: source-IP allowlisting is active, but avoid adding a Caddy header profile, custom TLS snippet, upstream Host override, Caddy auth, broad request-header changes, or request-body caps without testing. If hardening is reintroduced later, add exactly one change at a time and browser-test the ESXi UI before adding the next.

Future access-control options to consider for these hostnames:

  • Cloudflare Access / Zero Trust in front of Caddy.
  • VPN-first access by adding VPN/admin CIDRs to the existing Caddy allowlist.
  • Conditional allowlist expansion for known static admin networks where practical.
  • Continue relying on upstream vSphere auth, with logging and strong upstream account policy, if roaming clients make allowlisting impractical.

Decision for monitor.cfts.co: keep it public. It fronts PRTG, and the business value is being able to check monitoring from offsite. Do not make it LAN-only as a generic hardening move.

For monitor.cfts.co, avoid reflexively adding Caddy Basic Auth if it would interfere with PRTG workflows. Current Caddy posture is public reachability, lose_headers_no_csp, safe_tls, no_uploads_expected, dedicated JSON logging, and upstream PRTG authentication as the access-control layer. If this changes later, prefer a deliberate PRTG-tested pattern such as path-specific protection or a known admin allowlist.

Fail2Ban

Fail2Ban is enabled on rproxy.cfts.co as a host-level hardening layer for SSH.

Verified on 2026-05-15:

fail2ban.service: active (running), enabled
active jail: sshd
SSH port: 4422
backend: systemd
current bans: 0

Local override file:

/etc/fail2ban/jail.d/sshd.local

Expected SSH jail shape:

[sshd]
enabled = true
port = 4422
backend = systemd
maxretry = 5
findtime = 10m
bantime = 1h

Quick checks:

sudo systemctl status fail2ban --no-pager
sudo fail2ban-client status
sudo fail2ban-client status sshd
sudo grep -R "^[[:space:]]*port" /etc/fail2ban/jail.d /etc/fail2ban/jail.local /etc/fail2ban/jail.conf

Caddy log banning is configured in the source workspace under:

security/fail2ban/filter.d/caddy-unknown-host.conf
security/fail2ban/filter.d/caddy-scanner-paths.conf
security/fail2ban/jail.d/caddy-rproxy.local

caddy-unknown-host watches /var/log/caddy/unknown-host-access.log. Because known-host redirects are skipped with log_skip @known_hosts, each logged line represents an unknown plaintext HTTP host.

caddy-scanner-paths watches /var/log/caddy/*-access.log for obvious opportunistic scanner paths such as /.env, /.git, /wp-login.php, /phpmyadmin, /cgi-bin, /server-status, /actuator, and similar probes.

Deploy:

sudo cp /home/sysops/temp/security/fail2ban/filter.d/caddy-unknown-host.conf /etc/fail2ban/filter.d/caddy-unknown-host.conf
sudo cp /home/sysops/temp/security/fail2ban/filter.d/caddy-scanner-paths.conf /etc/fail2ban/filter.d/caddy-scanner-paths.conf
sudo cp /home/sysops/temp/security/fail2ban/jail.d/caddy-rproxy.local /etc/fail2ban/jail.d/caddy-rproxy.local
sudo fail2ban-client reload
sudo fail2ban-client status caddy-unknown-host
sudo fail2ban-client status caddy-scanner-paths

Check filters against live logs before or after reload:

sudo fail2ban-regex /var/log/caddy/unknown-host-access.log /etc/fail2ban/filter.d/caddy-unknown-host.conf
sudo fail2ban-regex /var/log/caddy/docs-access.log /etc/fail2ban/filter.d/caddy-scanner-paths.conf

Current jail thresholds are conservative:

caddy-unknown-host: 20 hits in 10 minutes -> 1 hour ban
caddy-scanner-paths: 5 hits in 10 minutes -> 4 hour ban
ignoreip: 127.0.0.1/8, ::1, 172.16.198.0/24

Keep web jails conservative to avoid blocking legitimate public users.

Unattended Security Updates

Security update automation is configured in the source workspace under:

security/apt/apt.conf.d/20auto-upgrades
security/apt/apt.conf.d/52unattended-upgrades-cfts-security

This enables daily package list updates and unattended Ubuntu security upgrades, removes unused dependencies, and does not automatically reboot.

Deploy and dry-run:

sudo apt update
sudo apt install unattended-upgrades apt-listchanges
sudo cp /home/sysops/temp/security/apt/apt.conf.d/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades
sudo cp /home/sysops/temp/security/apt/apt.conf.d/52unattended-upgrades-cfts-security /etc/apt/apt.conf.d/52unattended-upgrades-cfts-security
sudo unattended-upgrade --dry-run --debug
systemctl status apt-daily.timer apt-daily-upgrade.timer --no-pager

Useful checks:

grep -R "Unattended-Upgrade::Automatic-Reboot" /etc/apt/apt.conf.d
sudo tail -n 100 /var/log/unattended-upgrades/unattended-upgrades.log
apt list --upgradable

If systemctl reload caddy appears stuck after a failed reload, check:

systemctl status caddy --no-pager

A clean active (running) state after restart means recovery succeeded.

Validate And Reload

Always validate before reload:

sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
systemctl status caddy --no-pager

If reload does not settle after a previous failed reload, restart instead:

sudo systemctl restart caddy
systemctl status caddy --no-pager

Current Good Verification

On 2026-05-15, docs.cfts.co was verified through Caddy after the include split:

/ returned 200
/static/og-image.png returned 200
/var/log/caddy/docs-access.log received JSON access entries
Strict-Transport-Security, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, and Content-Security-Policy were present

On 2026-05-15, after moving docs JavaScript to a static file, tightening CSP, and adding Permissions-Policy, SecurityHeaders reported A+ for:

https://docs.cfts.co/

Evidence screenshot:

Project-Documentation/Evidence/securityheaders-docs-cfts-co-a-plus-2026-05-15.png

Useful public test links:

https://securityheaders.com/?q=https%3A%2F%2Fdocs.cfts.co%2F&followRedirects=on
https://www.ssllabs.com/ssltest/analyze.html?d=docs.cfts.co
https://developer.mozilla.org/en-US/observatory/analyze?host=docs.cfts.co
https://internet.nl/site/docs.cfts.co/

The Pentest-Tools website scanner previously reported only low findings, but continued to flag CSP heuristically after the live header was tightened. Treat SecurityHeaders, live curl -I, and any audit-specific scanner as the more actionable evidence for header state.

On 2026-05-26, the shared Caddy 403 error page was verified from an external browser against:

https://download.cfts.co/

The page rendered the custom 403 Access restricted HTML while preserving the 403 status. This verifies the POC for Caddy-generated LAN-gate denials. It does not imply upstream application 404 or 500 responses are replaced.

Change Discipline

Before broad rproxy changes:

  1. Update Project-Documentation/02. rproxy Exposure Matrix.md if a hostname, upstream, auth layer, or header profile changes.
  2. Validate Caddy on rproxy.cfts.co.
  3. Reload or restart Caddy.
  4. Check the affected hostname with curl -I and, where relevant, a browser.
  5. Commit the local rproxy workspace if Git is initialized.