Last updated: 2026-05-27

This document tracks the operating workflow for the CFTS reverse proxy at `rproxy.cfts.co`.

## Source Layout

Local workspace:

```text
F:\Avery_Vault_LLM\07-Projects\CFTS\Platforms\rproxy.cfts.co
```

Primary files:

```text
Project-Documentation/00. rproxy Caddy Overview For New Admins.md
Project-Documentation/03. rproxy Function and Feature Synopsis.md
caddy/Caddyfile
caddy/includes/00-common.caddy
caddy/errors/{403,404,500}.html
observability/install-caddy-goaccess-report.sh
observability/goaccess/render-caddy-goaccess-report.sh
observability/systemd/caddy-goaccess-report.{service,timer}
security/fail2ban/filter.d/*.conf
security/fail2ban/jail.d/caddy-rproxy.local
security/apt/apt.conf.d/*
Project-Documentation/02. rproxy Exposure Matrix.md
```

Production layout on `rproxy.cfts.co`:

```text
/etc/caddy/Caddyfile
/etc/caddy/includes/00-common.caddy
/etc/caddy/errors/{403,404,500}.html
/usr/local/sbin/render-caddy-goaccess-report
/etc/systemd/system/caddy-goaccess-report.{service,timer}
/var/www/caddy-log-report/index.html
```

## Caddy Include Model

The main Caddyfile imports reusable snippets near the top:

```caddy
import includes/*.caddy
```

The common include defines:

```text
json_access_log
safe_headers
lose_headers_no_csp
strip_response_identity
strip_spoofed_request_headers
strip_upstream_security_headers
no_uploads_expected
error_page_headers
default_error_pages
safe_tls
vsphere_tls
public_site_base
```

Use `public_site_base` only for simple public sites that can tolerate strict framing and CSP. Do not apply it blindly to dashboards, ticket systems, consoles, or legacy apps without browser testing.

`json_access_log` is the standard dedicated access-log block. Import it with the target log path instead of pasting another full `log { output file ... }` block.

`strip_response_identity` removes common software-identifying response headers such as `Server`, `Via`, `X-Powered-By`, and framework/version breadcrumbs. It is imported through `safe_tls` for the hostnames that use the shared TLS profile; the ESXi edge hostnames remain in a minimum-interference proxy mode.

`strip_spoofed_request_headers` removes client-supplied proxy identity headers such as `Forwarded`, `X-Forwarded-*`, `X-Real-IP`, and common CDN/client-IP hints before requests reach upstream apps. Caddy still adds its own trusted `X-Forwarded-*` values inside `reverse_proxy`, and site-specific `header_up` rules such as `X-Real-IP {remote_host}` still apply after the scrub.

`strip_upstream_security_headers` is used inside selected `reverse_proxy` blocks when Caddy owns the public header policy and the upstream app's own security headers would conflict with that policy.

`no_uploads_expected` caps request bodies at `1MB` for sites where browser-based uploads are not expected. It is currently applied to `docs.cfts.co`, `isp-status.cfts.co`, and `monitor.cfts.co`. The Caddy `request_body` directive requires Caddy v2.10.0 or newer; if `caddy validate` rejects it, update Caddy or remove this import before reload.

`vsphere_tls` keeps the same TLS protocol/cipher posture as `safe_tls` and still imports `strip_response_identity`, but deliberately does not import `strip_spoofed_request_headers`. Keep it reserved for ESXi/vSphere testing only; as of 2026-05-26, `edge-01.cfts.co` and `edge-02.cfts.co` do not import any header/TLS protection snippets because the ESXi Host Client is being debugged in a minimum-interference proxy mode.

For ESXi, Caddy now applies a source-IP gate before proxying: only clients in `172.16.198.0/24` reach `edge-01.cfts.co` and `edge-02.cfts.co`; everyone else receives `403 Access restricted`. Add trusted VPN or static admin CIDRs to the per-site `@admin_clients` matcher if remote admin access is required.

The only active ESXi request-header change is a compatibility scrub on `/sdk*` and `/screen*`: Caddy removes `Authorization` before proxying those paths. It prevents stale browser-cached Caddy Basic Auth credentials from being forwarded into VMware SOAP/screen calls after the earlier Caddy Basic Auth test.

The strict profile also sends `Permissions-Policy` with browser features denied by default.

The strict CSP in `safe_headers` intentionally avoids `script-src 'unsafe-inline'`, `style-src 'unsafe-inline'`, `data:` image sources, and inline script/style attributes. The only inline script currently allowed is the static JSON-LD block on the docs homepage, pinned by SHA-256 hash. If that JSON-LD changes, recompute the hash and update `caddy/includes/00-common.caddy`.

## Default Error Pages

The `default_error_pages` snippet serves shared HTML pages for Caddy-generated `403`, `404`, and `5xx` errors from:

```text
/etc/caddy/errors
```

Source workspace copies live under:

```text
caddy/errors/403.html
caddy/errors/404.html
caddy/errors/500.html
```

The snippet is imported by every active HTTPS site block. LAN-only deny paths use `error 403` rather than `respond "Access restricted" 403`, so they render the shared `403.html` page while preserving the `403` status code.

Caddy `handle_errors` does not rewrite ordinary upstream responses from `reverse_proxy`. If an upstream application returns its own `404` or `500`, Caddy passes that response through unless a site-specific `reverse_proxy handle_response` rule is added deliberately.

Verified on 2026-05-26: an external request to `https://download.cfts.co/` displayed the shared `403 Access restricted` page. This confirms the useful scope of the POC: cleaner Caddy-side denial and fallback pages, not replacement of upstream app error pages.

Known test caveats:

- Testing LAN-only hostnames from `rproxy.cfts.co` or the trusted LAN can return upstream `200`, because those clients are allowed by the `client_ip` gate.
- `https://isp-status.cfts.co/opps` from the WAN returns `401` when it falls into the Basic Auth handler; that is expected and does not exercise the shared `404`.
- `https://docs.cfts.co/surely-this-does-not-exist` returns the docs app's upstream `404`; that is also expected because Caddy passes proxied responses through.

Use an external client against LAN-only hostnames such as `download.cfts.co`, `console.dns.cfts.co`, `ai.cfts.co`, or `inventory.cfts.co` to verify the shared `403` page.

## Explicit HTTP Routing / Unknown Host Catch-All

The Caddyfile sets `auto_https disable_redirects` so Caddy keeps automatic certificate management but does not emit broad HTTP-to-HTTPS redirects before site routing. A single `http://` block then redirects only known hostnames to HTTPS and closes unknown plaintext HTTP hostnames with `abort`.

The `http://` block imports `strip_response_identity`, logs unknown hosts to `/var/log/caddy/unknown-host-access.log`, and matches known hostnames with `@known_hosts`. `log_skip @known_hosts` keeps normal HTTP-to-HTTPS redirects out of the unknown-host log so Fail2Ban can treat every logged line there as bad-host evidence. Add new public hostnames to that matcher when adding new HTTPS site blocks.

Do not add a broad `https://` catch-all unless wildcard or on-demand TLS is intentionally designed first. Unknown HTTPS names should fail before HTTP routing if Caddy has no matching certificate; accepting arbitrary HTTPS names would widen the TLS surface.

As of 2026-05-27, `@known_hosts` includes the current active hostnames `edge-01.cfts.co`, `edge-02.cfts.co`, `inventory.cfts.co`, and `rp-logs.cfts.co`. It also retains older `vps.cfts.co` and `hvps.cfts.co` names for compatibility/history.

## Global Server Guardrails

The global options block applies listener-level guardrails:

```caddy
servers :80 {
	timeouts {
		read_header 10s
		idle 2m
	}
	max_header_size 64KB
	protocols h1
}

servers :443 {
	timeouts {
		read_header 10s
		idle 2m
	}
	max_header_size 64KB
	protocols h1 h2
	strict_sni_host on
	0rtt off
}
```

This limits slow header reads, reduces idle connection dwell time, caps oversized request headers, disables HTTP/3, rejects TLS SNI/Host mismatches, and disables TLS early data.

After the `protocols h1 h2` change is live and verified, remove the no-longer-needed public UDP exposure:

```bash
sudo ufw delete allow 443/udp
sudo ufw status verbose
```

## SFTP Deployment Workflow

SFTP does not write directly to `/etc/caddy`. Stage files under:

```text
/home/sysops/temp
```

Then SSH into `rproxy.cfts.co` and copy with `sudo`:

```bash
sudo mkdir -p /etc/caddy/includes /etc/caddy/errors
sudo cp /home/sysops/temp/Caddyfile /etc/caddy/Caddyfile
sudo cp /home/sysops/temp/includes/00-common.caddy /etc/caddy/includes/00-common.caddy
sudo cp /home/sysops/temp/errors/*.html /etc/caddy/errors/
sudo chown root:root /etc/caddy/Caddyfile /etc/caddy/includes/00-common.caddy /etc/caddy/errors/*.html
sudo chmod 644 /etc/caddy/Caddyfile /etc/caddy/includes/00-common.caddy /etc/caddy/errors/*.html
```

## Access Log Permissions

Dedicated log files must be writable by the `caddy` service user before reload. Example for `docs.cfts.co`:

```bash
sudo install -d -o caddy -g caddy -m 0750 /var/log/caddy
sudo touch /var/log/caddy/docs-access.log
sudo chown caddy:caddy /var/log/caddy/docs-access.log
sudo chmod 0640 /var/log/caddy/docs-access.log
```

For `edge-01.cfts.co` and `edge-02.cfts.co`, the dedicated logs currently retain the older `vps` and `hvps` names:

```bash
sudo touch /var/log/caddy/vps-access.log /var/log/caddy/hvps-access.log
sudo chown caddy:caddy /var/log/caddy/vps-access.log /var/log/caddy/hvps-access.log
sudo chmod 0640 /var/log/caddy/vps-access.log /var/log/caddy/hvps-access.log
```

Other public app logs currently used by the Caddyfile:

```bash
sudo touch /var/log/caddy/tickets-access.log /var/log/caddy/redmine-access.log /var/log/caddy/tracks-access.log /var/log/caddy/monitor-access.log /var/log/caddy/unknown-host-access.log /var/log/caddy/inventory.log /var/log/caddy/rp-logs-access.log
sudo chown caddy:caddy /var/log/caddy/tickets-access.log /var/log/caddy/redmine-access.log /var/log/caddy/tracks-access.log /var/log/caddy/monitor-access.log /var/log/caddy/unknown-host-access.log /var/log/caddy/inventory.log /var/log/caddy/rp-logs-access.log
sudo chmod 0640 /var/log/caddy/tickets-access.log /var/log/caddy/redmine-access.log /var/log/caddy/tracks-access.log /var/log/caddy/monitor-access.log /var/log/caddy/unknown-host-access.log /var/log/caddy/inventory.log /var/log/caddy/rp-logs-access.log
```

If reload status mentions `opening log writer` and `permission denied`, create and chown the named log file before reloading again.

## Log Report GUI

`rp-logs.cfts.co` is a LAN-only HTTPS site that serves a static GoAccess report from:

```text
/var/www/caddy-log-report/index.html
```

The report turns Caddy JSON access logs into a browser-readable dashboard with request volume, status codes, paths, referrers, clients, and other useful summaries. It is intentionally static and refreshed by a systemd timer every five minutes rather than running a separate long-lived web service.

Deploy or update the report generator. Upload the whole `observability` folder to `/home/sysops/temp/observability`, then run these commands one line at a time on `rproxy.cfts.co`:

```bash
sudo apt update
```

```bash
sudo apt install goaccess
```

```bash
sh -n /home/sysops/temp/observability/install-caddy-goaccess-report.sh
```

```bash
sh -n /home/sysops/temp/observability/goaccess/render-caddy-goaccess-report.sh
```

```bash
sudo sh /home/sysops/temp/observability/install-caddy-goaccess-report.sh
```

Check it:

```bash
systemctl status caddy-goaccess-report.service --no-pager
systemctl list-timers caddy-goaccess-report.timer
ls -lh /var/www/caddy-log-report/index.html
```

Open from the LAN:

```text
https://rp-logs.cfts.co/
```

The systemd service runs as `caddy`, so Caddy log files must remain readable by the `caddy` group.

The installer copies the renderer to `/usr/local/sbin/render-caddy-goaccess-report`, installs the systemd unit and timer, prepares `/var/www/caddy-log-report`, prepares `/var/log/caddy/rp-logs-access.log`, smoke-tests GoAccess when a non-empty Caddy log exists, starts one report render, and enables the five-minute timer.

For the full paste-safe runbook and troubleshooting notes, use `observability/README.md`.

Do not use `goaccess --help | grep CADDY` as the compatibility check; some packages do not list every predefined format in short help. The useful check is whether `goaccess "$log" --log-format=CADDY -o /tmp/goaccess-caddy-smoke.html` can parse a real Caddy JSON access log and create a non-empty HTML file. If that command reports an unknown `CADDY` log format, install a newer GoAccess package before enabling the timer. The report generator uses GoAccess' built-in Caddy JSON parser.

## Basic Auth Hashes

For public admin-style surfaces, prefer a Caddy `basic_auth` layer in front of upstream authentication.

Generate a bcrypt hash with Caddy:

```bash
caddy hash-password --plaintext 'new password here'
```

Store only the generated hash in the Caddyfile:

```caddy
basic_auth {
    vps_acces <bcrypt-hash>
}
```

On 2026-05-15, Caddy Basic Auth was tested on the ESXi hostnames then in use, `vps.cfts.co` and `hvps.cfts.co`. It correctly returned `401 Unauthorized` before credentials, but it triggered vSphere web client errors after login. Basic Auth was removed live.

On 2026-05-17, the browser still showed cached `vps_access` Basic Auth prompts while the ESXi Host Client raised JavaScript errors around missing VM/host state. A short-lived Caddyfile test stripped stale Caddy Basic Auth credentials and forced the upstream `Host` header to the direct ESXi IP, but the ESXi Host Client error persisted.

Access logs then showed `POST /sdk/` returning `500` and `GET /screen?...` returning `401` with `Www-Authenticate: Basic realm="VMware HTTP server"`, while the browser was still sending an `Authorization` header. The current diagnostic fix strips `Authorization` only for `/sdk*` and `/screen*` on `edge-01.cfts.co` and `edge-02.cfts.co`.

These are vSphere 7.0.2 / 8 admin-console surfaces and can be cranky behind a reverse proxy. Keep them in minimum-interference mode while debugging: source-IP allowlisting is active, but avoid adding a Caddy header profile, custom TLS snippet, upstream `Host` override, Caddy auth, broad request-header changes, or request-body caps without testing. If hardening is reintroduced later, add exactly one change at a time and browser-test the ESXi UI before adding the next.

Future access-control options to consider for these hostnames:

- Cloudflare Access / Zero Trust in front of Caddy.
- VPN-first access by adding VPN/admin CIDRs to the existing Caddy allowlist.
- Conditional allowlist expansion for known static admin networks where practical.
- Continue relying on upstream vSphere auth, with logging and strong upstream account policy, if roaming clients make allowlisting impractical.

Decision for `monitor.cfts.co`: keep it public. It fronts PRTG, and the business value is being able to check monitoring from offsite. Do not make it LAN-only as a generic hardening move.

For `monitor.cfts.co`, avoid reflexively adding Caddy Basic Auth if it would interfere with PRTG workflows. Current Caddy posture is public reachability, `lose_headers_no_csp`, `safe_tls`, `no_uploads_expected`, dedicated JSON logging, and upstream PRTG authentication as the access-control layer. If this changes later, prefer a deliberate PRTG-tested pattern such as path-specific protection or a known admin allowlist.

## Fail2Ban

Fail2Ban is enabled on `rproxy.cfts.co` as a host-level hardening layer for SSH.

Verified on 2026-05-15:

```text
fail2ban.service: active (running), enabled
active jail: sshd
SSH port: 4422
backend: systemd
current bans: 0
```

Local override file:

```text
/etc/fail2ban/jail.d/sshd.local
```

Expected SSH jail shape:

```ini
[sshd]
enabled = true
port = 4422
backend = systemd
maxretry = 5
findtime = 10m
bantime = 1h
```

Quick checks:

```bash
sudo systemctl status fail2ban --no-pager
sudo fail2ban-client status
sudo fail2ban-client status sshd
sudo grep -R "^[[:space:]]*port" /etc/fail2ban/jail.d /etc/fail2ban/jail.local /etc/fail2ban/jail.conf
```

Caddy log banning is configured in the source workspace under:

```text
security/fail2ban/filter.d/caddy-unknown-host.conf
security/fail2ban/filter.d/caddy-scanner-paths.conf
security/fail2ban/jail.d/caddy-rproxy.local
```

`caddy-unknown-host` watches `/var/log/caddy/unknown-host-access.log`. Because known-host redirects are skipped with `log_skip @known_hosts`, each logged line represents an unknown plaintext HTTP host.

`caddy-scanner-paths` watches `/var/log/caddy/*-access.log` for obvious opportunistic scanner paths such as `/.env`, `/.git`, `/wp-login.php`, `/phpmyadmin`, `/cgi-bin`, `/server-status`, `/actuator`, and similar probes.

Deploy:

```bash
sudo cp /home/sysops/temp/security/fail2ban/filter.d/caddy-unknown-host.conf /etc/fail2ban/filter.d/caddy-unknown-host.conf
sudo cp /home/sysops/temp/security/fail2ban/filter.d/caddy-scanner-paths.conf /etc/fail2ban/filter.d/caddy-scanner-paths.conf
sudo cp /home/sysops/temp/security/fail2ban/jail.d/caddy-rproxy.local /etc/fail2ban/jail.d/caddy-rproxy.local
sudo fail2ban-client reload
sudo fail2ban-client status caddy-unknown-host
sudo fail2ban-client status caddy-scanner-paths
```

Check filters against live logs before or after reload:

```bash
sudo fail2ban-regex /var/log/caddy/unknown-host-access.log /etc/fail2ban/filter.d/caddy-unknown-host.conf
sudo fail2ban-regex /var/log/caddy/docs-access.log /etc/fail2ban/filter.d/caddy-scanner-paths.conf
```

Current jail thresholds are conservative:

```text
caddy-unknown-host: 20 hits in 10 minutes -> 1 hour ban
caddy-scanner-paths: 5 hits in 10 minutes -> 4 hour ban
ignoreip: 127.0.0.1/8, ::1, 172.16.198.0/24
```

Keep web jails conservative to avoid blocking legitimate public users.

## Unattended Security Updates

Security update automation is configured in the source workspace under:

```text
security/apt/apt.conf.d/20auto-upgrades
security/apt/apt.conf.d/52unattended-upgrades-cfts-security
```

This enables daily package list updates and unattended Ubuntu security upgrades, removes unused dependencies, and does not automatically reboot.

Deploy and dry-run:

```bash
sudo apt update
sudo apt install unattended-upgrades apt-listchanges
sudo cp /home/sysops/temp/security/apt/apt.conf.d/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades
sudo cp /home/sysops/temp/security/apt/apt.conf.d/52unattended-upgrades-cfts-security /etc/apt/apt.conf.d/52unattended-upgrades-cfts-security
sudo unattended-upgrade --dry-run --debug
systemctl status apt-daily.timer apt-daily-upgrade.timer --no-pager
```

Useful checks:

```bash
grep -R "Unattended-Upgrade::Automatic-Reboot" /etc/apt/apt.conf.d
sudo tail -n 100 /var/log/unattended-upgrades/unattended-upgrades.log
apt list --upgradable
```

If `systemctl reload caddy` appears stuck after a failed reload, check:

```bash
systemctl status caddy --no-pager
```

A clean `active (running)` state after restart means recovery succeeded.

## Validate And Reload

Always validate before reload:

```bash
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
systemctl status caddy --no-pager
```

If reload does not settle after a previous failed reload, restart instead:

```bash
sudo systemctl restart caddy
systemctl status caddy --no-pager
```

## Current Good Verification

On 2026-05-15, `docs.cfts.co` was verified through Caddy after the include split:

```text
/ returned 200
/static/og-image.png returned 200
/var/log/caddy/docs-access.log received JSON access entries
Strict-Transport-Security, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, and Content-Security-Policy were present
```

On 2026-05-15, after moving docs JavaScript to a static file, tightening CSP, and adding `Permissions-Policy`, SecurityHeaders reported `A+` for:

```text
https://docs.cfts.co/
```

Evidence screenshot:

```text
Project-Documentation/Evidence/securityheaders-docs-cfts-co-a-plus-2026-05-15.png
```

Useful public test links:

```text
https://securityheaders.com/?q=https%3A%2F%2Fdocs.cfts.co%2F&followRedirects=on
https://www.ssllabs.com/ssltest/analyze.html?d=docs.cfts.co
https://developer.mozilla.org/en-US/observatory/analyze?host=docs.cfts.co
https://internet.nl/site/docs.cfts.co/
```

The Pentest-Tools website scanner previously reported only low findings, but continued to flag CSP heuristically after the live header was tightened. Treat SecurityHeaders, live `curl -I`, and any audit-specific scanner as the more actionable evidence for header state.

On 2026-05-26, the shared Caddy `403` error page was verified from an external browser against:

```text
https://download.cfts.co/
```

The page rendered the custom `403 Access restricted` HTML while preserving the `403` status. This verifies the POC for Caddy-generated LAN-gate denials. It does not imply upstream application `404` or `500` responses are replaced.

## Change Discipline

Before broad rproxy changes:

1. Update `Project-Documentation/02. rproxy Exposure Matrix.md` if a hostname, upstream, auth layer, or header profile changes.
2. Validate Caddy on `rproxy.cfts.co`.
3. Reload or restart Caddy.
4. Check the affected hostname with `curl -I` and, where relevant, a browser.
5. Commit the local rproxy workspace if Git is initialized.