Edge Portal Backend Operations Guide

Internal version: 2026-06-11

This guide is for CFTS operators administering the Edge Portal backend. It is not client-facing documentation and must not contain passwords, private keys, API tokens, recovery codes, or live secret values.

1. Current Live Baseline

Quick health checks from the Edge host:

systemctl is-active cfts-edge-portal.service
curl -fsSI http://172.16.198.101:8090/login | grep -i X-Cfts-Edge-Version

The expected version header after the v3.7 deploy is:

X-Cfts-Edge-Version: v3.7

2. Operating Model

Client portal access is based on named ESXi accounts. A portal login still requires valid ESXi credentials, and ESXi permissions decide which VM objects the user can see or operate.

The portal adds these local controls around ESXi:

• username allow/block gates;
• optional require_configured_user;
• per-user source-IP allow/block rules;
• per-user portal action grants;
• public operational notices;
• client-visible notes, login history, and password-change facilities;
• local-admin break-glass password reset;
• temporary maintenance snapshot facilities.

Do not use a broad shared ESXi administrator account for normal client operations. Client accounts should have only the ESXi role and VM scope needed for their service.

3. Admin Access Page

The normal protected admin URL is:

https://edge.cfts.co/admin/access

Access requirements:

• Caddy on rproxy.cfts.co must allow the request to /admin/*;
• the immediate proxy must be listed in portal.trusted_proxy_networks;
• the forwarded client IP must match portal.admin_access_networks;
• the operator must unlock the page using the password held in VPS_PORTAL_ADMIN_PASSWORD.

Do not use direct origin/IP admin access as the routine operator path. Keep direct Edge origin checks for service smoke testing and troubleshooting only.

The page writes runtime controls to:

/var/log/cfts-edge-portal/access-controls.yaml

Use this page for routine changes instead of redeploying /etc/cfts-edge-portal/config.yaml.

The page also manages public operational notices. Notices are stored in:

/var/log/cfts-edge-portal/operational-notices.yaml

Use the GUI for routine planned-work, degraded-service, outage, or information notices. The public login/dashboard banner renders plain text only. Use Clear notice when the message no longer applies.

Client Notepad Runtime Data

Client notepad data is stored in:

/var/log/cfts-edge-portal/notepad.json

The file path is controlled by portal.notepad_path. Notes are account-owned and render sanitized Markdown in the client notepad panel.

Markdown task-list lines render as interactive client checklist items. Indented child task lines must be complete before the parent task can be marked complete. Completed top-level items and completed subtrees render under Complete inside the same note. If a child is unticked later, checked ancestors are cleared so the stored Markdown stays consistent.

Do not treat notepad data as a secrets store. Operators with server access may inspect note data during support, security, or recovery work, and Markdown exports leave the portal security boundary.

4. User Admission Controls

Admin page fields:

The login decision is:

not blocked
and, if allowed usernames is non-empty, listed in allowed usernames
and, if require configured user is enabled, present in users:
and source IP passes per-user network rules
and ESXi authentication succeeds
and ESXi permissions permit the requested VM/action

When editing network rules, do not save a rule set that locks out the client unless this is the intended security action. The portal should prevent saving self-locking rules for the signed-in user where it can detect the current client IP.

5. User Action Permissions

The admin page field User action permissions controls per-user portal action grants.

Line format:

username: console, power_on, shutdown_guest, restart_guest, power_off, reset, maintenance_snapshot

Copy-paste starting points:

client-user: console, power_on, shutdown_guest, restart_guest, power_off, reset, maintenance_snapshot

client-user: console, power_on, shutdown_guest, restart_guest, maintenance_snapshot

client-user: console, maintenance_snapshot

Replace client-user with the exact ESXi/portal username. Keep action keys lowercase and comma-separated.

Valid action keys:

Grant only the actions that the client needs. For example, a client who should create and restore their own temporary snapshots but should not use hard recovery actions might have:

client-user: console, power_on, shutdown_guest, restart_guest, maintenance_snapshot

If maintenance_snapshot is missing, the client should not see the snapshot manager or create-snapshot actions.

The dashboard snapshot manager is the primary client UI and opens as a compact slide-out drawer. The full /vm/<name>/snapshots route remains available as a fallback for direct links or non-JavaScript browser behavior.

6. Break-glass Password Reset

The local-admin password reset form on /admin/access is a break-glass operation.

Use it when:

• a client is locked out and normal password change is not possible;
• an approved operator is present on an approved admin network;
• the operator has valid ESXi admin credentials for the just-in-time reset.

Important behavior:

• the portal does not store the ESXi admin password from the reset form;
• the portal does not send reset email;
• the operator supplies a temporary password manually;
• success and failure are written to the audit log as admin_password_reset;
• the temporary password must be shared out-of-band;
• the user should change it after signing in.

Do not paste passwords into tickets, documentation, chat, screenshots, or logs. Do not leave admin credentials in the browser after the reset is complete.

7. Temporary Maintenance Snapshot Policy

Temporary maintenance snapshots are opt-in per user through maintenance_snapshot.

The live policy is controlled by:

portal:
  maintenance_snapshots:
    expiry_hours: 24
    max_per_vm: 1
    require_approval_for_production: true

Supported policy keys:

Snapshot creation is allowed only when the VM power state is online or offline. It is not available for suspended or unknown VM states.

Snapshots are created with:

• memory=False;
• quiesce=False;
• a name starting with CFTS temporary maintenance;
• description marker CFTS-Temporary-Maintenance: true;
• metadata lines for Created-At, Expires-At, and Created-By.

The portal treats these snapshots as temporary rollback checkpoints, not backups.

8. Snapshot Ownership and Visibility

Clients can list, restore, and delete only temporary maintenance snapshots whose description metadata matches their signed-in portal username.

Ownership check:

Created-By metadata, normalized, equals current portal username

This means:

• snapshots created by another portal user are hidden;
• snapshots created manually in ESXi without the marker/metadata are not shown as client-owned maintenance snapshots;
• direct restore/delete URLs for another user's snapshot return not found and write a denied audit entry;
• dashboard snapshot status counts only active snapshots owned by the signed-in user.

Expired snapshots:

• cannot be restored from the portal;
• may still be deleted by the owning user;
• are removed automatically during later snapshot creation cleanup.

9. Production Approval

Production approval is required when require_approval_for_production is enabled and either:

• the VM status is marked production by portal status data; or
• the user's VM config contains production: true for that vm_name.

Example config-backed production marker:

users:
  client-user:
    vms:
      - vm_name: "CLIENT-PROD-01"
        production: true

The client confirmation copy is:

I confirm this production maintenance action is approved

If approval is required but the checkbox is not submitted, the action is denied and audited.

10. Snapshot Action Audit Trail

The audit log line shape is:

<timestamp> <username> <client_ip> <action> <vm_name> <result> <message>

Relevant actions:

Useful checks:

sudo tail -n 100 /var/log/cfts-edge-portal/audit.log
sudo grep 'maintenance_snapshot' /var/log/cfts-edge-portal/audit.log | tail -n 50
sudo grep 'admin_password_reset' /var/log/cfts-edge-portal/audit.log | tail -n 20

Treat audit output as operational data. Do not paste full logs into client-facing channels without reviewing for sensitive context.

11. Grant Snapshot Access To A Client

1. Confirm the client has a named ESXi account with the correct VM scope.
2. Confirm the client should be allowed to create and manage temporary maintenance snapshots.
3. Open https://edge.cfts.co/admin/access from an approved admin network.
4. Unlock admin access.
5. In User action permissions, add or update a line for the user.
6. Include maintenance_snapshot and only the other actions the user needs.
7. Save controls.
8. Ask the client to sign out and back in.
9. Confirm the service card shows Maintenance actions.
10. Confirm the client sees Snapshot manager and Create snapshot only on assigned services.

Minimum validation:

Client can open Snapshot manager for assigned VM.
Client can create one temporary maintenance snapshot.
Dashboard shows Snapshot active with expiry.
Client cannot see another user's portal-created snapshot.
Audit log records success or denied entries as expected.

12. Responding To Common Issues

13. Deployment and Rollback Notes

Routine deployments are staged as timestamped bundles under /tmp and installed by Peter or an approved operator with:

sudo bash /tmp/cfts-edge-portal-deploy-YYYYMMDD-HHMMSS/deploy.sh

The deploy script should create a backup under:

/opt/cfts-edge-portal/.deploy-backups/YYYYMMDD-HHMMSS

After deployment, verify:

systemctl is-active cfts-edge-portal.service
curl -fsSI http://172.16.198.101:8090/login | grep -i X-Cfts-Edge-Version

If rollback is needed, use the timestamped backup created by the deploy bundle. Do not reset the live app tree or overwrite config from a stale local copy.

14. Change Safety

Before changing backend controls, confirm:

• whether the change is client-facing or internal-only;
• the exact username, VM name, and action key;
• whether the VM is production;
• whether the action is a break-glass operation;
• whether a backup restore, not a maintenance snapshot, is the correct recovery path.

After changing controls, record the validation result in the project test log or the relevant operational ticket.