Tag: pfsense

Date: October 12, 2026

Sites Involved: Campfire (campfire.madiganpoint.com – Maine) & Pivot4Port (pivot4port.madiganpoint.com – Stowe)

Devices: APC Back-UPS Pro RS 1500MS2 (Maine), APC Back-UPS Pro NS 1500M2 (Stowe)

1. Context & Objective

The goal of this project was to establish reliable, persistent power telemetry from our primary pfSense firewalls to our Prometheus and Grafana monitoring stack before the winter shutdown.

• Topology:
  • Maine Firewall (campfire – 192.168.76.1): APC UPS connected via USB to a Protectli vault chassis. Runs the FreeBSD nut package (usbhid-ups and upsd daemon listening on TCP 3493).
  • Collector (bear – 192.168.76.32): Runs nut_exporter in Docker, proxying scrapes between Prometheus and campfire:3493.
  • Metrics Stack: K3s cluster running Prometheus (scraping every 30s) and Grafana visualizing input voltage, battery load, charge, and online state.

2. Incident: 3:00 AM Telemetry Drop & Root Cause Analysis

The Failure

At 03:00 AM, telemetry went completely flat in Grafana. Prometheus reported:

Plaintext

server returned HTTP status 500 Internal Server Error
read tcp 10.169.0.4:35626->192.168.76.1:3493: i/o timeout

Locally on campfire, the pfSense WebGUI reported Alert: The UPS requires attention / Failed to retrieve status. Restarting the NUT service via the GUI did not recover the connection.

Root Cause

1. USB Bus Renegotiation: At 3:00 AM (during system log rotation / maintenance), the USB controller momentarily power-cycled its downstream ports, forcing FreeBSD to re-enumerate the bus.
2. Device Node Drift: The UPS shifted from ugen0.5 down to ugen0.4.
3. FreeBSD Permission Lockout: FreeBSD created the new character devices (/dev/usb/0.4.0 and /dev/usbctl) with default restrictive permissions (root:operator0644). The unprivileged nut user running usbhid-ups was denied access:
   Plaintextlibusb1: Could not open any HID devices: insufficient permissions on everything
4. Daemon Deadlock:usbhid-ups died and left a stale socket file in /var/db/nut/. The upsd server hung indefinitely trying to talk to the dead socket, causing incoming TCP 3493 connections from bear to time out.
5. Why pfSense Backups Don’t Cover This: Standard pfSense XML backups only save WebGUI application configuration (e.g., nut.conf, user definitions, port assignments). They do not back up or restore low-level FreeBSD OS configuration files like /etc/devfs.conf or custom /usr/local/etc/devd/ rules across upgrades or fresh reinstalls.

3. Runbook: Permanent Fix & Hardening

To make this configuration resilient against USB disconnects, power bounces, and port re-enumerations, we implemented a dual-layer fix on both campfire (Maine) and pivot4port (Stowe).

Step 1: Kernel-Level DevFS Persistence (/etc/devfs.conf)

We appended permissions rules to /etc/devfs.conf so FreeBSD enforces nut:nut and 0666 mode whenever nodes are created or devfs is reloaded:

Bash

printf '\n# NUT APC UPS Permissions\nown /dev/ugen* nut:nut\nperm /dev/ugen* 0666\nown /dev/usb/* nut:nut\nperm /dev/usb/* 0666\nown /dev/usbctl nut:nut\nperm /dev/usbctl 0666\n' >> /etc/devfs.conf
/etc/rc.d/devfs restart

Step 2: Dynamic Hotplug Event Handling (/usr/local/etc/devd/nut_apc.conf)

Because devices behind USB hubs can re-attach dynamically without triggering a full devfs reload, we added a catch-all devd event rule matching the APC USB Vendor ID (0x051d):

Bash

printf 'notify 100 {\n\tmatch "system" "USB";\n\tmatch "subsystem" "DEVICE";\n\tmatch "type" "ATTACH";\n\tmatch "vendor" "0x051d";\n\taction "chown -R nut:nut /dev/ugen* /dev/usb* /dev/usbctl; chmod -R 666 /dev/ugen* /dev/usb* /dev/usbctl";\n};\n' > /usr/local/etc/devd/nut_apc.conf
/etc/rc.d/devd restart

Step 3: Emergency Recovery Commands (If Ever Needed)

If a hung process or stale socket locks the service before permissions apply:

Bash

killall -9 usbhid-ups upsd upsmon 2>/dev/null
rm -f /var/db/nut/* /var/run/nut/*
/etc/rc.d/devfs restart
/usr/local/etc/rc.d/nut.sh start
upsc <ups_name>@localhost

4. Grafana Panel PromQL Correction

During setup, the pfSense Grid Attachment State panel displayed constant red because it was querying output load (network_ups_tools_ups_load) instead of status flags.

• Incorrect Query:network_ups_tools_ups_load{...} (returned 4–6% load, breaking the binary 0/1 color mapping).
• Corrected Query:
  Code snippetnetwork_ups_tools_ups_status{instance="camp-pfsense-ups", job="camp-pfsense-ups", flag="OL"} (Returns 1 when Online, 0 when On Battery, mapping cleanly to Dark Green / Dark Red).

5. Physical Cabling & Interconnect Topology

Plaintext

[APC Back-UPS 1500]
       │
       │ (Proprietary RJ50 / 10-pin modular to USB-A cable - APC AP9827)
       ▼
[USB-A Female to USB-C Male Adapter / Dongle]
       │
       ▼
[Protectli Vault Chassis] (Front USB-C Port)

The APC 1500 units use an RJ50 (10-pin) to USB Type-A data cable (AP9827). Because the Protectli firewall uses USB-C ports on the front I/O, a compact USB-A Female to USB-C Male adapter connects the UPS directly to the chassis, bypassing unnecessary intermediary powered hubs.

6. Battery Fleet Health Matrix & May 2027 Shopping List

Location	UPS Model	Battery Cartridge	Serial Number	Battery Mfg Date	Health / Action Plan
Camp Main House	Back-UPS Pro RS 1500MS2	APC RBC124	4B2115P17561	2021/04/16	Replace May 2027 (~5.5 yrs old; rapid voltage sag)
Camp Garage	Back-UPS Pro RS 1500MS2	APC RBC124	0B2352L39196	2023/12/29	Healthy (~2.8 yrs old; ~135 min runtime at 4% load)
Stowe Firewall	Back-UPS Pro NS 1500M2	APC RBC124	0B2422L11386	2024/05/28	Healthy (~2.4 yrs old; 100% capacity)

Items to Purchase Before Returning in May:

1. Replacement Battery Pack:
   • Part:APC RBC124 (Pre-wired cartridge consisting of two 12V 9Ah AGM lead-acid batteries with inline fuse and APC terminal connector).
   • Cost: ~$65 – $75 (OEM equivalent by ExpertPower or UPSBatteryCenter) / ~$95 – $105 (Schneider/APC Original).
2. Chassis USB Adapter:
   • Part:USB Type-A Female to USB Type-C Male Adapter (low-profile aluminum stub or 6″ pigtail, USB 3.0 OTG compliant, e.g., Anker or CableCreation 2-pack).
   • Cost: ~$8 – $11.