Maine Camp infrastructure remains fully operational and occupied, with all systems nominal under overcast, 31°F skies. The 24-hour vision intelligence summary confirmed a hive of domestic activity, analyzing 34 events that primarily chronicled the ceaseless patrols of a resident male, his canine companion(s), and two steadfast pickup trucks around the gravel driveway. Amidst this flurry of authorized movement, the system also logged four wildlife sightings—including a brief cameo by a “fox-like animal”—while confirming zero unrecognized vendor or vehicle incursions. On the digital front, Fidium Fiber is dutifully handling the data load at 173.63 GiB against HughesNet’s quiet 2.74 GiB, supporting 13 concurrent network leases. Core environmental health is green across all access points, the server rack is embracing the local climate at a brisk 38.85°F, and the power matrix is unflappable with a 5.0% load on a stable 121.0V utility feed and a 99% battery reserve.
Devices: APC Back-UPS Pro RS 1500MS2 (Maine), APC Back-UPS Pro NS 1500M2 (Stowe)
1. Context & Objective
The goal of this project was to establish reliable, persistent power telemetry from our primary pfSense firewalls to our Prometheus and Grafana monitoring stack before the winter shutdown.
Topology:
Maine Firewall (campfire – 192.168.76.1): APC UPS connected via USB to a Protectli vault chassis. Runs the FreeBSD nut package (usbhid-ups and upsd daemon listening on TCP 3493).
Collector (bear – 192.168.76.32): Runs nut_exporter in Docker, proxying scrapes between Prometheus and campfire:3493.
Metrics Stack: K3s cluster running Prometheus (scraping every 30s) and Grafana visualizing input voltage, battery load, charge, and online state.
2. Incident: 3:00 AM Telemetry Drop & Root Cause Analysis
The Failure
At 03:00 AM, telemetry went completely flat in Grafana. Prometheus reported:
Plaintext
server returned HTTP status 500 Internal Server Error
read tcp 10.169.0.4:35626->192.168.76.1:3493: i/o timeout
Locally on campfire, the pfSense WebGUI reported Alert: The UPS requires attention / Failed to retrieve status. Restarting the NUT service via the GUI did not recover the connection.
Root Cause
USB Bus Renegotiation: At 3:00 AM (during system log rotation / maintenance), the USB controller momentarily power-cycled its downstream ports, forcing FreeBSD to re-enumerate the bus.
Device Node Drift: The UPS shifted from ugen0.5 down to ugen0.4.
FreeBSD Permission Lockout: FreeBSD created the new character devices (/dev/usb/0.4.0 and /dev/usbctl) with default restrictive permissions (root:operator0644). The unprivileged nut user running usbhid-ups was denied access: Plaintextlibusb1: Could not open any HID devices: insufficient permissions on everything
Daemon Deadlock:usbhid-ups died and left a stale socket file in /var/db/nut/. The upsd server hung indefinitely trying to talk to the dead socket, causing incoming TCP 3493 connections from bear to time out.
Why pfSense Backups Don’t Cover This: Standard pfSense XML backups only save WebGUI application configuration (e.g., nut.conf, user definitions, port assignments). They do not back up or restore low-level FreeBSD OS configuration files like /etc/devfs.conf or custom /usr/local/etc/devd/ rules across upgrades or fresh reinstalls.
3. Runbook: Permanent Fix & Hardening
To make this configuration resilient against USB disconnects, power bounces, and port re-enumerations, we implemented a dual-layer fix on both campfire (Maine) and pivot4port (Stowe).
Because devices behind USB hubs can re-attach dynamically without triggering a full devfs reload, we added a catch-all devd event rule matching the APC USB Vendor ID (0x051d):
During setup, the pfSense Grid Attachment State panel displayed constant red because it was querying output load (network_ups_tools_ups_load) instead of status flags.
Incorrect Query:network_ups_tools_ups_load{...} (returned 4–6% load, breaking the binary 0/1 color mapping).
Corrected Query: Code snippetnetwork_ups_tools_ups_status{instance="camp-pfsense-ups", job="camp-pfsense-ups", flag="OL"} (Returns 1 when Online, 0 when On Battery, mapping cleanly to Dark Green / Dark Red).
The APC 1500 units use an RJ50 (10-pin) to USB Type-A data cable (AP9827). Because the Protectli firewall uses USB-C ports on the front I/O, a compact USB-A Female to USB-C Male adapter connects the UPS directly to the chassis, bypassing unnecessary intermediary powered hubs.
6. Battery Fleet Health Matrix & May 2027 Shopping List
Location
UPS Model
Battery Cartridge
Serial Number
Battery Mfg Date
Health / Action Plan
Camp Main House
Back-UPS Pro RS 1500MS2
APC RBC124
4B2115P17561
2021/04/16
Replace May 2027 (~5.5 yrs old; rapid voltage sag)
Camp Garage
Back-UPS Pro RS 1500MS2
APC RBC124
0B2352L39196
2023/12/29
Healthy (~2.8 yrs old; ~135 min runtime at 4% load)
Stowe Firewall
Back-UPS Pro NS 1500M2
APC RBC124
0B2422L11386
2024/05/28
Healthy (~2.4 yrs old; 100% capacity)
Items to Purchase Before Returning in May:
Replacement Battery Pack:
Part:APC RBC124 (Pre-wired cartridge consisting of two 12V 9Ah AGM lead-acid batteries with inline fuse and APC terminal connector).
Cost: ~$65 – $75 (OEM equivalent by ExpertPower or UPSBatteryCenter) / ~$95 – $105 (Schneider/APC Original).
Chassis USB Adapter:
Part:USB Type-A Female to USB Type-C Male Adapter (low-profile aluminum stub or 6″ pigtail, USB 3.0 OTG compliant, e.g., Anker or CableCreation 2-pack).
All systems report nominal with the camp occupied and humming along in the 49°F mist. The gravel driveway served as the primary theater of operations over the last 24 hours, with our vision intelligence analyzing 139 events chronicling the high-energy logistics of family life: a constant ballet of residents, dogs, kayaks, leaf blowers, and a truly remarkable quantity of red gas cans. Notable cameos included a delivery person bearing a large white package and a single wildlife sighting among the day’s standard activities. Supporting this flurry of movement, the digital infrastructure remains robust, with 14 concurrent network leases pulling down a hefty 130.20 GiB via Fidium Fiber, while all wireless access points report green. The core environmental and power matrices are solid, with the server rack holding a crisp 50.00°F and utility power steady at 121.0V under a negligible 4.0% load, keeping the camp’s digital heart beating strong.
Operational heartbeat remains strong and steady with the camp currently occupied. The vision intelligence system analyzed 26 events, primarily documenting the expected flurry of human activity, including driveway traffic, ATV usage, and late-autumn property maintenance. The detection model showed some creative flair, misclassifying subjects as a dog and, ambitiously, a bird, but ground truth confirms all activity was standard for residents and guests. Network performance is robust with 13 concurrent leases active, and Fidium Fiber continues to bear the brunt of data ingress at 41.42 GiB. Under an overcast sky and a current temperature of 35°F, all core infrastructure is healthy; the server rack is maintaining a brisk 38.85°F, powered by a nominal 122.0V utility input with a minimal 2.0% load and a fully charged battery backup.
All systems report nominal on the Maine Camp infrastructure ledger under sunny skies and a brisk 37°F. The property remains occupied, and vision intelligence confirms this with 48 analyzed events dominated by expected human and canine activity around the gravel driveway and a resident pickup truck; no vendor traffic was observed, though local fauna including a fawn and a wild canid made notable appearances. Network traffic is flowing smoothly with Fidium Fiber shouldering the primary load at 7.73 GiB against HughesNet’s 2.43 GiB, distributed across 10 concurrent leases. All access points are healthy, and the server rack is keeping pace with the chilly outdoor temperature at 37.27°F. The entire grid is sustained by stable 121.0V utility power with a minimal 2.0% load, leaving the battery backup fully charged and patiently standing by.
Camp operations are nominal with the family in residence. The last 24 hours remained quiet from a security intelligence perspective; vision systems analyzed two minor events, the most significant being the observation of a red pickup truck on the property, assessed with high confidence as a landscaping or lawn care vendor. All network services are performing as expected with four concurrent leases active, pushing 2.88 GiB over the Fidium fiber while the HughesNet satellite link idled. Amidst the local mist and 41°F temperatures, the core infrastructure matrix is green across the board: all access points are healthy, the server rack is holding at a cool 51.45°F, and the power system is stable under a minimal 2.0% load with battery reserves at 100%.
Maine Camp remains in standby security mode, with operations nominal under overcast skies and a current temperature of 52°F. The day’s sole excitement was a brief visit from a FedEx delivery van at 11:35 AM, an event duly analyzed by our vision intelligence and logged as a routine delivery. All monitoring systems are healthy, with three access points online and four active network leases mapped. The server rack is maintaining a cool 52.12°F, supported by stable utility power at 121.0V with a minimal 2.0% load against a fully charged battery backup. Data traffic remains robust on the primary Fidium Fiber link, which handled 2.02 GiB, while the HughesNet satellite failover saw negligible activity.
The Maine Camp infrastructure remains in a state of quiet vigilance under standby security, with vision intelligence confirming a profoundly uneventful period—the official narrative being “No notable security or activity events recorded across the past 24 hours.” While the local fauna and potential vendors apparently took the day off, the network showed signs of life, pulling 1.24 GiB over the Fidium fiber link to service four concurrent leases. All vital signs are green across the board: access points are online, utility power is stable at 121.0V against a negligible 2.0% load, and the server room is maintaining a brisk 48.20°F in the crisp 33°F morning sun.
