Madigan Point Camp

Author: padric

  • Infrastructure Log: Hardening pfSense USB NUT Telemetry & Remote Monitoring

    Date: October 12, 2026

    Sites Involved: Campfire (campfire.madiganpoint.com – Maine) & Pivot4Port (pivot4port.madiganpoint.com – Stowe)

    Devices: APC Back-UPS Pro RS 1500MS2 (Maine), APC Back-UPS Pro NS 1500M2 (Stowe)

    1. Context & Objective

    The goal of this project was to establish reliable, persistent power telemetry from our primary pfSense firewalls to our Prometheus and Grafana monitoring stack before the winter shutdown.

    • Topology:
      • Maine Firewall (campfire – 192.168.76.1): APC UPS connected via USB to a Protectli vault chassis. Runs the FreeBSD nut package (usbhid-ups and upsd daemon listening on TCP 3493).
      • Collector (bear – 192.168.76.32): Runs nut_exporter in Docker, proxying scrapes between Prometheus and campfire:3493.
      • Metrics Stack: K3s cluster running Prometheus (scraping every 30s) and Grafana visualizing input voltage, battery load, charge, and online state.

    2. Incident: 3:00 AM Telemetry Drop & Root Cause Analysis

    The Failure

    At 03:00 AM, telemetry went completely flat in Grafana. Prometheus reported:

    Plaintext

    server returned HTTP status 500 Internal Server Error
read tcp 10.169.0.4:35626->192.168.76.1:3493: i/o timeout

    Locally on campfire, the pfSense WebGUI reported Alert: The UPS requires attention / Failed to retrieve status. Restarting the NUT service via the GUI did not recover the connection.

    Root Cause

    1. USB Bus Renegotiation: At 3:00 AM (during system log rotation / maintenance), the USB controller momentarily power-cycled its downstream ports, forcing FreeBSD to re-enumerate the bus.
    2. Device Node Drift: The UPS shifted from ugen0.5 down to ugen0.4.
    3. FreeBSD Permission Lockout: FreeBSD created the new character devices (/dev/usb/0.4.0 and /dev/usbctl) with default restrictive permissions (root:operator0644). The unprivileged nut user running usbhid-ups was denied access:
      Plaintextlibusb1: Could not open any HID devices: insufficient permissions on everything
    4. Daemon Deadlock:usbhid-ups died and left a stale socket file in /var/db/nut/. The upsd server hung indefinitely trying to talk to the dead socket, causing incoming TCP 3493 connections from bear to time out.
    5. Why pfSense Backups Don’t Cover This: Standard pfSense XML backups only save WebGUI application configuration (e.g., nut.conf, user definitions, port assignments). They do not back up or restore low-level FreeBSD OS configuration files like /etc/devfs.conf or custom /usr/local/etc/devd/ rules across upgrades or fresh reinstalls.

    3. Runbook: Permanent Fix & Hardening

    To make this configuration resilient against USB disconnects, power bounces, and port re-enumerations, we implemented a dual-layer fix on both campfire (Maine) and pivot4port (Stowe).

    Step 1: Kernel-Level DevFS Persistence (/etc/devfs.conf)

    We appended permissions rules to /etc/devfs.conf so FreeBSD enforces nut:nut and 0666 mode whenever nodes are created or devfs is reloaded:

    Bash

    printf '\n# NUT APC UPS Permissions\nown /dev/ugen* nut:nut\nperm /dev/ugen* 0666\nown /dev/usb/* nut:nut\nperm /dev/usb/* 0666\nown /dev/usbctl nut:nut\nperm /dev/usbctl 0666\n' >> /etc/devfs.conf
/etc/rc.d/devfs restart

    Step 2: Dynamic Hotplug Event Handling (/usr/local/etc/devd/nut_apc.conf)

    Because devices behind USB hubs can re-attach dynamically without triggering a full devfs reload, we added a catch-all devd event rule matching the APC USB Vendor ID (0x051d):

    Bash

    printf 'notify 100 {\n\tmatch "system" "USB";\n\tmatch "subsystem" "DEVICE";\n\tmatch "type" "ATTACH";\n\tmatch "vendor" "0x051d";\n\taction "chown -R nut:nut /dev/ugen* /dev/usb* /dev/usbctl; chmod -R 666 /dev/ugen* /dev/usb* /dev/usbctl";\n};\n' > /usr/local/etc/devd/nut_apc.conf
/etc/rc.d/devd restart

    Step 3: Emergency Recovery Commands (If Ever Needed)

    If a hung process or stale socket locks the service before permissions apply:

    Bash

    killall -9 usbhid-ups upsd upsmon 2>/dev/null
rm -f /var/db/nut/* /var/run/nut/*
/etc/rc.d/devfs restart
/usr/local/etc/rc.d/nut.sh start
upsc <ups_name>@localhost

    4. Grafana Panel PromQL Correction

    During setup, the pfSense Grid Attachment State panel displayed constant red because it was querying output load (network_ups_tools_ups_load) instead of status flags.

    • Incorrect Query:network_ups_tools_ups_load{...} (returned 4–6% load, breaking the binary 0/1 color mapping).
    • Corrected Query:
      Code snippetnetwork_ups_tools_ups_status{instance="camp-pfsense-ups", job="camp-pfsense-ups", flag="OL"} (Returns 1 when Online, 0 when On Battery, mapping cleanly to Dark Green / Dark Red).

    5. Physical Cabling & Interconnect Topology

    Plaintext

    [APC Back-UPS 1500]
       │
       │ (Proprietary RJ50 / 10-pin modular to USB-A cable - APC AP9827)
       ▼
[USB-A Female to USB-C Male Adapter / Dongle]
       │
       ▼
[Protectli Vault Chassis] (Front USB-C Port)

    The APC 1500 units use an RJ50 (10-pin) to USB Type-A data cable (AP9827). Because the Protectli firewall uses USB-C ports on the front I/O, a compact USB-A Female to USB-C Male adapter connects the UPS directly to the chassis, bypassing unnecessary intermediary powered hubs.

    6. Battery Fleet Health Matrix & May 2027 Shopping List

    LocationUPS ModelBattery CartridgeSerial NumberBattery Mfg DateHealth / Action Plan
    Camp Main HouseBack-UPS Pro RS 1500MS2APC RBC1244B2115P175612021/04/16Replace May 2027 (~5.5 yrs old; rapid voltage sag)
    Camp GarageBack-UPS Pro RS 1500MS2APC RBC1240B2352L391962023/12/29Healthy (~2.8 yrs old; ~135 min runtime at 4% load)
    Stowe FirewallBack-UPS Pro NS 1500M2APC RBC1240B2422L113862024/05/28Healthy (~2.4 yrs old; 100% capacity)

    Items to Purchase Before Returning in May:

    1. Replacement Battery Pack:
      • Part:APC RBC124 (Pre-wired cartridge consisting of two 12V 9Ah AGM lead-acid batteries with inline fuse and APC terminal connector).
      • Cost: ~$65 – $75 (OEM equivalent by ExpertPower or UPSBatteryCenter) / ~$95 – $105 (Schneider/APC Original).
    2. Chassis USB Adapter:
      • Part:USB Type-A Female to USB Type-C Male Adapter (low-profile aluminum stub or 6″ pigtail, USB 3.0 OTG compliant, e.g., Anker or CableCreation 2-pack).
      • Cost: ~$8 – $11.

  • Camp Log: Family, Sweat Shirts, and a Fully Restored Camp Network — Memorial Day weekend 2026

    👥 The Crew Manifest

    A full house at the lake this weekend for the holiday stretch. The roster included:

    • Drew O’Brien, his son Jack, and their Goldendoodle, Rosie.
    • Patrick and Dublin (Rosie’s littermate—the doodles had the run of the property).
    • My sister’s boys, Brian and John, who put on a clinic playing poker before packing up to head out late Monday afternoon.
    • Maggie, who joined the fray Saturday afternoon and braved the camp layout before heading back early Monday morning.

    ⚓ Camp Operations & Activities

    Despite a torn right knee meniscus leaving Patrick pretty useless and entirely dependent on the crew to wait on him hand and foot for most everything, the camp team made serious headway getting the property ready for summer.

    The Exit Strategy: The remaining crew is locking down the structures tonight, looking to pull out and head south tomorrow morning, Tuesday, around 10:00 AM.

    Heavy Lifting: Brian and John crushed the seasonal setup, getting all the chairs out onto the farmer’s porch and completely setting up the eastern shore furniture. Meanwhile, Drew and Jack got the fabric top successfully mounted on the “martini” hut down on the eastern shore. Solo Drew got the tarp warp off the eating porch.

    The Camp Chef: Brian stepped up in a major way on Saturday, cooking three straight meals. Breakfast was a massive spread of scrambled eggs; lunch featured red hot dogs with Brian’s custom sautéed red onions; and dinner was a round of hamburgers that he and John went out and bought.

    On the Water: Drew conquered a massive nine-mile kayak loop around the lake. The entire team—including both Goldendoodles—piled onto the pontoon boat for a cruise before the heavy weather rolled through.

    Camp 2.0: The groundwork for Drew and Helena’s new camp officially felt real this weekend. The team took several walks out to the site, and Drew and Jack even charted a path into the raw territory with the new Bronco. Standing on the plot, structural drawings were reviewed and design ideas were debated. One great mystery remains: what street number will this new camp actually get? Up here, matching a plot to an address is a total black art left entirely in the hands of the local fire department.

    The Weather: Memorial Day didn’t cooperate. The clouds opened up and it rained hard across the point on Monday, prompting Maggie’s early departure around 9:00 AM.

    ⚡ Infrastructure Status Report: “Tech Corner” Fully Restored

    After an intensive troubleshooting pipeline over the site-to-site network link, the camp’s primary infrastructure layer is completely stabilized. WiFi has been updated, all Access Points are functioning. Both the main ISP (Fidium) and the back up ISP (Hughes Net) are working