StreamRecorder/README.md
Aaron e2566fdefa
Some checks failed
Build Go Binaries / build (push) Has been cancelled
Claude bugfixes
2026-08-07 03:49:37 +00:00

17 KiB

m3u8-downloader

A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master .m3u8 playlist URL and it will:

  1. Download every quality variant (1080p, 720p, ...) in real time as new segments appear.
  2. Transfer finished segments to NAS storage for archival, with retry logic and local cleanup.
  3. Process the archived segments into a single playable .mp4 per event via FFmpeg.

Download and transfer run together, in one invocation. When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.

Processing is a separate, later invocation (see Usage below), because it reads finished segments from NAS, after transfer has fully drained — it can't run concurrently with the other two in any useful way.

There's also a standalone -transfer mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).

The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.

Project docs also refer to this as "StreamRecorder" (see DEPLOYMENT.md/TESTING.md) — same project, the Go module is named m3u8-downloader.

Architecture

                 ┌──────────────┐        ┌──────────────┐        ┌──────────────┐
   .m3u8 URL ──▶ │   DOWNLOAD   │──────▶ │   TRANSFER   │──────▶ │  PROCESSING  │──▶ event.mp4
                 │  (pkg/media) │  .ts   │(pkg/transfer,│  NAS   │(pkg/processing)
                 └──────────────┘ files  │   pkg/nas)   │ files  └──────────────┘
                        │                └──────────────┘               │
                        └──────────────┬─────────────────────────────────┘
                                        ▼
                          pkg/logging (JSON files)  +  pkg/health (HTTP /status, dashboard)

Download — pkg/media

  • stream.go: GetAllVariants() fetches and parses the master playlist, extracting every quality variant. VariantDownloader() runs one per-variant polling loop (every Core.RefreshDelay, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (Core.WorkerCount per variant, default 4).
  • playlist.go: LoadMediaPlaylist() fetches and parses an individual variant's chunklist.
  • segment.go: DownloadSegment() downloads one .ts file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
  • manifest.go: ManifestWriter tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.

Segments land on disk under Paths.LocalOutput/<event>/<resolution>/.

Transfer — pkg/transfer + pkg/nas

  • service.go: TransferService wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
  • watcher.go: FileWatcher (via fsnotify) watches the local event directory for new .ts files and queues them after a short settling delay (Transfer.FileSettlingDelay, default 5s), so a file isn't queued while still being written.
  • queue.go: TransferQueue is a priority-queued worker pool (Transfer.WorkerCount, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to transfer_queue.json so it survives a restart.
  • nas.go / pkg/nas: NASService attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See NAS connectivity.
  • cleanup.go: CleanupService deletes local copies after a successful, verified transfer (Cleanup.AfterTransfer, default true).

During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone -transfer mode exists only to catch up files that a prior download session ended without fully transferring.

Processing — pkg/processing

  • ProcessingService reads segments from NAS (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one .mp4.
  • This is why processing is normally the last step, run after transfer has fully drained — it only sees what's actually on NAS.
  • Requires a working ffmpeg on PATH (or Processing.FFmpegPath / FFMPEG_PATH), and connects to NAS on startup regardless of whether Processing.Enabled is set.

Configuration — pkg/config + pkg/constants

pkg/config/config.go defines defaults and environment-variable overrides for every setting below. pkg/constants.MustGetConfig() lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.

Logging — pkg/logging

A small factory (modeled after a structured-logging pattern used in another project) that hands out one log/slog JSON logger per pipeline stage: App, Download, Transfer, Processing. Each writes to its own file under Logging.Dir (default data/logs/{app,download,transfer,processing}.log). This is additive — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.

Health / status — pkg/health

An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:

  • GET /status — full JSON snapshot, status: "healthy" or "degraded" (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
  • GET /healthz — plain liveness check.
  • GET / — a small auto-refreshing dashboard (polls /status every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.

Important scope note: the health server only reflects whatever is live in the current process. During a normal download run, /status shows real-time download + transfer state together (they're the same process) with processing reported as idle. Running -process afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.

By default the health server binds all network interfaces (:8090) so it can be checked from another device on the same LAN (e.g. a phone) during an event. There is no authentication or TLS — this is fine on a trusted local network only, never expose it to the open internet.

Configuration

All settings have sane defaults; override via environment variables as needed.

Env var Default Meaning
WORKER_COUNT 4 Concurrent segment downloads per variant
REFRESH_DELAY_SECONDS 3 How often each variant's chunklist is polled
ENABLE_NAS_TRANSFER true Whether to run the transfer service alongside download
NAS_OUTPUT_PATH \\HomeLabNAS\dci\streams NAS root segments are copied to. UNC (\\server\share\dir), //server/share/dir, smb://server/share/dir, or a local/mounted directory — see NAS connectivity
NAS_USERNAME / NAS_PASSWORD (see note below) NAS auth credentials. Used by net use on Windows and by NAS_AUTO_MOUNT on Linux; ignored for an already-mounted path
NAS_AUTO_MOUNT false Linux only: mount the SMB share if the host hasn't. Requires root
NAS_MOUNT_POINT (derived) Where NAS_AUTO_MOUNT attaches the share (default /mnt/streamrecorder/<server>/<share>)
LOCAL_OUTPUT_DIR data Local base directory for downloaded segments
PROCESS_OUTPUT_DIR out Output directory for the final concatenated .mp4
FFMPEG_PATH ffmpeg Path to the FFmpeg binary used by the processing stage
LOG_DIR data/logs Directory for the per-subsystem JSON log files
LOG_LEVEL info debug | info | warn | error (also forced to debug by the -debug flag)
HEALTH_ENABLED true Whether to start the health/status HTTP server
HEALTH_ADDR :8090 Bind address for the health server + dashboard

Security note: the default NAS credentials currently ship as hardcoded defaults in pkg/config/config.go for this home-lab setup. If you fork or redeploy this, set NAS_USERNAME/NAS_PASSWORD explicitly and rotate the shipped default — see TODO.md for the full flagged list of housekeeping items like this one.

NAS connectivity (Windows and Linux)

The same NAS_OUTPUT_PATH works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.

Host Destination What happens
Windows UNC / //server/share / smb:// net use \\server\share with NAS_USERNAME/NAS_PASSWORD, then the UNC path is used directly
Linux UNC / //server/share / smb:// The share is looked up in /proc/mounts and the UNC path is rewritten onto the mount point (\\nas\dci\streams + //nas/dci mounted at /mnt/dci → /mnt/dci/streams). Set NAS_AUTO_MOUNT=true to mount it instead
Either A local or already-mounted directory Used as-is, no network step

Notes:

  • Server and share names match case-insensitively, and both cifs/smb3 and nfs/nfs4 mounts are recognised. If the share is mounted at a deeper level than configured (//nas/dci/streams at /mnt/streams), the deepest matching mount wins.
  • Nothing the host already set up is torn down. A share found already mounted (or already connected via net use) is left alone at shutdown; only a mount or connection this process created is undone.
  • NAS_AUTO_MOUNT requires root, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point NAS_OUTPUT_PATH at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to mount.cifs through a 0600 credentials file, never on the command line where the process table would expose the password.
  • A Windows-shaped path on Linux is rejected, not silently created. Backslashes and drive letters are legal filename characters on Linux, so C:\data or an unresolved \\nas\dci would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.

Run -check to see which strategy applies and where the destination resolved to.

Building

go build -o m3u8-downloader ./cmd/main
# or, via the Makefile (produces stream-recorder(.exe)):
make build

Usage

All modes are invoked through the single cmd/main binary:

m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process] [-check]
Flag Applies to Meaning
-url download mode Master .m3u8 playlist URL. If omitted (and neither -transfer nor -process is set), you'll be prompted for it interactively.
-event all modes Event name — used to namespace local output, NAS destination, and the manifest file. Always set this explicitly; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In -transfer/-process mode, omitting it shows an interactive picker over existing event directories.
-debug download mode Restricts download to the 1080p variant only (for faster iteration) and forces log level to debug.
-transfer transfer-only mode Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires ENABLE_NAS_TRANSFER=true.
-process process-only mode Reads an event's segments from NAS and concatenates the highest-quality ones into a single .mp4 via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config.
-check connection-check mode Tests connectivity to everything a recording depends on, then exits without downloading or transferring anything. Exit code is 0 only if no check failed.

Which command for which situation

Verifying the setup before an event (in the spirit of dbt debug — proves the stream host and NAS are reachable without recording anything):

m3u8-downloader -check -url "https://example.com/live/master.m3u8"

It runs three checks and prints pass/fail for each:

  • configuration — config loads, and the local/process/log directories resolve and are creatable.
  • stream playlist — only when -url is given (skipped otherwise). Fetches the master playlist with the same User-Agent and Referer the downloader uses, requires HTTP 200, decodes it, and lists the variants (and the resolution labels that would become output directories) it found.
  • nas storage — only when ENABLE_NAS_TRANSFER=true (skipped otherwise). Runs the same connect → mkdir → write-probe sequence the transfer service does at startup, bounded by the NAS timeout so an unresponsive share reports a timeout instead of hanging.

Skipped checks don't fail the run; the exit code is 1 if any check actually failed, so this can gate a scripted start:

m3u8-downloader -check -url "$URL" && m3u8-downloader -url "$URL" -event "2026-regionals"

Recording a live event (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):

m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"

Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.

Quick test/dry run (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):

m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug

Catching up NAS transfer after the fact (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):

m3u8-downloader -event "2026-regionals" -transfer

Producing the final video (after transfer has fully drained — check /status or the NAS directory to confirm nothing's still pending):

m3u8-downloader -event "2026-regionals" -process

Checking on a running instance (from any device on the same network as the recorder):

http://<recorder-host>:8090/          # dashboard
http://<recorder-host>:8090/status    # raw JSON

Data layout

data/
├── {event}.json              # download manifest (sequence numbers + resolutions)
├── {event}/
│   ├── 1080p/                # downloaded segments, before NAS transfer + cleanup
│   ├── 720p/
│   └── ...
├── logs/
│   ├── app.log                # per-subsystem structured JSON logs
│   ├── download.log
│   ├── transfer.log
│   └── processing.log
└── transfer_queue.json       # persisted transfer queue state

\\<NAS>\...\{event}\          # mirrors the local structure once transferred
    ├── 1080p/
    └── ...

out/
└── {event}/
    └── {event}.mp4           # final FFmpeg output (process-only mode)

More documentation

  • DEPLOYMENT.md — Docker/Kubernetes deployment examples and production considerations.
  • TESTING.md — test suite layout and how to run it (make test, make test-coverage, etc.).
  • TODO.md — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.