17 KiB
m3u8-downloader
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master .m3u8 playlist URL and it will:
- Download every quality variant (1080p, 720p, ...) in real time as new segments appear.
- Transfer finished segments to NAS storage for archival, with retry logic and local cleanup.
- Process the archived segments into a single playable
.mp4per event via FFmpeg.
Download and transfer run together, in one invocation. When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
Processing is a separate, later invocation (see Usage below), because it reads finished segments from NAS, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
There's also a standalone -transfer mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
Project docs also refer to this as "StreamRecorder" (see
DEPLOYMENT.md/TESTING.md) — same project, the Go module is namedm3u8-downloader.
Architecture
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
│ └──────────────┘ │
└──────────────┬─────────────────────────────────┘
▼
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
Download — pkg/media
stream.go:GetAllVariants()fetches and parses the master playlist, extracting every quality variant.VariantDownloader()runs one per-variant polling loop (everyCore.RefreshDelay, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (Core.WorkerCountper variant, default 4).playlist.go:LoadMediaPlaylist()fetches and parses an individual variant's chunklist.segment.go:DownloadSegment()downloads one.tsfile, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).manifest.go:ManifestWritertracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
Segments land on disk under Paths.LocalOutput/<event>/<resolution>/.
Transfer — pkg/transfer + pkg/nas
service.go:TransferServicewires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.watcher.go:FileWatcher(viafsnotify) watches the local event directory for new.tsfiles and queues them after a short settling delay (Transfer.FileSettlingDelay, default 5s), so a file isn't queued while still being written.queue.go:TransferQueueis a priority-queued worker pool (Transfer.WorkerCount, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state totransfer_queue.jsonso it survives a restart.nas.go/pkg/nas:NASServiceattaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See NAS connectivity.cleanup.go:CleanupServicedeletes local copies after a successful, verified transfer (Cleanup.AfterTransfer, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone -transfer mode exists only to catch up files that a prior download session ended without fully transferring.
Processing — pkg/processing
ProcessingServicereads segments from NAS (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one.mp4.- This is why processing is normally the last step, run after transfer has fully drained — it only sees what's actually on NAS.
- Requires a working
ffmpegonPATH(orProcessing.FFmpegPath/FFMPEG_PATH), and connects to NAS on startup regardless of whetherProcessing.Enabledis set.
Configuration — pkg/config + pkg/constants
pkg/config/config.go defines defaults and environment-variable overrides for every setting below. pkg/constants.MustGetConfig() lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
Logging — pkg/logging
A small factory (modeled after a structured-logging pattern used in another project) that hands out one log/slog JSON logger per pipeline stage: App, Download, Transfer, Processing. Each writes to its own file under Logging.Dir (default data/logs/{app,download,transfer,processing}.log). This is additive — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
Health / status — pkg/health
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
GET /status— full JSON snapshot,status: "healthy"or"degraded"(degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).GET /healthz— plain liveness check.GET /— a small auto-refreshing dashboard (polls/statusevery 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
Important scope note: the health server only reflects whatever is live in the current process. During a normal download run, /status shows real-time download + transfer state together (they're the same process) with processing reported as idle. Running -process afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
By default the health server binds all network interfaces (:8090) so it can be checked from another device on the same LAN (e.g. a phone) during an event. There is no authentication or TLS — this is fine on a trusted local network only, never expose it to the open internet.
Configuration
All settings have sane defaults; override via environment variables as needed.
| Env var | Default | Meaning |
|---|---|---|
WORKER_COUNT |
4 |
Concurrent segment downloads per variant |
REFRESH_DELAY_SECONDS |
3 |
How often each variant's chunklist is polled |
ENABLE_NAS_TRANSFER |
true |
Whether to run the transfer service alongside download |
NAS_OUTPUT_PATH |
\\HomeLabNAS\dci\streams |
NAS root segments are copied to. UNC (\\server\share\dir), //server/share/dir, smb://server/share/dir, or a local/mounted directory — see NAS connectivity |
NAS_USERNAME / NAS_PASSWORD |
(see note below) | NAS auth credentials. Used by net use on Windows and by NAS_AUTO_MOUNT on Linux; ignored for an already-mounted path |
NAS_AUTO_MOUNT |
false |
Linux only: mount the SMB share if the host hasn't. Requires root |
NAS_MOUNT_POINT |
(derived) | Where NAS_AUTO_MOUNT attaches the share (default /mnt/streamrecorder/<server>/<share>) |
LOCAL_OUTPUT_DIR |
data |
Local base directory for downloaded segments |
PROCESS_OUTPUT_DIR |
out |
Output directory for the final concatenated .mp4 |
FFMPEG_PATH |
ffmpeg |
Path to the FFmpeg binary used by the processing stage |
LOG_DIR |
data/logs |
Directory for the per-subsystem JSON log files |
LOG_LEVEL |
info |
debug | info | warn | error (also forced to debug by the -debug flag) |
HEALTH_ENABLED |
true |
Whether to start the health/status HTTP server |
HEALTH_ADDR |
:8090 |
Bind address for the health server + dashboard |
Security note: the default NAS credentials currently ship as hardcoded defaults in
pkg/config/config.gofor this home-lab setup. If you fork or redeploy this, setNAS_USERNAME/NAS_PASSWORDexplicitly and rotate the shipped default — seeTODO.mdfor the full flagged list of housekeeping items like this one.
NAS connectivity (Windows and Linux)
The same NAS_OUTPUT_PATH works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
| Host | Destination | What happens |
|---|---|---|
| Windows | UNC / //server/share / smb:// |
net use \\server\share with NAS_USERNAME/NAS_PASSWORD, then the UNC path is used directly |
| Linux | UNC / //server/share / smb:// |
The share is looked up in /proc/mounts and the UNC path is rewritten onto the mount point (\\nas\dci\streams + //nas/dci mounted at /mnt/dci → /mnt/dci/streams). Set NAS_AUTO_MOUNT=true to mount it instead |
| Either | A local or already-mounted directory | Used as-is, no network step |
Notes:
- Server and share names match case-insensitively, and both
cifs/smb3andnfs/nfs4mounts are recognised. If the share is mounted at a deeper level than configured (//nas/dci/streamsat/mnt/streams), the deepest matching mount wins. - Nothing the host already set up is torn down. A share found already mounted (or already connected via
net use) is left alone at shutdown; only a mount or connection this process created is undone. NAS_AUTO_MOUNTrequires root, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either pointNAS_OUTPUT_PATHat the mount point or leave it as the UNC path and let it resolve. Credentials are passed tomount.cifsthrough a0600credentials file, never on the command line where the process table would expose the password.- A Windows-shaped path on Linux is rejected, not silently created. Backslashes and drive letters are legal filename characters on Linux, so
C:\dataor an unresolved\\nas\dciwould otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
Run -check to see which strategy applies and where the destination resolved to.
Building
go build -o m3u8-downloader ./cmd/main
# or, via the Makefile (produces stream-recorder(.exe)):
make build
Usage
All modes are invoked through the single cmd/main binary:
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process] [-check]
| Flag | Applies to | Meaning |
|---|---|---|
-url |
download mode | Master .m3u8 playlist URL. If omitted (and neither -transfer nor -process is set), you'll be prompted for it interactively. |
-event |
all modes | Event name — used to namespace local output, NAS destination, and the manifest file. Always set this explicitly; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In -transfer/-process mode, omitting it shows an interactive picker over existing event directories. |
-debug |
download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to debug. |
-transfer |
transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires ENABLE_NAS_TRANSFER=true. |
-process |
process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single .mp4 via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
-check |
connection-check mode | Tests connectivity to everything a recording depends on, then exits without downloading or transferring anything. Exit code is 0 only if no check failed. |
Which command for which situation
Verifying the setup before an event (in the spirit of dbt debug — proves the stream host and NAS are reachable without recording anything):
m3u8-downloader -check -url "https://example.com/live/master.m3u8"
It runs three checks and prints pass/fail for each:
- configuration — config loads, and the local/process/log directories resolve and are creatable.
- stream playlist — only when
-urlis given (skipped otherwise). Fetches the master playlist with the same User-Agent and Referer the downloader uses, requires HTTP 200, decodes it, and lists the variants (and the resolution labels that would become output directories) it found. - nas storage — only when
ENABLE_NAS_TRANSFER=true(skipped otherwise). Runs the same connect → mkdir → write-probe sequence the transfer service does at startup, bounded by the NAS timeout so an unresponsive share reports a timeout instead of hanging.
Skipped checks don't fail the run; the exit code is 1 if any check actually failed, so this can gate a scripted start:
m3u8-downloader -check -url "$URL" && m3u8-downloader -url "$URL" -event "2026-regionals"
Recording a live event (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
Quick test/dry run (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
Catching up NAS transfer after the fact (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
m3u8-downloader -event "2026-regionals" -transfer
Producing the final video (after transfer has fully drained — check /status or the NAS directory to confirm nothing's still pending):
m3u8-downloader -event "2026-regionals" -process
Checking on a running instance (from any device on the same network as the recorder):
http://<recorder-host>:8090/ # dashboard
http://<recorder-host>:8090/status # raw JSON
Data layout
data/
├── {event}.json # download manifest (sequence numbers + resolutions)
├── {event}/
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
│ ├── 720p/
│ └── ...
├── logs/
│ ├── app.log # per-subsystem structured JSON logs
│ ├── download.log
│ ├── transfer.log
│ └── processing.log
└── transfer_queue.json # persisted transfer queue state
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
├── 1080p/
└── ...
out/
└── {event}/
└── {event}.mp4 # final FFmpeg output (process-only mode)
More documentation
DEPLOYMENT.md— Docker/Kubernetes deployment examples and production considerations.TESTING.md— test suite layout and how to run it (make test,make test-coverage, etc.).TODO.md— known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.