StreamRecorder/README.md
Aaron e2566fdefa
Some checks failed
Build Go Binaries / build (push) Has been cancelled
Claude bugfixes
2026-08-07 03:49:37 +00:00

218 lines
17 KiB
Markdown

# m3u8-downloader
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
## Architecture
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
│ └──────────────┘ │
└──────────────┬─────────────────────────────────┘
▼
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
```
### Download — `pkg/media`
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
### Transfer — `pkg/transfer` + `pkg/nas`
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
- `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux).
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
### Processing — `pkg/processing`
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
### Configuration — `pkg/config` + `pkg/constants`
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
### Logging — `pkg/logging`
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
### Health / status — `pkg/health`
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
- `GET /healthz` — plain liveness check.
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
## Configuration
All settings have sane defaults; override via environment variables as needed.
| Env var | Default | Meaning |
|---|---|---|
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path |
| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root |
| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder/<server>/<share>`) |
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
## NAS connectivity (Windows and Linux)
The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
| Host | Destination | What happens |
|---|---|---|
| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly |
| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead |
| Either | A local or already-mounted directory | Used as-is, no network step |
Notes:
- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins.
- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone.
- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password.
- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
Run `-check` to see which strategy applies and where the destination resolved to.
## Building
```bash
go build -o m3u8-downloader ./cmd/main
# or, via the Makefile (produces stream-recorder(.exe)):
make build
```
## Usage
All modes are invoked through the single `cmd/main` binary:
```
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process] [-check]
```
| Flag | Applies to | Meaning |
|---|---|---|
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
| `-check` | connection-check mode | Tests connectivity to everything a recording depends on, then exits without downloading or transferring anything. Exit code is 0 only if no check failed. |
### Which command for which situation
**Verifying the setup before an event** (in the spirit of `dbt debug` — proves the stream host and NAS are reachable without recording anything):
```bash
m3u8-downloader -check -url "https://example.com/live/master.m3u8"
```
It runs three checks and prints pass/fail for each:
- **configuration** — config loads, and the local/process/log directories resolve and are creatable.
- **stream playlist** — only when `-url` is given (skipped otherwise). Fetches the master playlist with the same User-Agent and Referer the downloader uses, requires HTTP 200, decodes it, and lists the variants (and the resolution labels that would become output directories) it found.
- **nas storage** — only when `ENABLE_NAS_TRANSFER=true` (skipped otherwise). Runs the same connect → mkdir → write-probe sequence the transfer service does at startup, bounded by the NAS timeout so an unresponsive share reports a timeout instead of hanging.
Skipped checks don't fail the run; the exit code is 1 if any check actually failed, so this can gate a scripted start:
```bash
m3u8-downloader -check -url "$URL" && m3u8-downloader -url "$URL" -event "2026-regionals"
```
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
```
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
```
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
```bash
m3u8-downloader -event "2026-regionals" -transfer
```
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
```bash
m3u8-downloader -event "2026-regionals" -process
```
**Checking on a running instance** (from any device on the same network as the recorder):
```
http://<recorder-host>:8090/ # dashboard
http://<recorder-host>:8090/status # raw JSON
```
## Data layout
```
data/
├── {event}.json # download manifest (sequence numbers + resolutions)
├── {event}/
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
│ ├── 720p/
│ └── ...
├── logs/
│ ├── app.log # per-subsystem structured JSON logs
│ ├── download.log
│ ├── transfer.log
│ └── processing.log
└── transfer_queue.json # persisted transfer queue state
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
├── 1080p/
└── ...
out/
└── {event}/
└── {event}.mp4 # final FFmpeg output (process-only mode)
```
## More documentation
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.