197 lines
14 KiB
Markdown
197 lines
14 KiB
Markdown
# m3u8-downloader
|
|
|
|
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
|
|
|
|
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
|
|
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
|
|
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
|
|
|
|
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
|
|
|
|
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
|
|
|
|
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
|
|
|
|
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
|
|
|
|
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
|
|
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
|
|
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
|
|
│ └──────────────┘ │
|
|
└──────────────┬─────────────────────────────────┘
|
|
▼
|
|
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
|
|
```
|
|
|
|
### Download — `pkg/media`
|
|
|
|
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
|
|
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
|
|
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
|
|
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
|
|
|
|
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
|
|
|
|
### Transfer — `pkg/transfer` + `pkg/nas`
|
|
|
|
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
|
|
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
|
|
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
|
|
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification.
|
|
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
|
|
|
|
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
|
|
|
|
### Processing — `pkg/processing`
|
|
|
|
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
|
|
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
|
|
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
|
|
|
|
### Configuration — `pkg/config` + `pkg/constants`
|
|
|
|
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
|
|
|
|
### Logging — `pkg/logging`
|
|
|
|
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
|
|
|
|
### Health / status — `pkg/health`
|
|
|
|
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
|
|
|
|
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
|
|
- `GET /healthz` — plain liveness check.
|
|
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
|
|
|
|
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
|
|
|
|
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
|
|
|
|
## Configuration
|
|
|
|
All settings have sane defaults; override via environment variables as needed.
|
|
|
|
| Env var | Default | Meaning |
|
|
|---|---|---|
|
|
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
|
|
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
|
|
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
|
|
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to |
|
|
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials |
|
|
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
|
|
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
|
|
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
|
|
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
|
|
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
|
|
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
|
|
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
|
|
|
|
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
|
|
|
|
## Building
|
|
|
|
```bash
|
|
go build -o m3u8-downloader ./cmd/main
|
|
# or, via the Makefile (produces stream-recorder(.exe)):
|
|
make build
|
|
```
|
|
|
|
## Usage
|
|
|
|
All modes are invoked through the single `cmd/main` binary:
|
|
|
|
```
|
|
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process] [-check]
|
|
```
|
|
|
|
| Flag | Applies to | Meaning |
|
|
|---|---|---|
|
|
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
|
|
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
|
|
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
|
|
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
|
|
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
|
|
| `-check` | connection-check mode | Tests connectivity to everything a recording depends on, then exits without downloading or transferring anything. Exit code is 0 only if no check failed. |
|
|
|
|
### Which command for which situation
|
|
|
|
**Verifying the setup before an event** (in the spirit of `dbt debug` — proves the stream host and NAS are reachable without recording anything):
|
|
```bash
|
|
m3u8-downloader -check -url "https://example.com/live/master.m3u8"
|
|
```
|
|
It runs three checks and prints pass/fail for each:
|
|
|
|
- **configuration** — config loads, and the local/process/log directories resolve and are creatable.
|
|
- **stream playlist** — only when `-url` is given (skipped otherwise). Fetches the master playlist with the same User-Agent and Referer the downloader uses, requires HTTP 200, decodes it, and lists the variants (and the resolution labels that would become output directories) it found.
|
|
- **nas storage** — only when `ENABLE_NAS_TRANSFER=true` (skipped otherwise). Runs the same connect → mkdir → write-probe sequence the transfer service does at startup, bounded by the NAS timeout so an unresponsive share reports a timeout instead of hanging.
|
|
|
|
Skipped checks don't fail the run; the exit code is 1 if any check actually failed, so this can gate a scripted start:
|
|
```bash
|
|
m3u8-downloader -check -url "$URL" && m3u8-downloader -url "$URL" -event "2026-regionals"
|
|
```
|
|
|
|
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
|
|
```bash
|
|
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
|
|
```
|
|
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
|
|
|
|
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
|
|
```bash
|
|
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
|
|
```
|
|
|
|
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
|
|
```bash
|
|
m3u8-downloader -event "2026-regionals" -transfer
|
|
```
|
|
|
|
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
|
|
```bash
|
|
m3u8-downloader -event "2026-regionals" -process
|
|
```
|
|
|
|
**Checking on a running instance** (from any device on the same network as the recorder):
|
|
```
|
|
http://<recorder-host>:8090/ # dashboard
|
|
http://<recorder-host>:8090/status # raw JSON
|
|
```
|
|
|
|
## Data layout
|
|
|
|
```
|
|
data/
|
|
├── {event}.json # download manifest (sequence numbers + resolutions)
|
|
├── {event}/
|
|
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
|
|
│ ├── 720p/
|
|
│ └── ...
|
|
├── logs/
|
|
│ ├── app.log # per-subsystem structured JSON logs
|
|
│ ├── download.log
|
|
│ ├── transfer.log
|
|
│ └── processing.log
|
|
└── transfer_queue.json # persisted transfer queue state
|
|
|
|
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
|
|
├── 1080p/
|
|
└── ...
|
|
|
|
out/
|
|
└── {event}/
|
|
└── {event}.mp4 # final FFmpeg output (process-only mode)
|
|
```
|
|
|
|
## More documentation
|
|
|
|
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
|
|
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
|
|
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.
|