Health/status http endpoint, logging
Some checks failed
Build Go Binaries / build (push) Has been cancelled

This commit is contained in:
kacarmichael 2026-08-06 01:24:29 -05:00
parent ce2c5f3181
commit 309eeef59a
20 changed files with 1239 additions and 10 deletions

180
README.md Normal file
View File

@ -0,0 +1,180 @@
# m3u8-downloader
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
## Architecture
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
│ └──────────────┘ │
└──────────────┬─────────────────────────────────┘
▼
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
```
### Download — `pkg/media`
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
### Transfer — `pkg/transfer` + `pkg/nas`
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification.
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
### Processing — `pkg/processing`
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
### Configuration — `pkg/config` + `pkg/constants`
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
### Logging — `pkg/logging`
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
### Health / status — `pkg/health`
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
- `GET /healthz` — plain liveness check.
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
## Configuration
All settings have sane defaults; override via environment variables as needed.
| Env var | Default | Meaning |
|---|---|---|
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials |
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
## Building
```bash
go build -o m3u8-downloader ./cmd/main
# or, via the Makefile (produces stream-recorder(.exe)):
make build
```
## Usage
All modes are invoked through the single `cmd/main` binary:
```
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process]
```
| Flag | Applies to | Meaning |
|---|---|---|
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
### Which command for which situation
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
```
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
```
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
```bash
m3u8-downloader -event "2026-regionals" -transfer
```
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
```bash
m3u8-downloader -event "2026-regionals" -process
```
**Checking on a running instance** (from any device on the same network as the recorder):
```
http://<recorder-host>:8090/ # dashboard
http://<recorder-host>:8090/status # raw JSON
```
## Data layout
```
data/
├── {event}.json # download manifest (sequence numbers + resolutions)
├── {event}/
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
│ ├── 720p/
│ └── ...
├── logs/
│ ├── app.log # per-subsystem structured JSON logs
│ ├── download.log
│ ├── transfer.log
│ └── processing.log
└── transfer_queue.json # persisted transfer queue state
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
├── 1080p/
└── ...
out/
└── {event}/
└── {event}.mp4 # final FFmpeg output (process-only mode)
```
## More documentation
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.

View File

@ -4,6 +4,8 @@ import (
"context"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/media"
"m3u8-downloader/pkg/transfer"
"m3u8-downloader/pkg/utils"
@ -15,6 +17,8 @@ import (
)
func Download(masterURL string, eventName string, debug bool) {
appLog := logging.MustGetFactory().App
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
@ -24,11 +28,26 @@ func Download(masterURL string, eventName string, debug bool) {
go func() {
<-sigChan
log.Println("Shutting down...")
appLog.Info("shutdown signal received")
cancel()
}()
cfg := constants.MustGetConfig()
if cfg.Health.Enabled {
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
var wg sync.WaitGroup
var transferService *transfer.TransferService
if cfg.NAS.EnableTransfer {
@ -36,6 +55,7 @@ func Download(masterURL string, eventName string, debug bool) {
if err != nil {
log.Printf("Failed to create transfer service: %v", err)
log.Println("Continuing without transfer service...")
appLog.Error("failed to create transfer service, continuing without it", logging.KeyError, err.Error())
} else {
transferService = ts
wg.Add(1)
@ -43,9 +63,11 @@ func Download(masterURL string, eventName string, debug bool) {
defer wg.Done()
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err)
appLog.Error("transfer service error", logging.KeyError, err.Error())
}
}()
log.Println("Transfer service started.")
appLog.Info("transfer service started")
}
}
@ -53,11 +75,13 @@ func Download(masterURL string, eventName string, debug bool) {
eventPath := cfg.GetEventPath(eventName)
if err := utils.EnsureDir(eventPath); err != nil {
appLog.Error("failed to create event directory", logging.KeyError, err.Error())
log.Fatalf("Failed to create event directory: %v", err)
}
variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter)
if err != nil {
appLog.Error("failed to get variants", logging.KeyError, err.Error())
log.Fatalf("Failed to get variants: %v", err)
}
log.Printf("Found %d variants", len(variants))
@ -82,6 +106,7 @@ func Download(masterURL string, eventName string, debug bool) {
wg.Wait()
log.Println("All variant downloaders finished.")
appLog.Info("all variant downloaders finished")
if transferService != nil {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
@ -90,7 +115,9 @@ func Download(masterURL string, eventName string, debug bool) {
}
log.Println("All Services shut down.")
appLog.Info("all services shut down")
manifestWriter.WriteManifest()
log.Println("Manifest written.")
appLog.Info("manifest written", logging.KeyEvent, eventName)
}

View File

@ -7,6 +7,8 @@ import (
"m3u8-downloader/cmd/downloader"
"m3u8-downloader/cmd/processor"
"m3u8-downloader/cmd/transfer"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"os"
"strings"
)
@ -20,6 +22,14 @@ func main() {
flag.Parse()
// Must happen before the first logging.MustGetFactory() call anywhere in
// the program, since the logging factory reads its level from this
// config singleton once and caches it.
if *debug {
constants.MustGetConfig().Logging.Level = "debug"
}
logging.MustGetFactory().App.Info("m3u8-downloader starting", "debug", *debug)
if *transferOnly {
transfer.RunTransferOnly(*eventName)
return

View File

@ -4,17 +4,39 @@ import (
"context"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/processing"
"time"
)
func Process(eventName string) {
appLog := logging.MustGetFactory().App
log.Printf("Starting processing for event: %s", eventName)
appLog.Info("starting processing", logging.KeyEvent, eventName)
cfg := constants.MustGetConfig()
if cfg.Health.Enabled {
healthSrv, err := health.Start(context.Background(), &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
ps, err := processing.NewProcessingService(eventName, cfg)
if err != nil {
appLog.Error("failed to create processing service", logging.KeyError, err.Error())
log.Fatalf("Failed to create processing service: %v", err)
}
if err := ps.Start(context.Background()); err != nil {
appLog.Error("processing service failed", logging.KeyError, err.Error())
log.Fatalf("Failed to run processing service: %v", err)
}
}

View File

@ -7,6 +7,8 @@ import (
"log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/transfer"
"m3u8-downloader/pkg/utils"
"os"
@ -32,10 +34,12 @@ func getEventDirs(cfg *config.Config) ([]string, error) {
}
func RunTransferOnly(eventName string) {
appLog := logging.MustGetFactory().App
cfg := constants.MustGetConfig()
// Check if NAS transfer is enabled
if !cfg.NAS.EnableTransfer {
appLog.Error("NAS transfer disabled, cannot run transfer-only mode")
log.Fatal("NAS transfer is disabled in configuration. Please enable it to use transfer-only mode.")
}
@ -79,30 +83,50 @@ func RunTransferOnly(eventName string) {
go func() {
<-sigChan
log.Println("Shutting down transfer service...")
appLog.Info("shutdown signal received")
cancel()
}()
if cfg.Health.Enabled {
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
// Verify local event directory exists
localEventPath := cfg.GetEventPath(eventName)
if !utils.PathExists(localEventPath) {
appLog.Error("local event directory does not exist", logging.KeyEvent, eventName)
log.Fatalf("Local event directory does not exist: %s", localEventPath)
}
// Create transfer service
transferService, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
if err != nil {
appLog.Error("failed to create transfer service", logging.KeyError, err.Error())
log.Fatalf("Failed to create transfer service: %v", err)
}
// Find and queue existing files
if err := transferService.QueueExistingFiles(localEventPath); err != nil {
appLog.Error("failed to queue existing files", logging.KeyError, err.Error())
log.Fatalf("Failed to queue existing files: %v", err)
}
// Start transfer service
log.Println("Starting transfer service...")
appLog.Info("starting transfer service", logging.KeyEvent, eventName)
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err)
appLog.Error("transfer service error", logging.KeyError, err.Error())
}
// Graceful shutdown
@ -111,4 +135,5 @@ func RunTransferOnly(eventName string) {
transferService.Shutdown(shutdownCtx)
log.Println("Transfer-only mode completed.")
appLog.Info("transfer-only mode completed", logging.KeyEvent, eventName)
}

View File

@ -16,6 +16,8 @@ type Config struct {
Transfer TransferConfig
Cleanup CleanupConfig
Paths PathsConfig
Logging LoggingConfig
Health HealthConfig
}
type CoreConfig struct {
@ -67,6 +69,16 @@ type PathsConfig struct {
PersistenceFile string
}
type LoggingConfig struct {
Dir string
Level string
}
type HealthConfig struct {
Enabled bool
Addr string
}
var defaultConfig = Config{
Core: CoreConfig{
WorkerCount: 4,
@ -110,6 +122,14 @@ var defaultConfig = Config{
ManifestDir: "data",
PersistenceFile: "transfer_queue.json",
},
Logging: LoggingConfig{
Dir: "data/logs",
Level: "info",
},
Health: HealthConfig{
Enabled: true,
Addr: ":8090",
},
}
func Load() (*Config, error) {
@ -167,6 +187,22 @@ func (c *Config) loadFromEnvironment() error {
c.Processing.FFmpegPath = val
}
if val := os.Getenv("LOG_DIR"); val != "" {
c.Logging.Dir = val
}
if val := os.Getenv("LOG_LEVEL"); val != "" {
c.Logging.Level = val
}
if val := os.Getenv("HEALTH_ENABLED"); val != "" {
c.Health.Enabled = val == "true"
}
if val := os.Getenv("HEALTH_ADDR"); val != "" {
c.Health.Addr = val
}
return nil
}
@ -192,12 +228,16 @@ func (c *Config) resolveAndValidatePaths() error {
if !filepath.IsAbs(c.Paths.PersistenceFile) {
c.Paths.PersistenceFile = filepath.Join(c.Paths.BaseDir, c.Paths.PersistenceFile)
}
if !filepath.IsAbs(c.Logging.Dir) {
c.Logging.Dir = filepath.Join(cwd, c.Logging.Dir)
}
requiredDirs := []string{
c.Paths.BaseDir,
c.Paths.LocalOutput,
c.Paths.ProcessOutput,
c.Paths.ManifestDir,
c.Logging.Dir,
}
for _, dir := range requiredDirs {

323
pkg/health/dashboard.html Normal file
View File

@ -0,0 +1,323 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>m3u8-downloader status</title>
<style>
:root {
--bg: #0f1115;
--panel: #161922;
--border: #262b38;
--text: #e6e8ee;
--muted: #8b909e;
--green: #3ecf8e;
--red: #f2545b;
--amber: #f2b705;
--blue: #4f8ef7;
}
* { box-sizing: border-box; }
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: var(--bg);
color: var(--text);
}
header {
display: flex;
align-items: center;
justify-content: space-between;
flex-wrap: wrap;
gap: 12px;
padding: 16px 24px;
border-bottom: 1px solid var(--border);
}
header h1 {
font-size: 18px;
margin: 0;
font-weight: 600;
}
.header-meta {
display: flex;
align-items: center;
gap: 16px;
color: var(--muted);
font-size: 13px;
}
.badge {
display: inline-flex;
align-items: center;
gap: 6px;
padding: 4px 12px;
border-radius: 999px;
font-size: 13px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.03em;
}
.badge-dot { width: 8px; height: 8px; border-radius: 50%; }
.badge.healthy { background: rgba(62,207,142,0.15); color: var(--green); }
.badge.healthy .badge-dot { background: var(--green); }
.badge.degraded { background: rgba(242,84,91,0.15); color: var(--red); }
.badge.degraded .badge-dot { background: var(--red); }
.badge.unknown { background: rgba(139,144,158,0.15); color: var(--muted); }
.badge.unknown .badge-dot { background: var(--muted); }
#conn-banner {
display: none;
background: var(--red);
color: #fff;
text-align: center;
padding: 8px;
font-size: 13px;
font-weight: 600;
}
#conn-banner.show { display: block; }
main {
padding: 24px;
display: grid;
grid-template-columns: repeat(auto-fit, minmax(340px, 1fr));
gap: 20px;
max-width: 1400px;
margin: 0 auto;
}
.card {
background: var(--panel);
border: 1px solid var(--border);
border-radius: 10px;
padding: 18px 20px;
}
.card h2 {
font-size: 14px;
text-transform: uppercase;
letter-spacing: 0.05em;
color: var(--muted);
margin: 0 0 14px 0;
display: flex;
align-items: center;
justify-content: space-between;
}
table { width: 100%; border-collapse: collapse; font-size: 13px; }
th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); }
th { color: var(--muted); font-weight: 500; font-size: 12px; }
tr.failed-row { background: rgba(242,84,91,0.08); }
tr.inactive-row { opacity: 0.5; }
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 6px; }
.dot.on { background: var(--green); }
.dot.off { background: var(--red); }
.stat-grid {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 10px;
margin-bottom: 14px;
}
.stat {
background: rgba(255,255,255,0.03);
border-radius: 8px;
padding: 10px 12px;
}
.stat .value { font-size: 20px; font-weight: 600; }
.stat .label { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: 0.03em; }
.stat.failed .value { color: var(--red); }
.state-pill {
display: inline-block;
padding: 3px 10px;
border-radius: 999px;
font-size: 12px;
font-weight: 600;
text-transform: uppercase;
}
.state-idle { background: rgba(139,144,158,0.15); color: var(--muted); }
.state-scanning, .state-running { background: rgba(79,142,247,0.15); color: var(--blue); }
.state-completed { background: rgba(62,207,142,0.15); color: var(--green); }
.state-failed { background: rgba(242,84,91,0.15); color: var(--red); }
.kv { display: flex; justify-content: space-between; padding: 4px 0; font-size: 13px; border-bottom: 1px solid var(--border); }
.kv:last-child { border-bottom: none; }
.kv .k { color: var(--muted); }
.error-list { margin: 10px 0 0 0; padding: 0; list-style: none; }
.error-list li {
font-size: 12px;
padding: 6px 8px;
margin-bottom: 4px;
background: rgba(242,84,91,0.08);
border-left: 3px solid var(--red);
border-radius: 4px;
color: var(--text);
}
.error-list .err-time { color: var(--muted); margin-right: 6px; }
.empty-note { color: var(--muted); font-size: 13px; font-style: italic; }
</style>
</head>
<body>
<div id="conn-banner">Lost connection to status endpoint - retrying...</div>
<header>
<h1>m3u8-downloader</h1>
<div class="header-meta">
<span id="overall-badge" class="badge unknown"><span class="badge-dot"></span>unknown</span>
<span id="uptime">uptime: -</span>
<span id="last-updated">last updated: -</span>
</div>
</header>
<main>
<section class="card">
<h2>Download</h2>
<table>
<thead>
<tr><th>Resolution</th><th>Active</th><th>OK</th><th>Failed</th><th>Last activity</th></tr>
</thead>
<tbody id="variant-rows"></tbody>
</table>
<div id="download-empty" class="empty-note" style="display:none;">No variants reporting yet.</div>
<ul id="download-errors" class="error-list"></ul>
</section>
<section class="card">
<h2>Transfer</h2>
<div class="kv"><span class="k">NAS connection</span><span><span id="nas-dot" class="dot off"></span><span id="nas-text">unknown</span></span></div>
<div class="stat-grid" style="margin-top:12px;">
<div class="stat"><div class="value" id="t-added">-</div><div class="label">Added</div></div>
<div class="stat"><div class="value" id="t-completed">-</div><div class="label">Completed</div></div>
<div class="stat failed"><div class="value" id="t-failed">-</div><div class="label">Failed</div></div>
<div class="stat"><div class="value" id="t-pending">-</div><div class="label">Pending</div></div>
<div class="stat"><div class="value" id="t-queue">-</div><div class="label">Queue size</div></div>
<div class="stat"><div class="value" id="t-cleanup">-</div><div class="label">Cleanup pending</div></div>
</div>
<div class="kv"><span class="k">Transferred</span><span id="t-bytes">-</span></div>
<ul id="transfer-errors" class="error-list"></ul>
</section>
<section class="card">
<h2>Processing</h2>
<div class="kv"><span class="k">State</span><span id="p-state" class="state-pill state-idle">idle</span></div>
<div class="kv"><span class="k">Event</span><span id="p-event">-</span></div>
<div class="kv"><span class="k">Segments found</span><span id="p-segments">-</span></div>
<div class="kv"><span class="k">Started</span><span id="p-started">-</span></div>
<div class="kv"><span class="k">Finished</span><span id="p-finished">-</span></div>
<ul id="processing-errors" class="error-list"></ul>
</section>
</main>
<script>
const POLL_INTERVAL_MS = 3000;
const connBanner = document.getElementById('conn-banner');
function fmtBytes(n) {
if (!n || n <= 0) return '0 B';
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
let i = 0;
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
return n.toFixed(i === 0 ? 0 : 1) + ' ' + units[i];
}
function fmtDuration(sec) {
sec = Math.floor(sec || 0);
const h = Math.floor(sec / 3600);
const m = Math.floor((sec % 3600) / 60);
const s = sec % 60;
const parts = [];
if (h) parts.push(h + 'h');
if (m || h) parts.push(m + 'm');
parts.push(s + 's');
return parts.join(' ');
}
function fmtTime(iso) {
if (!iso) return '-';
const d = new Date(iso);
if (isNaN(d.getTime()) || d.getFullYear() < 1971) return '-';
return d.toLocaleTimeString();
}
function renderErrors(listEl, errors) {
listEl.innerHTML = '';
(errors || []).slice().reverse().forEach(e => {
const li = document.createElement('li');
const t = document.createElement('span');
t.className = 'err-time';
t.textContent = fmtTime(e.time);
li.appendChild(t);
li.appendChild(document.createTextNode(e.message));
listEl.appendChild(li);
});
}
function render(data) {
const badge = document.getElementById('overall-badge');
badge.className = 'badge ' + (data.status === 'healthy' ? 'healthy' : 'degraded');
badge.innerHTML = '<span class="badge-dot"></span>' + data.status;
document.getElementById('uptime').textContent = 'uptime: ' + fmtDuration(data.uptime_seconds);
document.getElementById('last-updated').textContent = 'last updated: ' + new Date().toLocaleTimeString();
// Download
const variants = (data.download && data.download.variants) || [];
const rows = document.getElementById('variant-rows');
rows.innerHTML = '';
document.getElementById('download-empty').style.display = variants.length ? 'none' : 'block';
variants.forEach(v => {
const tr = document.createElement('tr');
if (v.segments_failed > 0) tr.classList.add('failed-row');
if (!v.active) tr.classList.add('inactive-row');
tr.innerHTML =
'<td>' + v.resolution + '</td>' +
'<td><span class="dot ' + (v.active ? 'on' : 'off') + '"></span>' + (v.active ? 'active' : 'stopped') + '</td>' +
'<td>' + v.segments_ok + '</td>' +
'<td>' + v.segments_failed + '</td>' +
'<td>' + fmtTime(v.last_activity) + '</td>';
rows.appendChild(tr);
});
renderErrors(document.getElementById('download-errors'), data.download && data.download.recent_errors);
// Transfer
const t = data.transfer || {};
const nasDot = document.getElementById('nas-dot');
nasDot.className = 'dot ' + (t.nas_connected ? 'on' : 'off');
document.getElementById('nas-text').textContent = t.nas_connected ? 'connected' : 'disconnected';
document.getElementById('t-added').textContent = t.added ?? '-';
document.getElementById('t-completed').textContent = t.completed ?? '-';
document.getElementById('t-failed').textContent = t.failed ?? '-';
document.getElementById('t-pending').textContent = t.pending ?? '-';
document.getElementById('t-queue').textContent = t.queue_size ?? '-';
document.getElementById('t-cleanup').textContent = t.cleanup_pending ?? '-';
document.getElementById('t-bytes').textContent = fmtBytes(t.bytes_transferred);
renderErrors(document.getElementById('transfer-errors'), t.recent_errors);
// Processing
const p = data.processing || {};
const stateEl = document.getElementById('p-state');
const state = p.state || 'idle';
stateEl.textContent = state;
stateEl.className = 'state-pill state-' + state;
document.getElementById('p-event').textContent = p.event_name || '-';
document.getElementById('p-segments').textContent = p.segments_found ?? '-';
document.getElementById('p-started').textContent = fmtTime(p.started_at);
document.getElementById('p-finished').textContent = fmtTime(p.finished_at);
renderErrors(document.getElementById('processing-errors'), p.recent_errors);
}
async function poll() {
try {
const resp = await fetch('/status', { cache: 'no-store' });
if (!resp.ok) throw new Error('status ' + resp.status);
const data = await resp.json();
connBanner.classList.remove('show');
render(data);
} catch (e) {
connBanner.classList.add('show');
}
}
poll();
setInterval(poll, POLL_INTERVAL_MS);
</script>
</body>
</html>

352
pkg/health/registry.go Normal file
View File

@ -0,0 +1,352 @@
package health
import (
"sort"
"sync"
"time"
)
const maxRecentErrors = 10
type ErrorEntry struct {
Time time.Time `json:"time"`
Message string `json:"message"`
}
type VariantStatus struct {
Resolution string `json:"resolution"`
Bandwidth uint32 `json:"bandwidth"`
Active bool `json:"active"`
SegmentsOK int `json:"segments_ok"`
SegmentsFailed int `json:"segments_failed"`
LastActivity time.Time `json:"last_activity"`
LastError string `json:"last_error,omitempty"`
}
// StatsProvider is satisfied by *transfer.QueueStats without pkg/health
// needing to import pkg/transfer (which would create an import cycle, since
// pkg/transfer calls into pkg/health to report status).
type StatsProvider interface {
GetStats() (added, completed, failed, pending int, bytes int64)
}
type downloadState struct {
variants map[string]*VariantStatus
recentErrors []ErrorEntry
}
type transferState struct {
statsProvider StatsProvider
queueSizeFn func() int
cleanupPendingFn func() int
nasConnectedFn func() bool
nasConnected bool
recentErrors []ErrorEntry
}
type processingState struct {
state string // idle | scanning | running | completed | failed
eventName string
segmentsFound int
startedAt time.Time
finishedAt time.Time
lastError string
recentErrors []ErrorEntry
}
// Registry is the process-wide health/metrics store. All subsystems report
// into it via the Record* methods; the health HTTP server reads it via
// Snapshot(). Safe for concurrent use.
type Registry struct {
mu sync.RWMutex
startedAt time.Time
download downloadState
transfer transferState
processing processingState
}
var (
globalRegistry *Registry
registryOnce sync.Once
)
// MustGetRegistry returns the process-wide registry, mirroring the
// constants.MustGetConfig() / logging.MustGetFactory() singleton style.
func MustGetRegistry() *Registry {
registryOnce.Do(func() {
globalRegistry = New()
})
return globalRegistry
}
func New() *Registry {
return &Registry{
startedAt: time.Now(),
download: downloadState{
variants: make(map[string]*VariantStatus),
},
processing: processingState{
state: "idle",
},
}
}
func appendBounded(errs []ErrorEntry, e ErrorEntry) []ErrorEntry {
errs = append(errs, e)
if len(errs) > maxRecentErrors {
errs = errs[len(errs)-maxRecentErrors:]
}
return errs
}
func errString(err error) string {
if err == nil {
return ""
}
return err.Error()
}
// --- Download ---
func (r *Registry) RecordVariantStarted(resolution string, bandwidth uint32) {
r.mu.Lock()
defer r.mu.Unlock()
r.download.variants[resolution] = &VariantStatus{
Resolution: resolution,
Bandwidth: bandwidth,
Active: true,
LastActivity: time.Now(),
}
}
func (r *Registry) RecordVariantStopped(resolution string) {
r.mu.Lock()
defer r.mu.Unlock()
if v, ok := r.download.variants[resolution]; ok {
v.Active = false
}
}
func (r *Registry) RecordSegmentDownloaded(resolution string) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.SegmentsOK++
v.LastActivity = time.Now()
}
func (r *Registry) RecordSegmentFailed(resolution string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.SegmentsFailed++
v.LastActivity = time.Now()
v.LastError = errString(err)
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
Time: time.Now(),
Message: resolution + ": " + errString(err),
})
}
func (r *Registry) RecordPlaylistError(resolution string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.LastError = errString(err)
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
Time: time.Now(),
Message: resolution + " playlist: " + errString(err),
})
}
func (r *Registry) ensureVariantLocked(resolution string) *VariantStatus {
v, ok := r.download.variants[resolution]
if !ok {
v = &VariantStatus{Resolution: resolution, Active: true}
r.download.variants[resolution] = v
}
return v
}
// --- Transfer ---
// AttachTransfer wires the registry to the live counters/state already
// maintained by pkg/transfer instead of duplicating them. Called once from
// TransferService construction.
func (r *Registry) AttachTransfer(stats StatsProvider, queueSizeFn, cleanupPendingFn func() int, nasConnectedFn func() bool) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.statsProvider = stats
r.transfer.queueSizeFn = queueSizeFn
r.transfer.cleanupPendingFn = cleanupPendingFn
r.transfer.nasConnectedFn = nasConnectedFn
}
func (r *Registry) RecordNASConnectivity(connected bool) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.nasConnected = connected
}
// RecordTransferError captures failures QueueStats' cumulative counters
// don't individually retain (pre-queue failures, cleanup failures) so they
// still surface in the bounded recent-errors list.
func (r *Registry) RecordTransferError(sourcePath string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.recentErrors = appendBounded(r.transfer.recentErrors, ErrorEntry{
Time: time.Now(),
Message: sourcePath + ": " + errString(err),
})
}
// --- Processing ---
func (r *Registry) RecordProcessingStageChange(state, eventName string) {
r.mu.Lock()
defer r.mu.Unlock()
r.processing.state = state
if eventName != "" {
r.processing.eventName = eventName
}
switch state {
case "scanning":
r.processing.startedAt = time.Now()
r.processing.finishedAt = time.Time{}
r.processing.lastError = ""
case "completed", "failed":
r.processing.finishedAt = time.Now()
}
}
func (r *Registry) RecordProcessingSegmentsFound(count int) {
r.mu.Lock()
defer r.mu.Unlock()
r.processing.segmentsFound = count
}
func (r *Registry) RecordProcessingError(err error) {
r.mu.Lock()
defer r.mu.Unlock()
msg := errString(err)
r.processing.lastError = msg
r.processing.recentErrors = appendBounded(r.processing.recentErrors, ErrorEntry{
Time: time.Now(),
Message: msg,
})
}
// --- Snapshot ---
type DownloadSnapshot struct {
Variants []*VariantStatus `json:"variants"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type TransferSnapshot struct {
NASConnected bool `json:"nas_connected"`
QueueSize int `json:"queue_size"`
CleanupPending int `json:"cleanup_pending"`
Added int `json:"added"`
Completed int `json:"completed"`
Failed int `json:"failed"`
Pending int `json:"pending"`
BytesTransferred int64 `json:"bytes_transferred"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type ProcessingSnapshot struct {
State string `json:"state"`
EventName string `json:"event_name,omitempty"`
SegmentsFound int `json:"segments_found"`
StartedAt *time.Time `json:"started_at,omitempty"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
LastError string `json:"last_error,omitempty"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type Snapshot struct {
Status string `json:"status"`
UptimeSec float64 `json:"uptime_seconds"`
Download DownloadSnapshot `json:"download"`
Transfer TransferSnapshot `json:"transfer"`
Processing ProcessingSnapshot `json:"processing"`
}
func (r *Registry) Snapshot() Snapshot {
r.mu.RLock()
defer r.mu.RUnlock()
variants := make([]*VariantStatus, 0, len(r.download.variants))
for _, v := range r.download.variants {
vCopy := *v
variants = append(variants, &vCopy)
}
sort.Slice(variants, func(i, j int) bool { return variants[i].Resolution < variants[j].Resolution })
dl := DownloadSnapshot{
Variants: variants,
RecentErrors: append([]ErrorEntry(nil), r.download.recentErrors...),
}
tr := TransferSnapshot{
NASConnected: r.transfer.nasConnected,
RecentErrors: append([]ErrorEntry(nil), r.transfer.recentErrors...),
}
if r.transfer.statsProvider != nil {
added, completed, failed, pending, bytes := r.transfer.statsProvider.GetStats()
tr.Added, tr.Completed, tr.Failed, tr.Pending, tr.BytesTransferred = added, completed, failed, pending, bytes
}
if r.transfer.queueSizeFn != nil {
tr.QueueSize = r.transfer.queueSizeFn()
}
if r.transfer.cleanupPendingFn != nil {
tr.CleanupPending = r.transfer.cleanupPendingFn()
}
if r.transfer.nasConnectedFn != nil {
tr.NASConnected = r.transfer.nasConnectedFn()
}
proc := ProcessingSnapshot{
State: r.processing.state,
EventName: r.processing.eventName,
SegmentsFound: r.processing.segmentsFound,
LastError: r.processing.lastError,
RecentErrors: append([]ErrorEntry(nil), r.processing.recentErrors...),
}
if !r.processing.startedAt.IsZero() {
t := r.processing.startedAt
proc.StartedAt = &t
}
if !r.processing.finishedAt.IsZero() {
t := r.processing.finishedAt
proc.FinishedAt = &t
}
snap := Snapshot{
UptimeSec: time.Since(r.startedAt).Seconds(),
Download: dl,
Transfer: tr,
Processing: proc,
}
snap.Status = computeStatus(dl, tr, proc)
return snap
}
// computeStatus gives one glanceable field: "degraded" the moment anything
// worth investigating shows up, "healthy" otherwise. Transfer is only
// considered for degradation once it's actually in use (Attached + has seen
// activity), so a download-only run with NAS transfer disabled doesn't
// falsely report degraded.
func computeStatus(dl DownloadSnapshot, tr TransferSnapshot, proc ProcessingSnapshot) string {
if len(dl.RecentErrors) > 0 || len(tr.RecentErrors) > 0 || len(proc.RecentErrors) > 0 {
return "degraded"
}
if !tr.NASConnected && (tr.Added > 0 || tr.QueueSize > 0) {
return "degraded"
}
if proc.State == "failed" {
return "degraded"
}
return "healthy"
}

60
pkg/health/server.go Normal file
View File

@ -0,0 +1,60 @@
package health
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/logging"
"net"
"net/http"
)
//go:embed dashboard.html
var dashboardHTML []byte
// Start binds the health/status HTTP server and returns immediately once the
// listener is bound, so callers see bind failures (e.g. port already in use)
// synchronously rather than losing them in a background goroutine. The
// returned *http.Server should be shut down by the caller via Shutdown(ctx).
//
// Security note: cfg.Addr is expected to be a bind-all address (e.g. ":8090")
// so status can be checked from other devices on the local network during a
// live event. There is no authentication or TLS on these endpoints - this is
// only safe on a trusted local network, never exposed to the open internet.
func Start(ctx context.Context, cfg *config.HealthConfig, reg *Registry) (*http.Server, error) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Write(dashboardHTML)
})
mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(reg.Snapshot()); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
})
ln, err := net.Listen("tcp", cfg.Addr)
if err != nil {
return nil, fmt.Errorf("failed to bind health server to %s: %w", cfg.Addr, err)
}
srv := &http.Server{Handler: mux}
go func() {
if err := srv.Serve(ln); err != nil && err != http.ErrServerClosed {
logging.MustGetFactory().App.Error("health server stopped unexpectedly", logging.KeyError, err.Error())
}
}()
return srv, nil
}

72
pkg/logging/factory.go Normal file
View File

@ -0,0 +1,72 @@
package logging
import (
"log/slog"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"os"
"path/filepath"
"strings"
"sync"
)
// Factory holds one structured logger per pipeline stage, each writing JSON
// lines to its own file under LoggingConfig.Dir. These are additive to the
// existing stdlib log.* calls, not a replacement for them.
type Factory struct {
App *slog.Logger
Download *slog.Logger
Transfer *slog.Logger
Processing *slog.Logger
}
var (
globalFactory *Factory
factoryOnce sync.Once
)
// MustGetFactory returns the process-wide logging factory, building it from
// the config singleton on first use. Mirrors constants.MustGetConfig().
func MustGetFactory() *Factory {
factoryOnce.Do(func() {
globalFactory = New(&constants.MustGetConfig().Logging)
})
return globalFactory
}
func New(cfg *config.LoggingConfig) *Factory {
level := parseLevel(cfg.Level)
return &Factory{
App: newSubsystemLogger(cfg.Dir, "app.log", "app", level),
Download: newSubsystemLogger(cfg.Dir, "download.log", "download", level),
Transfer: newSubsystemLogger(cfg.Dir, "transfer.log", "transfer", level),
Processing: newSubsystemLogger(cfg.Dir, "processing.log", "processing", level),
}
}
// newSubsystemLogger opens (or creates) the subsystem's log file in append
// mode. A logging misconfiguration must never take down an active recording,
// so failure to open the file falls back to stderr instead of panicking.
func newSubsystemLogger(dir, filename, component string, level slog.Level) *slog.Logger {
path := filepath.Join(dir, filename)
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
fallback := slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: level})).With("component", component)
fallback.Error("failed to open log file, falling back to stderr", KeyFile, path, KeyError, err.Error())
return fallback
}
return slog.New(slog.NewJSONHandler(f, &slog.HandlerOptions{Level: level})).With("component", component)
}
func parseLevel(level string) slog.Level {
switch strings.ToLower(strings.TrimSpace(level)) {
case "debug":
return slog.LevelDebug
case "warn", "warning":
return slog.LevelWarn
case "error":
return slog.LevelError
default:
return slog.LevelInfo
}
}

13
pkg/logging/keys.go Normal file
View File

@ -0,0 +1,13 @@
package logging
const (
KeyResolution = "resolution"
KeyBandwidth = "bandwidth"
KeySegment = "segment"
KeySeq = "seq"
KeyFile = "file"
KeyBytes = "bytes"
KeyEvent = "event"
KeyError = "error"
KeyAttempt = "attempt"
)

View File

@ -4,6 +4,7 @@ import (
"encoding/json"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/utils"
"os"
"sort"
@ -61,17 +62,20 @@ func (m *ManifestWriter) WriteManifest() {
data, err := json.MarshalIndent(m.Segments, "", " ")
if err != nil {
log.Printf("Failed to marshal manifest: %v", err)
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
return
}
if err := utils.ValidateWritablePath(m.ManifestPath); err != nil {
log.Printf("Manifest path validation failed: %v", err)
logging.MustGetFactory().Download.Error("manifest path validation failed", logging.KeyError, err.Error())
return
}
file, err := os.Create(m.ManifestPath)
if err != nil {
log.Printf("Failed to create manifest file: %v", err)
logging.MustGetFactory().Download.Error("failed to create manifest file", logging.KeyError, err.Error())
return
}
@ -80,6 +84,7 @@ func (m *ManifestWriter) WriteManifest() {
_, err = file.Write(data)
if err != nil {
log.Printf("Failed to write manifest file: %v", err)
logging.MustGetFactory().Download.Error("failed to write manifest file", logging.KeyError, err.Error())
return
}
}

View File

@ -6,6 +6,7 @@ import (
"io"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"net/http"
"net/url"
"os"
@ -33,6 +34,7 @@ func (j SegmentJob) Key() string {
func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error {
for attempt := 0; attempt < 2; attempt++ {
if attempt > 0 {
logging.MustGetFactory().Download.Debug("retrying segment download", logging.KeyAttempt, attempt, "url", segmentURL)
time.Sleep(300 * time.Millisecond)
}
req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil)

View File

@ -7,7 +7,9 @@ import (
"github.com/grafov/m3u8"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"net/http"
"net/url"
"path"
@ -53,12 +55,14 @@ func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter)
req.Header.Set("Referer", constants.REFERRER)
resp, err := client.Do(req)
if err != nil {
logging.MustGetFactory().Download.Error("failed to fetch master playlist", logging.KeyError, err.Error())
return nil, err
}
defer resp.Body.Close()
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
if err != nil {
logging.MustGetFactory().Download.Error("failed to decode master playlist", logging.KeyError, err.Error())
return nil, err
}
@ -96,11 +100,17 @@ func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter)
OutputDir: outputDir,
})
}
logging.MustGetFactory().Download.Info("discovered variants", "count", len(variants))
return variants, nil
}
func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}, manifest *ManifestWriter) {
log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth)
dlLog := logging.MustGetFactory().Download.With(logging.KeyResolution, variant.Resolution)
registry := health.MustGetRegistry()
dlLog.Info("variant downloader started", logging.KeyBandwidth, variant.Bandwidth)
registry.RecordVariantStarted(variant.Resolution, variant.Bandwidth)
ticker := time.NewTicker(constants.RefreshDelay)
defer ticker.Stop()
client := &http.Client{}
@ -109,6 +119,7 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
for {
select {
case <-ctx.Done():
registry.RecordVariantStopped(variant.Resolution)
return
default:
}
@ -117,6 +128,8 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
playlist, err := LoadMediaPlaylist(variant.URL)
if err != nil {
log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err)
dlLog.Warn("failed to load media playlist", logging.KeyError, err.Error())
registry.RecordPlaylistError(variant.Resolution, err)
goto waitTick
}
seq = playlist.SeqNo
@ -149,6 +162,8 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
if err == nil {
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
dlLog.Debug("segment downloaded", logging.KeySegment, name)
registry.RecordSegmentDownloaded(variant.Resolution)
return
}
@ -159,21 +174,27 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
if httpClient.IsHTTPStatus(err, 403) {
log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name)
dlLog.Warn("segment download failed (403)", logging.KeySegment, name)
} else {
log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err)
dlLog.Error("segment download failed", logging.KeySegment, name, logging.KeyError, err.Error())
}
registry.RecordSegmentFailed(variant.Resolution, err)
}(job)
seq++
}
if playlist.Closed {
log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution)
dlLog.Info("playlist closed")
registry.RecordVariantStopped(variant.Resolution)
return
}
waitTick:
select {
case <-ctx.Done():
registry.RecordVariantStopped(variant.Resolution)
return
case <-ticker.C:
}

View File

@ -6,6 +6,8 @@ import (
"fmt"
"log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas"
"m3u8-downloader/pkg/utils"
"os"
@ -42,6 +44,7 @@ func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingServ
nasService := nas.NewNASService(nasConfig)
if err := nasService.TestConnection(); err != nil {
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
}
@ -76,18 +79,32 @@ func (ps *ProcessingService) GetEventDirs() ([]string, error) {
}
func (ps *ProcessingService) Start(ctx context.Context) error {
procLog := logging.MustGetFactory().Processing
registry := health.MustGetRegistry()
if !ps.config.Processing.Enabled {
log.Println("Processing service disabled")
return nil
}
registry.RecordProcessingStageChange("scanning", ps.eventName)
// fail centralizes the log/registry bookkeeping for every error return
// site below instead of repeating it at each one.
fail := func(err error) error {
procLog.Error("processing failed", logging.KeyEvent, ps.eventName, logging.KeyError, err.Error())
registry.RecordProcessingStageChange("failed", ps.eventName)
registry.RecordProcessingError(err)
return err
}
if ps.eventName == "" {
events, err := ps.GetEventDirs()
if err != nil {
return fmt.Errorf("failed to get event directories: %w", err)
return fail(fmt.Errorf("failed to get event directories: %w", err))
}
if len(events) == 0 {
return fmt.Errorf("no events found")
return fail(fmt.Errorf("no events found"))
}
if len(events) > 1 {
fmt.Println("Multiple events found, please select one:")
@ -99,21 +116,23 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
input = strings.TrimSpace(input)
index, err := strconv.Atoi(input)
if err != nil {
return fmt.Errorf("failed to parse input: %w", err)
return fail(fmt.Errorf("failed to parse input: %w", err))
}
if index < 1 || index > len(events) {
return fmt.Errorf("invalid input")
return fail(fmt.Errorf("invalid input"))
}
ps.eventName = events[index-1]
} else {
ps.eventName = events[0]
}
registry.RecordProcessingStageChange("scanning", ps.eventName)
}
procLog.Info("processing started", logging.KeyEvent, ps.eventName)
//Get all present resolutions
dirs, err := ps.GetResolutions()
if err != nil {
return fmt.Errorf("Failed to get resolutions: %w", err)
return fail(fmt.Errorf("Failed to get resolutions: %w", err))
}
//Spawn a worker per resolution
@ -131,25 +150,29 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
segments, err := ps.AggregateSegmentInfo(ch)
if err != nil {
return fmt.Errorf("Failed to aggregate segment info: %w", err)
return fail(fmt.Errorf("Failed to aggregate segment info: %w", err))
}
registry.RecordProcessingSegmentsFound(len(segments))
aggFile, err := ps.WriteConcatFile(segments)
if err != nil {
return fmt.Errorf("Failed to write concat file: %w", err)
return fail(fmt.Errorf("Failed to write concat file: %w", err))
}
// Feed info to ffmpeg to stitch files together
outPath := ps.config.GetProcessOutputPath(ps.eventName)
if err := utils.EnsureDir(outPath); err != nil {
return fmt.Errorf("failed to create output directory: %w", err)
return fail(fmt.Errorf("failed to create output directory: %w", err))
}
registry.RecordProcessingStageChange("running", ps.eventName)
concatErr := ps.RunFFmpeg(aggFile, outPath)
if concatErr != nil {
return concatErr
return fail(concatErr)
}
procLog.Info("processing completed", logging.KeyEvent, ps.eventName, "segments", len(segments))
registry.RecordProcessingStageChange("completed", ps.eventName)
return nil
}
@ -217,6 +240,7 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in
for segment := range ch {
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
current, exists := segmentMap[segment.SeqNo]
if !exists || rank[segment.Resolution] > rank[current.Resolution] {
segmentMap[segment.SeqNo] = segment
@ -318,9 +342,11 @@ func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error {
fileOutPath := utils.SafeJoin(outputPath, ps.eventName+".mp4")
fmt.Println("Input path:", inputPath)
fmt.Println("Output path:", fileOutPath)
logging.MustGetFactory().Processing.Info("running ffmpeg", "input", inputPath, "output", fileOutPath)
path, err := ps.getFFmpegPath()
if err != nil {
logging.MustGetFactory().Processing.Error("failed to find ffmpeg", logging.KeyError, err.Error())
return fmt.Errorf("failed to find FFmpeg: %w", err)
}
@ -329,9 +355,11 @@ func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error {
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
logging.MustGetFactory().Processing.Error("ffmpeg run failed", logging.KeyError, err.Error())
return fmt.Errorf("failed to run ffmpeg: %w", err)
}
fmt.Println("FFmpeg completed successfully")
logging.MustGetFactory().Processing.Info("ffmpeg completed successfully", "output", fileOutPath)
return nil
}

View File

@ -4,6 +4,7 @@ import (
"context"
"fmt"
"log"
"m3u8-downloader/pkg/logging"
"os"
"sync"
"time"
@ -41,6 +42,7 @@ func (cs *CleanupService) Start(ctx context.Context) error {
}
log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize)
logging.MustGetFactory().Transfer.Info("cleanup service started", "retention", cs.config.RetentionPeriod.String(), "batch_size", cs.config.BatchSize)
ticker := time.NewTicker(cs.config.CheckInterval)
defer ticker.Stop()
@ -97,6 +99,7 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
}
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors))
if len(errors) > 0 {
for i, err := range errors {
@ -105,6 +108,7 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
break
}
log.Printf("Error: %v", err)
logging.MustGetFactory().Transfer.Warn("cleanup error", logging.KeyError, err.Error())
}
}

View File

@ -4,6 +4,7 @@ import (
"context"
"fmt"
"log"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas"
"os"
"path/filepath"
@ -32,6 +33,7 @@ func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) e
}
log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath)
logging.MustGetFactory().Transfer.Debug("nas copy completed", logging.KeyFile, item.SourcePath, "dest", destPath)
return nil
}

View File

@ -6,6 +6,8 @@ import (
"encoding/json"
"fmt"
"log"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas"
"os"
"sync"
@ -74,7 +76,10 @@ func (tq *TransferQueue) Add(item TransferItem) error {
defer tq.mu.Unlock()
if tq.items.Len() >= tq.config.MaxQueueSize {
return fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
err := fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
logging.MustGetFactory().Transfer.Error("queue full, dropping file", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
return err
}
heap.Push(tq.items, &item)
@ -187,9 +192,11 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
if tq.cleanup != nil {
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
log.Printf("Failed to add file to cleanup list: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to schedule cleanup", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
}
}
log.Printf("File transfer completed: %s", item.SourcePath)
logging.MustGetFactory().Transfer.Info("file transfer completed", logging.KeyFile, item.SourcePath, logging.KeyBytes, item.FileSize)
return
}
@ -197,11 +204,14 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
item.RetryCount++
log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err)
logging.MustGetFactory().Transfer.Warn("file transfer attempt failed", logging.KeyFile, item.SourcePath, logging.KeyAttempt, item.RetryCount, logging.KeyError, err.Error())
if attempt == maxRetries {
item.Status = StatusFailed
tq.stats.IncrementFailed()
log.Printf("Transfer permanently failed for file: %s", item.SourcePath)
logging.MustGetFactory().Transfer.Error("transfer permanently failed", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
return
}
}
@ -225,10 +235,12 @@ func (tq *TransferQueue) SaveState() error {
"timestamp": time.Now(),
}, "", " ")
if err != nil {
logging.MustGetFactory().Transfer.Error("failed to marshal queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to marshal queue state: %w", err)
}
if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil {
logging.MustGetFactory().Transfer.Error("failed to save queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to save queue state: %w", err)
}
@ -241,6 +253,7 @@ func (tq *TransferQueue) LoadState() error {
if os.IsNotExist(err) {
return nil
}
logging.MustGetFactory().Transfer.Error("failed to load queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to load queue state: %w", err)
}

View File

@ -5,6 +5,8 @@ import (
"fmt"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
nas2 "m3u8-downloader/pkg/nas"
"m3u8-downloader/pkg/utils"
"os"
@ -36,8 +38,11 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
nas := nas2.NewNASService(nasConfig)
if err := nas.TestConnection(); err != nil {
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
}
logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir)
health.MustGetRegistry().RecordNASConnectivity(true)
cleanupConfig := CleanupConfig{
Enabled: cfg.Cleanup.AfterTransfer,
@ -66,6 +71,8 @@ func NewTrasferService(outputDir string, eventName string) (*TransferService, er
return nil, fmt.Errorf("failed to create file watcher: %w", err)
}
health.MustGetRegistry().AttachTransfer(queue.stats, queue.GetQueueSize, cleanup.GetPendingCount, nas.IsConnected)
return &TransferService{
watcher: watcher,
queue: queue,
@ -109,6 +116,7 @@ func (ts *TransferService) Start(ctx context.Context) error {
}()
log.Println("Transfer service started")
logging.MustGetFactory().Transfer.Info("transfer service started")
wg.Wait()
return nil
@ -128,6 +136,9 @@ func (ts *TransferService) reportStats(ctx context.Context) {
cleanupPending := ts.cleanup.GetPendingCount()
log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending)
logging.MustGetFactory().Transfer.Info("transfer stats",
"added", added, "completed", completed, "failed", failed, "pending", pending,
logging.KeyBytes, bytes, "queue_size", queueSize, "cleanup_pending", cleanupPending)
}
}
}
@ -146,9 +157,12 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
// Disconnect from NAS
if err := ts.nas.Disconnect(); err != nil {
log.Printf("Warning: failed to disconnect from NAS: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
}
health.MustGetRegistry().RecordNASConnectivity(false)
log.Println("Transfer service shut down")
logging.MustGetFactory().Transfer.Info("transfer service shut down")
return nil
}
@ -178,6 +192,8 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
relPath, err := filepath.Rel(localEventPath, path)
if err != nil {
log.Printf("Failed to get relative path for %s: %v", path, err)
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, path, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(path, err)
return nil
}
@ -218,6 +234,8 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
// Add to queue
if err := ts.queue.Add(item); err != nil {
log.Printf("Failed to queue file %s: %v", path, err)
logging.MustGetFactory().Transfer.Error("failed to queue file", logging.KeyFile, path, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(path, err)
} else {
log.Printf("Queued file: %s (%s, %d bytes)", path, resolution, info.Size())
fileCount++

View File

@ -4,6 +4,8 @@ import (
"context"
"fmt"
"log"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"math/rand"
"os"
"path/filepath"
@ -45,11 +47,13 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
}
log.Printf("Starting file watcher on %s", fw.outputDir)
logging.MustGetFactory().Transfer.Info("file watcher started", "dir", fw.outputDir)
for {
select {
case <-ctx.Done():
log.Println("File watcher shutting down...")
logging.MustGetFactory().Transfer.Info("file watcher shutting down")
return ctx.Err()
case event, ok := <-fw.watcher.Events:
@ -63,6 +67,7 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
return fmt.Errorf("Watcher errors channel closed")
}
log.Printf("Watcher error: %v", err)
logging.MustGetFactory().Transfer.Error("watcher error", logging.KeyError, err.Error())
}
}
}
@ -144,6 +149,8 @@ func (fw *FileWatcher) processFile(filePath string) {
info, err := os.Stat(filePath)
if err != nil {
log.Printf("Failed to stat file %s: %v", filePath, err)
logging.MustGetFactory().Transfer.Warn("failed to stat file", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
return
}
@ -152,6 +159,8 @@ func (fw *FileWatcher) processFile(filePath string) {
relPath, err := filepath.Rel(fw.outputDir, filePath)
if err != nil {
log.Printf("Failed to get relative path for file %s: %v", filePath, err)
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
return
}
@ -167,8 +176,11 @@ func (fw *FileWatcher) processFile(filePath string) {
if err := fw.queue.Add(item); err != nil {
log.Printf("Failed to add file to queue: %v", err)
logging.MustGetFactory().Transfer.Error("failed to add file to queue", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
} else {
log.Printf("Added file to queue: %s", filePath)
logging.MustGetFactory().Transfer.Debug("file queued", logging.KeyFile, filePath)
}
}