Compare commits

..

No commits in common. "main" and "manifest" have entirely different histories.

55 changed files with 310 additions and 7689 deletions

103
CLAUDE.md
View File

@ -11,11 +11,9 @@ This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlis
The project follows a modular architecture with clear separation of concerns: The project follows a modular architecture with clear separation of concerns:
- **cmd/**: Entry points for different execution modes - **cmd/**: Entry points for different execution modes
- **main/main.go**: Primary CLI entry point with URL input, event naming, and mode selection - **main/main.go**: Primary CLI entry point with URL input and event naming
- **downloader/download.go**: Core download orchestration logic with transfer service integration - **downloader/download.go**: Core download orchestration logic with transfer service integration
- **processor/process.go**: Alternative processing entry point - **processor/process.go**: Alternative processing entry point
- **transfer/transfer.go**: Transfer-only mode entry point
- **check/check.go**: Connection-check mode entry point (stream host + NAS reachability, no downloads)
- **pkg/**: Core packages containing the application logic - **pkg/**: Core packages containing the application logic
- **media/**: HLS streaming and download logic - **media/**: HLS streaming and download logic
- **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`) - **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`)
@ -29,24 +27,7 @@ The project follows a modular architecture with clear separation of concerns:
- **nas.go**: NAS file transfer with retry logic - **nas.go**: NAS file transfer with retry logic
- **cleanup.go**: Local file cleanup after successful transfer - **cleanup.go**: Local file cleanup after successful transfer
- **types.go**: Transfer system data structures - **types.go**: Transfer system data structures
- **processing/**: Video processing and concatenation system - **constants/constants.go**: Configuration constants (paths, timeouts, NAS settings)
- **service.go**: Processing service orchestration with FFmpeg integration
- **segment.go**: Individual segment processing logic
- **types.go**: Processing system data structures
- **nas/**: NAS connection and file operations (platform-agnostic: Windows and Linux)
- **config.go**: NAS configuration structure
- **nas.go**: `NASService` — resolves the destination to a host-native path, then copies/verifies files against it
- **location.go**: Parses UNC / `//server/share` / `smb://` / local paths into a host-neutral `Location`
- **connector.go**: `Connector` interface + runtime selection by GOOS (no build tags, so both platforms' code is compiled and tested everywhere)
- **connector_netuse.go**: Windows strategy — `net use`, UNC path used directly
- **connector_mount.go**: Linux strategy — resolve the share against `/proc/mounts`, optional root-only `mount.cifs`
- **connector_local.go**: Already-mounted/plain directory on either platform
- **mounts.go**: `/proc/mounts` parsing and share-to-mount-point matching
- **config/**: Centralized configuration management with validation
- **config.go**: Configuration loading, validation, and path resolution
- **utils/**: Utility functions for cross-platform compatibility
- **paths.go**: Path manipulation and validation utilities
- **constants/constants.go**: Configuration constants and singleton access
- **httpClient/error.go**: HTTP error handling utilities - **httpClient/error.go**: HTTP error handling utilities
## Core Functionality ## Core Functionality
@ -66,13 +47,6 @@ The project follows a modular architecture with clear separation of concerns:
4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally 4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally
5. **State Persistence**: Queue state is persisted to survive crashes and restarts 5. **State Persistence**: Queue state is persisted to survive crashes and restarts
### Video Processing Workflow (Optional)
1. **Segment Collection**: Processing service reads downloaded segments from NAS storage
2. **Quality Selection**: Automatically selects the highest quality variant available
3. **FFmpeg Processing**: Uses FFmpeg to concatenate segments into a single MP4 file
4. **Output Management**: Processed videos are saved to the configured output directory
5. **Concurrent Processing**: Multiple events can be processed simultaneously with worker pools
## Key Data Structures ## Key Data Structures
- `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer - `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
@ -81,60 +55,35 @@ The project follows a modular architecture with clear separation of concerns:
- `ManifestItem`: Individual segment record with sequence number and resolution - `ManifestItem`: Individual segment record with sequence number and resolution
- `TransferItem`: Transfer queue item with source, destination, retry count, and status - `TransferItem`: Transfer queue item with source, destination, retry count, and status
- `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup - `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup
- `ProcessingService`: Manages video processing operations with FFmpeg integration
- `ProcessConfig`: Configuration for processing operations including worker count and paths
- `NASService`: Handles NAS connection, authentication, and file operations
- `NASConfig`: Configuration structure for NAS connection parameters
## Configuration ## Configuration
Configuration is managed through a centralized system in `pkg/config/config.go` with environment variable support for deployment flexibility. The system provides validation, cross-platform path resolution, and sensible defaults: Key configuration is managed in `pkg/constants/constants.go`:
### Core Settings ### Core Settings
- `Core.WorkerCount`: Number of concurrent segment downloaders per variant (4) - ENV: `WORKER_COUNT` - `WorkerCount`: Number of concurrent segment downloaders per variant (4)
- `Core.RefreshDelay`: How often to check for playlist updates (3 seconds) - ENV: `REFRESH_DELAY_SECONDS` - `RefreshDelay`: How often to check for playlist updates (3 seconds)
- `LocalOutputDirPath`: Base directory for local downloads (`./data/`)
### Path Configuration - `ManifestPath`: Directory for manifest JSON files (`./data`)
- `Paths.LocalOutput`: Base directory for local downloads (`data/`) - ENV: `LOCAL_OUTPUT_DIR`
- `Paths.ProcessOutput`: Directory for processed videos (`out/`) - ENV: `PROCESS_OUTPUT_DIR`
- `Paths.ManifestDir`: Directory for manifest JSON files (`data/`)
- `Paths.PersistenceFile`: Transfer queue state file location
### HTTP Settings ### HTTP Settings
- `HTTPUserAgent`: User agent string for HTTP requests - `HTTPUserAgent`: User agent string for HTTP requests
- `REFERRER`: Referer header for HTTP requests (`https://www.flomarching.com`) - `REFERRER`: Referer header for HTTP requests
### NAS Transfer Settings ### NAS Transfer Settings
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER` - `EnableNASTransfer`: Enable/disable automatic NAS transfer (true)
- `NAS.OutputPath`: NAS destination (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`. Accepts UNC, `//server/share`, `smb://`, or a local/mounted directory; Windows connects it with `net use`, Linux resolves it against `/proc/mounts` - `NASOutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`)
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD` - `NASUsername`/`NASPassword`: NAS credentials (empty for current user)
- `NAS.AutoMount`: Linux-only, root-required auto-mount of the SMB share (false) - ENV: `NAS_AUTO_MOUNT` - `TransferWorkerCount`: Concurrent transfer workers (2)
- `NAS.MountPoint`: Override for the auto-mount location - ENV: `NAS_MOUNT_POINT` - `TransferRetryLimit`: Max retry attempts per file (3)
- `Transfer.WorkerCount`: Concurrent transfer workers (2) - `TransferTimeout`: Timeout per file transfer (30 seconds)
- `Transfer.RetryLimit`: Max retry attempts per file (3) - `FileSettlingDelay`: Wait before queuing new files (5 seconds)
- `Transfer.Timeout`: Timeout per file transfer (30 seconds) - `PersistencePath`: Transfer queue state file (`./data/transfer_queue.json`)
- `Transfer.FileSettlingDelay`: Wait before queuing new files (5 seconds)
- `Transfer.QueueSize`: Maximum queue size (100000)
- `Transfer.BatchSize`: Batch processing size (1000)
### Processing Settings
- `Processing.AutoProcess`: Enable automatic processing after download (true)
- `Processing.Enabled`: Enable processing functionality (true)
- `Processing.WorkerCount`: Concurrent processing workers (2)
- `Processing.FFmpegPath`: Path to FFmpeg executable (`ffmpeg`) - ENV: `FFMPEG_PATH`
### Cleanup Settings ### Cleanup Settings
- `Cleanup.AfterTransfer`: Delete local files after NAS transfer (true) - `CleanupAfterTransfer`: Delete local files after NAS transfer (true)
- `Cleanup.BatchSize`: Files processed per cleanup batch (1000) - `CleanupBatchSize`: Files processed per cleanup batch (10)
- `Cleanup.RetainHours`: Hours to keep local files (0 = immediate cleanup) - `RetainLocalHours`: Hours to keep local files (0 = immediate cleanup)
### Configuration Access
```go
cfg := constants.MustGetConfig() // Get validated config singleton
eventPath := cfg.GetEventPath("my-event") // Get cross-platform paths
```
See `DEPLOYMENT.md` for detailed environment variable configuration and deployment examples.
## Common Development Commands ## Common Development Commands
@ -163,9 +112,6 @@ go fmt ./...
- `-url`: M3U8 playlist URL (if not provided, prompts for input) - `-url`: M3U8 playlist URL (if not provided, prompts for input)
- `-event`: Event name for organizing downloads (defaults to current date) - `-event`: Event name for organizing downloads (defaults to current date)
- `-debug`: Debug mode (only downloads 1080p variant for easier testing) - `-debug`: Debug mode (only downloads 1080p variant for easier testing)
- `-transfer`: Transfer-only mode (transfer existing files without downloading)
- `-process`: Process-only mode (process existing files without downloading)
- `-check`: Connection-check mode (test stream URL + NAS connectivity and exit; exit code 1 on any failure)
## Monitoring and Downloads ## Monitoring and Downloads
@ -216,9 +162,7 @@ Downloaded files are organized as:
│ ├── 1080p/ # High quality segments │ ├── 1080p/ # High quality segments
│ ├── 720p/ # Medium quality segments │ ├── 720p/ # Medium quality segments
│ └── 480p/ # Lower quality segments │ └── 480p/ # Lower quality segments
├── transfer_queue.json # Transfer queue state └── transfer_queue.json # Transfer queue state
├── refresh_token.txt # Authentication tokens
└── tokens.txt # Session tokens
``` ```
NAS files mirror the local structure: NAS files mirror the local structure:
@ -228,11 +172,4 @@ NAS files mirror the local structure:
├── 1080p/ ├── 1080p/
├── 720p/ ├── 720p/
└── 480p/ └── 480p/
```
Processed files are output to:
```
./out/
└── {event-name}/
└── concatenated_segments.mp4 # Final processed video
``` ```

View File

@ -1,205 +0,0 @@
# Deployment Guide
This document outlines how to deploy and configure the StreamRecorder application in different environments.
## Environment Variables
The application supports configuration through environment variables for flexible deployment:
### Core Settings
- `WORKER_COUNT`: Number of concurrent segment downloaders per variant (default: 4)
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
### NAS Transfer Settings
- `NAS_OUTPUT_PATH`: NAS destination (default: "\\\\HomeLabNAS\\dci\\streams"). Accepts a UNC path, `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory
- `NAS_USERNAME`: NAS authentication username
- `NAS_PASSWORD`: NAS authentication password
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
- `NAS_AUTO_MOUNT`: Linux only — mount the SMB share if the host hasn't already (default: false, requires root)
- `NAS_MOUNT_POINT`: Where `NAS_AUTO_MOUNT` attaches the share (default: `/mnt/streamrecorder/<server>/<share>`)
The same `NAS_OUTPUT_PATH` works on Windows and Linux. Windows authenticates the
UNC path with `net use`; Linux resolves the share against `/proc/mounts` and
rewrites the path onto the mount point, so `\\\\HomeLabNAS\\dci\\streams` with
`//homelabnas/dci` mounted at `/mnt/dci` becomes `/mnt/dci/streams`. Mounting the
share on the host (fstab/autofs/systemd, or `docker run -v`) is the recommended
Linux setup; `NAS_AUTO_MOUNT` is the root-privileged fallback. Verify either with:
```bash
./stream-recorder -check
```
### Path Configuration
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")
- `PROCESS_OUTPUT_DIR`: Output directory for processed videos (default: "out")
### Processing Settings
- `FFMPEG_PATH`: Path to FFmpeg executable (default: "ffmpeg")
## Docker Deployment
### Dockerfile Example
```dockerfile
FROM golang:1.23-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o stream-recorder ./cmd/main
FROM alpine:latest
RUN apk --no-cache add ca-certificates ffmpeg
WORKDIR /root/
COPY --from=builder /app/stream-recorder .
CMD ["./stream-recorder"]
```
### Docker Compose Example
```yaml
version: '3.8'
services:
stream-recorder:
build: .
environment:
- NAS_OUTPUT_PATH=/mnt/nas/streams
- NAS_USERNAME=${NAS_USERNAME}
- NAS_PASSWORD=${NAS_PASSWORD}
- LOCAL_OUTPUT_DIR=/app/data
- PROCESS_OUTPUT_DIR=/app/out
- FFMPEG_PATH=ffmpeg
volumes:
- ./data:/app/data
- ./out:/app/out
- nas_mount:/mnt/nas
networks:
- stream_network
volumes:
nas_mount:
driver: local
driver_opts:
type: cifs
device: "//HomeLabNAS/dci"
o: username=${NAS_USERNAME},password=${NAS_PASSWORD},iocharset=utf8
networks:
stream_network:
driver: bridge
```
## Kubernetes Deployment
### ConfigMap Example
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: stream-recorder-config
data:
WORKER_COUNT: "4"
REFRESH_DELAY_SECONDS: "3"
ENABLE_NAS_TRANSFER: "true"
LOCAL_OUTPUT_DIR: "/app/data"
PROCESS_OUTPUT_DIR: "/app/out"
FFMPEG_PATH: "ffmpeg"
```
### Secret Example
```yaml
apiVersion: v1
kind: Secret
metadata:
name: stream-recorder-secrets
type: Opaque
data:
NAS_USERNAME: <base64-encoded-username>
NAS_PASSWORD: <base64-encoded-password>
```
### Deployment Example
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: stream-recorder
spec:
replicas: 1
selector:
matchLabels:
app: stream-recorder
template:
metadata:
labels:
app: stream-recorder
spec:
containers:
- name: stream-recorder
image: stream-recorder:latest
envFrom:
- configMapRef:
name: stream-recorder-config
- secretRef:
name: stream-recorder-secrets
volumeMounts:
- name: data-storage
mountPath: /app/data
- name: output-storage
mountPath: /app/out
volumes:
- name: data-storage
persistentVolumeClaim:
claimName: stream-data-pvc
- name: output-storage
persistentVolumeClaim:
claimName: stream-output-pvc
```
## Production Considerations
### Security
- Never commit credentials to version control
- Use environment variables or secret management systems for sensitive data
- Consider using service accounts or IAM roles for cloud deployments
- Rotate credentials regularly
### Monitoring
- Implement health checks for the application
- Monitor disk space for download directories
- Set up alerts for failed transfers or processing
- Log to centralized logging systems
### Scaling
- Use horizontal scaling for multiple concurrent streams
- Consider using message queues for segment processing
- Implement distributed storage for high availability
- Use load balancers for multiple instances
### Backup and Recovery
- Regular backups of configuration and state files
- Test recovery procedures
- Document rollback processes
- Maintain disaster recovery plans
## Configuration Validation
The application validates configuration at startup and will fail fast if:
- Required directories cannot be created
- NAS paths are invalid when transfer is enabled
- FFmpeg is not found when processing is enabled
- Critical environment variables are malformed
## Troubleshooting
### Common Issues
1. **Path Permission Errors**: Ensure the application has write access to configured directories
2. **NAS Connection Failures**: Verify network connectivity and credentials
3. **FFmpeg Not Found**: Install FFmpeg or set correct FFMPEG_PATH
4. **Environment Variable Format**: Check for typos and correct boolean values
### Debug Mode
Run with `-debug=true` to enable debug logging and download only 1080p variants for testing.

128
Makefile
View File

@ -1,128 +0,0 @@
# StreamRecorder Makefile
.PHONY: test test-verbose test-coverage build clean help
# Default target
all: build
# Build the application
build:
@echo "🔨 Building StreamRecorder..."
go build -o stream-recorder.exe ./cmd/main
# Build for different platforms
build-windows:
@echo "🔨 Building for Windows..."
GOOS=windows GOARCH=amd64 go build -o stream-recorder-windows.exe ./cmd/main
build-linux:
@echo "🔨 Building for Linux..."
GOOS=linux GOARCH=amd64 go build -o stream-recorder-linux ./cmd/main
build-all: build-windows build-linux
# Run unit tests
test:
@echo "🧪 Running unit tests..."
go run test_runner.go
# Run tests with verbose output
test-verbose:
@echo "🧪 Running unit tests (verbose)..."
go test -v ./pkg/...
# Run tests with coverage
test-coverage:
@echo "🧪 Running tests with coverage..."
go test -coverprofile=coverage.out ./pkg/...
go tool cover -html=coverage.out -o coverage.html
@echo "📊 Coverage report generated: coverage.html"
# Run tests for a specific package
test-pkg:
@if [ -z "$(PKG)" ]; then \
echo "❌ Please specify package: make test-pkg PKG=./pkg/config"; \
exit 1; \
fi
@echo "🧪 Testing package: $(PKG)"
go test -v $(PKG)
# Run benchmarks
benchmark:
@echo "🏃 Running benchmarks..."
go test -bench=. -benchmem ./pkg/...
# Clean build artifacts
clean:
@echo "🧹 Cleaning build artifacts..."
rm -f stream-recorder.exe stream-recorder-windows.exe stream-recorder-linux
rm -f coverage.out coverage.html
rm -rf data/ out/ *.json
# Format code
fmt:
@echo "🎨 Formatting code..."
go fmt ./...
# Lint code (requires golangci-lint)
lint:
@echo "🔍 Linting code..."
golangci-lint run
# Tidy dependencies
tidy:
@echo "📦 Tidying dependencies..."
go mod tidy
# Run security check (requires gosec)
security:
@echo "🔒 Running security check..."
gosec ./...
# Install development tools
install-tools:
@echo "🛠️ Installing development tools..."
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
go install github.com/securecodewarrior/gosec/v2/cmd/gosec@latest
# Quick development cycle: format, tidy, build, test
dev: fmt tidy build test
# CI pipeline: format check, lint, security, test, build
ci: fmt tidy lint security test build
# Help
help:
@echo "StreamRecorder Build Commands"
@echo "============================="
@echo ""
@echo "Build Commands:"
@echo " build - Build the main application"
@echo " build-windows - Build for Windows (x64)"
@echo " build-linux - Build for Linux (x64)"
@echo " build-all - Build for all platforms"
@echo ""
@echo "Test Commands:"
@echo " test - Run unit tests with custom runner"
@echo " test-verbose - Run tests with verbose output"
@echo " test-coverage - Run tests with coverage report"
@echo " test-pkg PKG=<pkg> - Test specific package"
@echo " benchmark - Run benchmarks"
@echo ""
@echo "Quality Commands:"
@echo " fmt - Format code"
@echo " lint - Lint code (requires golangci-lint)"
@echo " security - Security analysis (requires gosec)"
@echo " tidy - Tidy dependencies"
@echo ""
@echo "Development Commands:"
@echo " dev - Quick dev cycle (fmt, tidy, build, test)"
@echo " ci - Full CI pipeline"
@echo " clean - Clean build artifacts"
@echo " install-tools - Install development tools"
@echo ""
@echo "Examples:"
@echo " make test"
@echo " make test-pkg PKG=./pkg/config"
@echo " make build-all"
@echo " make dev"

217
README.md
View File

@ -1,217 +0,0 @@
# m3u8-downloader
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
## Architecture
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
│ └──────────────┘ │
└──────────────┬─────────────────────────────────┘
▼
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
```
### Download — `pkg/media`
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
### Transfer — `pkg/transfer` + `pkg/nas`
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
- `nas.go` / `pkg/nas`: `NASService` attaches the NAS destination for whatever host it's running on and does the actual file copy + size verification. See [NAS connectivity](#nas-connectivity-windows-and-linux).
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
### Processing — `pkg/processing`
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
### Configuration — `pkg/config` + `pkg/constants`
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
### Logging — `pkg/logging`
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
### Health / status — `pkg/health`
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
- `GET /healthz` — plain liveness check.
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
## Configuration
All settings have sane defaults; override via environment variables as needed.
| Env var | Default | Meaning |
|---|---|---|
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | NAS root segments are copied to. UNC (`\\server\share\dir`), `//server/share/dir`, `smb://server/share/dir`, or a local/mounted directory — see [NAS connectivity](#nas-connectivity-windows-and-linux) |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials. Used by `net use` on Windows and by `NAS_AUTO_MOUNT` on Linux; ignored for an already-mounted path |
| `NAS_AUTO_MOUNT` | `false` | Linux only: mount the SMB share if the host hasn't. Requires root |
| `NAS_MOUNT_POINT` | *(derived)* | Where `NAS_AUTO_MOUNT` attaches the share (default `/mnt/streamrecorder/<server>/<share>`) |
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
## NAS connectivity (Windows and Linux)
The same `NAS_OUTPUT_PATH` works on both platforms; only the mechanism for reaching it differs, and the recorder picks it at runtime.
| Host | Destination | What happens |
|---|---|---|
| Windows | UNC / `//server/share` / `smb://` | `net use \\server\share` with `NAS_USERNAME`/`NAS_PASSWORD`, then the UNC path is used directly |
| Linux | UNC / `//server/share` / `smb://` | The share is looked up in `/proc/mounts` and the UNC path is rewritten onto the mount point (`\\nas\dci\streams` + `//nas/dci` mounted at `/mnt/dci` → `/mnt/dci/streams`). Set `NAS_AUTO_MOUNT=true` to mount it instead |
| Either | A local or already-mounted directory | Used as-is, no network step |
Notes:
- **Server and share names match case-insensitively**, and both `cifs`/`smb3` and `nfs`/`nfs4` mounts are recognised. If the share is mounted at a deeper level than configured (`//nas/dci/streams` at `/mnt/streams`), the deepest matching mount wins.
- **Nothing the host already set up is torn down.** A share found already mounted (or already connected via `net use`) is left alone at shutdown; only a mount or connection this process created is undone.
- **`NAS_AUTO_MOUNT` requires root**, so the usual Linux setup is to mount the share yourself via fstab/autofs/systemd and either point `NAS_OUTPUT_PATH` at the mount point or leave it as the UNC path and let it resolve. Credentials are passed to `mount.cifs` through a `0600` credentials file, never on the command line where the process table would expose the password.
- **A Windows-shaped path on Linux is rejected, not silently created.** Backslashes and drive letters are legal filename characters on Linux, so `C:\data` or an unresolved `\\nas\dci` would otherwise be created as a local directory named exactly that — transfers would report success, cleanup would delete the local originals, and nothing would reach the NAS.
Run `-check` to see which strategy applies and where the destination resolved to.
## Building
```bash
go build -o m3u8-downloader ./cmd/main
# or, via the Makefile (produces stream-recorder(.exe)):
make build
```
## Usage
All modes are invoked through the single `cmd/main` binary:
```
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process] [-check]
```
| Flag | Applies to | Meaning |
|---|---|---|
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
| `-check` | connection-check mode | Tests connectivity to everything a recording depends on, then exits without downloading or transferring anything. Exit code is 0 only if no check failed. |
### Which command for which situation
**Verifying the setup before an event** (in the spirit of `dbt debug` — proves the stream host and NAS are reachable without recording anything):
```bash
m3u8-downloader -check -url "https://example.com/live/master.m3u8"
```
It runs three checks and prints pass/fail for each:
- **configuration** — config loads, and the local/process/log directories resolve and are creatable.
- **stream playlist** — only when `-url` is given (skipped otherwise). Fetches the master playlist with the same User-Agent and Referer the downloader uses, requires HTTP 200, decodes it, and lists the variants (and the resolution labels that would become output directories) it found.
- **nas storage** — only when `ENABLE_NAS_TRANSFER=true` (skipped otherwise). Runs the same connect → mkdir → write-probe sequence the transfer service does at startup, bounded by the NAS timeout so an unresponsive share reports a timeout instead of hanging.
Skipped checks don't fail the run; the exit code is 1 if any check actually failed, so this can gate a scripted start:
```bash
m3u8-downloader -check -url "$URL" && m3u8-downloader -url "$URL" -event "2026-regionals"
```
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
```
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
```
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
```bash
m3u8-downloader -event "2026-regionals" -transfer
```
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
```bash
m3u8-downloader -event "2026-regionals" -process
```
**Checking on a running instance** (from any device on the same network as the recorder):
```
http://<recorder-host>:8090/ # dashboard
http://<recorder-host>:8090/status # raw JSON
```
## Data layout
```
data/
├── {event}.json # download manifest (sequence numbers + resolutions)
├── {event}/
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
│ ├── 720p/
│ └── ...
├── logs/
│ ├── app.log # per-subsystem structured JSON logs
│ ├── download.log
│ ├── transfer.log
│ └── processing.log
└── transfer_queue.json # persisted transfer queue state
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
├── 1080p/
└── ...
out/
└── {event}/
└── {event}.mp4 # final FFmpeg output (process-only mode)
```
## More documentation
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.

View File

@ -1,253 +0,0 @@
# Testing Guide
This document describes the test suite for the StreamRecorder application.
## Overview
The test suite provides comprehensive coverage of core application components without requiring external dependencies like video files, NAS connectivity, or FFmpeg. All tests are self-contained and clean up after themselves.
## Test Structure
### Unit Tests by Package
#### `pkg/config`
- **File**: `config_test.go`
- **Coverage**: Configuration loading, environment variable override, path validation, validation errors
- **Key Tests**:
- Default config loading
- Environment variable overrides
- Path resolution and creation
- Validation error scenarios
#### `pkg/utils`
- **File**: `paths_test.go`
- **Coverage**: Cross-platform path utilities, directory operations, validation
- **Key Tests**:
- Safe path joining
- Directory creation
- Path existence checking
- Path validation
- Write permission testing
#### `pkg/constants`
- **File**: `constants_test.go`
- **Coverage**: Constants values, configuration singleton, integration
- **Key Tests**:
- Constant value verification
- Singleton pattern testing
- Config integration
- Concurrent access safety
#### `pkg/httpClient`
- **File**: `error_test.go`
- **Coverage**: HTTP error handling, status code management
- **Key Tests**:
- HTTP error creation and formatting
- Error comparison and detection
- Status code extraction
- Error wrapping support
#### `pkg/media`
- **File**: `manifest_test.go`
- **Coverage**: Manifest generation, segment tracking, JSON serialization
- **Key Tests**:
- Manifest writer initialization
- Segment addition and updates
- Quality resolution logic
- JSON file generation
- Sorting and deduplication
#### `pkg/processing`
- **File**: `service_test.go`
- **Coverage**: Processing service logic, path resolution, FFmpeg handling
- **Key Tests**:
- Service initialization
- Event directory scanning
- Resolution detection
- Segment aggregation
- File concatenation list generation
- FFmpeg path resolution
## Running Tests
### Quick Test Run
```bash
make test
```
### Verbose Output
```bash
make test-verbose
```
### Coverage Report
```bash
make test-coverage
```
Generates `coverage.html` with detailed coverage report.
### Test Specific Package
```bash
make test-pkg PKG=./pkg/config
```
### Manual Test Execution
```bash
# Run custom test runner
go run test_runner.go
# Run standard go test
go test ./pkg/...
# Run with coverage
go test -coverprofile=coverage.out ./pkg/...
go tool cover -html=coverage.out
```
## Test Features
### ✅ Self-Contained
- No external file dependencies
- No network connections required
- No NAS or FFmpeg installation needed
### ✅ Automatic Cleanup
- All temporary files/directories removed after tests
- Original environment variables restored
- No side effects on host system
### ✅ Isolated Environment
- Tests use temporary directories
- Environment variables safely overridden
- Configuration isolated from production settings
### ✅ Cross-Platform
- Path handling tested on Windows/Unix
- Platform-specific behavior validated
- Cross-platform compatibility verified
### ✅ Comprehensive Coverage
- Configuration management
- Path utilities and validation
- Error handling patterns
- Data structures and serialization
- Business logic without external dependencies
## Test Environment
The test suite automatically:
1. **Creates Temporary Workspace**: Each test run uses a fresh temporary directory
2. **Sets Test Environment**: Overrides environment variables to use test settings
3. **Disables External Dependencies**: Sets flags to disable NAS transfer and processing
4. **Cleans Up Completely**: Removes all test artifacts and restores environment
### Environment Variables Set During Tests
- `LOCAL_OUTPUT_DIR`: Points to temp directory
- `PROCESS_OUTPUT_DIR`: Points to temp directory
- `ENABLE_NAS_TRANSFER`: Set to `false`
- `PROCESSING_ENABLED`: Set to `false`
## Extending Tests
### Adding New Test Cases
1. **Create test file**: `pkg/yourpackage/yourfile_test.go`
2. **Follow naming convention**: `TestFunctionName`
3. **Use temp directories**: Always clean up created files
4. **Mock external dependencies**: Avoid real file operations where possible
### Test Template
```go
package yourpackage
import (
"os"
"testing"
)
func TestYourFunction(t *testing.T) {
// Setup
tempDir, err := os.MkdirTemp("", "test_*")
if err != nil {
t.Fatalf("Setup failed: %v", err)
}
defer os.RemoveAll(tempDir)
// Test logic
result := YourFunction()
// Assertions
if result != expected {
t.Errorf("Expected %v, got %v", expected, result)
}
}
```
### Best Practices
- **Always clean up**: Use `defer os.RemoveAll()` for temp directories
- **Test error cases**: Don't just test happy paths
- **Use table-driven tests**: For multiple similar test cases
- **Mock external dependencies**: Use echo/dummy commands instead of real tools
- **Validate cleanup**: Ensure tests don't leave artifacts
## CI/CD Integration
The test suite is designed for automated environments:
```bash
# Complete CI pipeline
make ci
# Just run tests in CI
make test
```
The custom test runner provides:
- ✅ Colored output for easy reading
- ✅ Test count and timing statistics
- ✅ Failure details and summaries
- ✅ Automatic environment management
- ✅ Exit codes for CI integration
## Troubleshooting
### Common Issues
**Tests fail with permission errors**
- Ensure write permissions in temp directory
- Check antivirus software isn't blocking file operations
**Config tests fail**
- Verify no conflicting environment variables are set
- Check that temp directories can be created
**Path tests fail on Windows**
- Confirm path separator handling is correct
- Verify Windows path validation logic
### Debug Mode
```bash
# Run with verbose output to see detailed failures
go test -v ./pkg/...
# Run specific failing test
go test -v -run TestSpecificFunction ./pkg/config
```
## Coverage Goals
Current test coverage targets:
- **Configuration**: 95%+ (critical for startup validation)
- **Path utilities**: 90%+ (cross-platform compatibility critical)
- **Constants**: 85%+ (verify all values and singleton behavior)
- **HTTP client**: 90%+ (error handling is critical)
- **Media handling**: 85%+ (core business logic)
- **Processing**: 70%+ (limited by external FFmpeg dependency)
Generate coverage report to verify:
```bash
make test-coverage
open coverage.html
```

111
TODO.md
View File

@ -1,111 +0,0 @@
# Return-from-a-year-away Audit — 2026-08-05
Scope reviewed: `main` (checked out), `manifest`, `cleanup` (all local + `origin` equivalents).
Remote: `https://git.aaronic.cc/kacarmichael/StreamRecorder.git`
## Branch situation (read this first)
The three branches are **not parallel efforts** — they're one linear line of development that `main` never caught up to:
```
main (0f4a39f, 2025-07-26) ──┐ stale, single-file-era downloader only
│
manifest (718d69c, 2025-08-10) ── +restructured cmd/pkg layout, config-less
│
cleanup (7d67cec, 2025-08-11) ── +pkg/config, tests, CI, Makefile, deployment docs
```
`manifest` is literally an ancestor of `cleanup` (same commit `718d69c`). `cleanup` is the most complete state that ever existed and was the last thing touched (2025-08-11) before the year gap. **None of this reached `main`.** If you pick this project back up, decide this before anything else:
- [ ] **Merge `cleanup` → `main`** (recommended — it's a strict superset: config validation, `pkg/nas`, `pkg/processing`, unit tests, CI workflow, `DEPLOYMENT.md`/`TESTING.md`). Then delete `manifest` (fully subsumed) and re-evaluate `cleanup` for deletion once merged.
- [ ] Or, if `cleanup`'s direction (FFmpeg concatenation, NAS-mirroring processing pipeline) is no longer where you want to go, explicitly archive it and restart `main` from a clean design — but do this as a decision, not by default.
---
## 🔴 Critical — fix/rotate before running this again
1. **Leaked NAS credentials, committed in plaintext.** `pkg/config/config.go` (branch `cleanup`) hardcodes a real default:
```go
Username: "NASAdmin",
Password: "s3tkY6tzA&KN6M",
```
This is in git history on a remote (`git.aaronic.cc`) regardless of whether you later change the default. **Rotate this NAS account's password now** — changing the code doesn't un-leak a password that's sat in a repo for a year. Then remove the credential entirely (fail closed: require `NAS_USERNAME`/`NAS_PASSWORD` env vars with no default, error out if unset when transfer is enabled) rather than shipping a working default.
2. **Guaranteed crash: double mutex unlock in `pkg/transfer/cleanup.go`.** In `ExecuteCleanup` (present on **`main` too**, not just `cleanup`):
```go
cs.mu.Lock()
...
cs.mu.Unlock() // line 72 — releases the lock
...
batch := make([]string, batchSize)
copy(batch, cs.pendingFiles[:batchSize])
cs.pendingFiles = cs.pendingFiles[batchSize:] // mutated WITHOUT the lock held
cs.mu.Unlock() // line 79 — unlocking an already-unlocked mutex
```
An unlock of an unlocked `sync.Mutex` is a **fatal, unrecoverable runtime error** (`fatal error: sync: unlock of unlocked mutex`) — it kills the whole process, not just a goroutine. Since `Cleanup.AfterTransfer` defaults to `true` and `RetainHours: 0` (immediate eligibility), this fires the first time any file finishes transferring and cleanup runs. This is very likely why any end-to-end run with NAS transfer enabled never survived long. Fix: hold the lock for the whole slice mutation, or copy-then-release-then-mutate-under-a-second-short-lock, but don't unlock twice.
3. **Nil-pointer crash in `pkg/media/stream.go` (`VariantDownloader`, `main` branch):**
```go
playlist, err := LoadMediaPlaylist(variant.URL)
seq := playlist.SeqNo // <-- dereferenced before the err check
if err != nil {
...
goto waitTick
}
```
`LoadMediaPlaylist` returns `nil, err` on any HTTP/parse failure, so `playlist` is a typed-nil pointer here — `playlist.SeqNo` panics. A panic in a goroutine takes down the entire process (Go doesn't isolate goroutine panics). Practically: any single transient network hiccup on any variant's playlist fetch kills the whole download session, mid-recording. Move the `err != nil` check above the `SeqNo` read.
---
## 🟠 High-priority bugs / correctness
4. **`PriorityQueue.Less` in `pkg/transfer/queue.go` sorts newest-first** (`pq[i].Timestamp.After(pq[j].Timestamp)`), making the transfer queue LIFO. For a live-recording pipeline that keeps producing new segments, this means the *oldest* queued segments can be starved indefinitely as long as new ones keep arriving — the opposite of what you'd want (drain oldest first so nothing waits forever). Worth confirming this was intentional; if not, flip the comparison.
5. **Retry backoff runs before the *first* attempt, not just retries** — `pkg/transfer/queue.go`, `processItem`:
```go
for attempt := 1; attempt <= maxRetries; attempt++ {
if attempt > 0 { // always true, since attempt starts at 1
... backoff ...
```
Every single transfer sleeps `1s` before even trying once. Should be `attempt > 1`.
6. **`pkg/constants/constants.go` (main) hardcodes a live FloSports auth token** directly in source (`MasterURL` with `token=...`, `hmac2=...`). It's expired now (epoch ~2025-07), so no live exposure, but it's the same pattern as #1 — secrets/tokens don't belong in source. Worth a rule going forward: URLs/tokens come from CLI args or env, never constants.
7. **`http.Client{}` created with no timeout** in `pkg/media/stream.go`, `playlist.go`, and `segment.go`. A stalled TCP connection (dead CDN edge, network partition) can hang a fetch forever — nothing in the surrounding code will time it out except the 10s context wrapped specifically around segment downloads in `VariantDownloader`. Playlist-fetch calls (`GetAllVariants`, `LoadMediaPlaylist`) have no such wrapper. Give the shared client a sane `Timeout` (e.g. 15–30s) as a backstop.
---
## 🟡 Cleanup / dead weight
8. **`cmd/proc/main.go` is a one-line stub** (`package proc`) on `main` — dead entry point, no `func main()`. Either finish it or delete it; right now `go build ./...` would fail on this if it were ever wired as a real `main` package (it currently silently isn't, which is its own trap for future-you).
9. **Built binaries committed to git** (`bin/flo_download`, `bin/flo_download.exe`, ~18MB, on `cleanup`/`manifest`). The `.gitea/workflows/build.yml` already cross-compiles and uploads artifacts — there's no reason binaries should also live in the repo/history. Remove and add `bin/` to `.gitignore`.
10. **`scratch.txt` is tracked despite being in `.gitignore`.** Adding a path to `.gitignore` after it's already tracked does nothing — it's still in the repo (and it's just a raw debug log dump). `git rm --cached scratch.txt`.
11. `.idea/*` (JetBrains project files) are committed. Low stakes solo-repo, but standard practice is to gitignore the whole `.idea/` directory rather than cherry-pick.
---
## 🔵 Program / architecture — bigger picture
12. **The project's own `CLAUDE.md` states the goal is "a listening REST API [that] take[s] in m3u8 urls, parse[s] them, and eventually send[s] to a conversion service."** None of the three branches implement any HTTP server — all three are CLI-only, one-shot-per-invocation tools. This is the largest gap between stated intent and actual code. Worth deciding: is the REST-API framing still the goal, or has the project's actual direction (CLI recorder + NAS mirror + FFmpeg concat, per `cleanup`'s `CLAUDE.md`) superseded it? Right now the docs and the code disagree with each other.
13. **`pkg/nas/nas.go`'s "connection" is a Windows-only shell-out to `net use`** (`EstablishConnection`), while CI (`build.yml`) cross-compiles for `linux`/`darwin`/`windows`. On non-Windows the UNC-path logic (`\\server\share`) and `net use` simply don't apply — there's no SMB client abstraction, so the Linux/macOS builds can produce a binary that can't actually do NAS transfer. If cross-platform deployment is real (not just "it happens to compile"), this needs a real cross-platform SMB/CIFS story (e.g., assume the NAS is pre-mounted by the OS and just treat it as a local path, which is actually simpler and already half-true for the Linux CIFS example in `DEPLOYMENT.md`).
14. **Zero tests on `main`.** `cleanup` has real coverage (`pkg/config`, `pkg/utils`, `pkg/constants`, `pkg/httpClient`, `pkg/media/manifest`, `pkg/processing`) plus a `Makefile` (`test`, `test-coverage`, `lint`, `security` via `gosec`) and a `TESTING.md`. All of this is stuck on a branch that was never merged — it's not a "write tests" TODO so much as a "land the tests you already wrote" TODO.
15. **CI never actually runs tests, lint, or `gosec`** despite the `Makefile` defining a full `ci` target (`fmt tidy lint security test build`). `.gitea/workflows/build.yml` only cross-compiles. Wire `make ci` (or its steps) into the pipeline so a broken build/test doesn't silently ship a binary artifact.
16. **Unbounded `seen` map** in `VariantDownloader` (`pkg/media/stream.go`) grows for the lifetime of a variant's polling loop with no eviction — fine for a few-hour event, worth a note if this is ever meant to run continuously/24-7.
---
## 🟢 Deployment
17. `DEPLOYMENT.md` (on `cleanup`) documents Docker/Compose/Kubernetes deployment in detail, but there's no actual `Dockerfile` committed anywhere in the repo — it's a doc-only example. If containerized deployment is the real target, commit the Dockerfile and at least smoke-test the build.
18. No secrets-management story beyond "use env vars" — fine for a home lab, but the doc's own "Security" section says "use secret management systems," which isn't backed by anything concrete (no `.env.example`, no sealed-secrets/vault reference). Given finding #1, this is worth actually doing, not just documenting.
19. Health checks, metrics/log shipping are listed as "considerations" in `DEPLOYMENT.md` but nothing is implemented — the app is a fire-and-forget CLI process with `log.Printf` to stdout only. If this runs unattended (e.g., in a container watching for the next event), you have no way to know it's alive short of watching logs.
---
## Suggested order of attack
1. Rotate the NAS credential (#1) — this is time-sensitive regardless of anything else.
2. Fix the two crash bugs (#2, #3) — nothing else is worth doing until a full recording session can survive.
3. Decide the branch-merge question and land `cleanup` → `main` (or consciously don't).
4. Wire CI to actually test/lint (#15) so regressions get caught going forward.
5. Then tackle the architecture question (#12: REST API vs. CLI-recorder-as-is) before investing more in either direction.

BIN
bin/flo_download Executable file → Normal file

Binary file not shown.

BIN
bin/flo_download.exe Normal file

Binary file not shown.

View File

@ -1,284 +0,0 @@
// Package check implements connection-check mode: it exercises the external
// dependencies a recording needs — the stream host and the NAS share — and
// reports pass/fail for each without downloading or transferring anything.
// Same spirit as `dbt debug`: prove the wiring before starting real work.
package check
import (
"context"
"fmt"
"log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/media"
"m3u8-downloader/pkg/nas"
"net/http"
"net/url"
"strings"
"time"
"github.com/grafov/m3u8"
)
// playlistTimeout bounds the master playlist fetch. Deliberately short: a
// stream host that takes longer than this is not usable for live recording.
const playlistTimeout = 15 * time.Second
// Run executes every applicable check and returns a process exit code, 0 only
// when nothing failed, so the command can gate a scripted recording start.
func Run(masterURL string) int {
fmt.Println("Running connection checks...")
fmt.Println()
cfg, err := constants.GetConfig()
if err != nil {
reportFail("configuration", nil, err)
fmt.Println()
fmt.Println("1 check failed.")
return 1
}
appLog := logging.MustGetFactory().App
appLog.Info("connection check starting", "url_provided", masterURL != "")
reportPass("configuration", []string{
"local output: " + cfg.Paths.LocalOutput,
"process output: " + cfg.Paths.ProcessOutput,
"log dir: " + cfg.Logging.Dir,
fmt.Sprintf("nas transfer: %s", enabledLabel(cfg.NAS.EnableTransfer)),
})
failures := 0
if masterURL == "" {
reportSkip("stream playlist", "no -url provided")
} else if details, err := checkMasterPlaylist(masterURL); err != nil {
reportFail("stream playlist", details, err)
appLog.Error("stream playlist check failed", logging.KeyError, err.Error())
failures++
} else {
reportPass("stream playlist", details)
appLog.Info("stream playlist check passed")
}
if !cfg.NAS.EnableTransfer {
reportSkip("nas storage", "transfer disabled in configuration")
} else if details, err := checkNAS(cfg); err != nil {
reportFail("nas storage", details, err)
appLog.Error("nas check failed", logging.KeyError, err.Error())
failures++
} else {
reportPass("nas storage", details)
appLog.Info("nas check passed", "path", cfg.NAS.OutputPath)
}
if failures > 0 {
fmt.Printf("%d check(s) failed.\n", failures)
appLog.Error("connection check finished with failures", "failures", failures)
return 1
}
fmt.Println("All checks passed.")
appLog.Info("connection check passed")
return 0
}
// checkMasterPlaylist fetches the master playlist exactly as the downloader
// would — same headers, same decoder — and summarizes what it found. Details
// gathered before the failure point are returned alongside the error so a
// failed check still shows how far it got.
func checkMasterPlaylist(masterURL string) ([]string, error) {
parsed, err := url.Parse(masterURL)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return nil, fmt.Errorf("not a valid absolute URL: %q", masterURL)
}
details := []string{"host: " + parsed.Host}
ctx, cancel := context.WithTimeout(context.Background(), playlistTimeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, masterURL, nil)
if err != nil {
return details, fmt.Errorf("failed to build request: %w", err)
}
req.Header.Set("User-Agent", constants.HTTPUserAgent)
req.Header.Set("Referer", constants.REFERRER)
client := &http.Client{Timeout: playlistTimeout}
start := time.Now()
resp, err := client.Do(req)
if err != nil {
return details, fmt.Errorf("request failed: %w", err)
}
defer resp.Body.Close()
details = append(details, fmt.Sprintf("HTTP %d in %s", resp.StatusCode, formatDuration(time.Since(start))))
if resp.StatusCode != http.StatusOK {
return details, httpClient.NewHTTPError(resp.StatusCode, "unexpected status fetching master playlist")
}
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
if err != nil {
return details, fmt.Errorf("response is not a valid m3u8 playlist: %w", err)
}
switch listType {
case m3u8.MASTER:
master, ok := playlist.(*m3u8.MasterPlaylist)
if !ok {
return details, fmt.Errorf("decoder returned an unexpected playlist type")
}
if len(master.Variants) == 0 {
return details, fmt.Errorf("master playlist contains no variants")
}
details = append(details, fmt.Sprintf("master playlist with %d variant(s):", len(master.Variants)))
for _, v := range master.Variants {
if v == nil {
continue
}
details = append(details, fmt.Sprintf(" %s (%d bps)", media.ExtractResolution(v), v.Bandwidth))
}
case m3u8.MEDIA:
mediaPlaylist, ok := playlist.(*m3u8.MediaPlaylist)
if !ok {
return details, fmt.Errorf("decoder returned an unexpected playlist type")
}
details = append(details,
fmt.Sprintf("media playlist with %d segment(s), recorded as a single \"unknown\" variant", mediaPlaylist.Count()))
default:
return details, fmt.Errorf("unrecognized playlist type")
}
return details, nil
}
// checkNAS runs the same attach/mkdir/write sequence the transfer service
// performs at startup, so the check exercises the real code path for this host
// — net use on Windows, mount resolution on Linux — rather than a simulation of
// it. It disconnects afterwards so the check leaves no mount behind.
func checkNAS(cfg *config.Config) ([]string, error) {
svc, err := nas.New(nas.NASConfig{
Path: cfg.NAS.OutputPath,
Username: cfg.NAS.Username,
Password: cfg.NAS.Password,
Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
})
if err != nil {
return []string{"path: " + cfg.NAS.OutputPath}, err
}
start := time.Now()
result := <-probeAsync(cfg.NAS.Timeout, func() ([]string, error) {
ctx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout)
defer cancel()
if err := svc.Connect(ctx); err != nil {
return svc.ConnectionInfo(), err
}
defer func() {
if err := svc.Disconnect(context.Background()); err != nil {
log.Printf("Warning: failed to release NAS connection after check: %v", err)
}
}()
return append(svc.ConnectionInfo(), "attach, mkdir and write probe succeeded"), nil
})
details := result.details
if len(details) == 0 {
details = svc.ConnectionInfo()
}
if result.err != nil {
return details, result.err
}
return append(details, "completed in "+formatDuration(time.Since(start))), nil
}
func formatDuration(d time.Duration) string {
if d < time.Millisecond {
return "<1ms"
}
return d.Round(time.Millisecond).String()
}
type probeResult struct {
details []string
err error
}
// probeAsync bounds a filesystem probe with a timeout. An unresponsive SMB
// mount can block indefinitely in the kernel, which would otherwise hang the
// check with no output; the orphaned goroutine dies with the process.
func probeAsync(timeout time.Duration, probe func() ([]string, error)) <-chan probeResult {
out := make(chan probeResult, 1)
done := make(chan probeResult, 1)
go func() {
details, err := probe()
done <- probeResult{details: details, err: err}
}()
go func() {
select {
case res := <-done:
out <- res
case <-time.After(timeout):
out <- probeResult{err: fmt.Errorf("timed out after %s", timeout)}
}
}()
return out
}
func reportPass(name string, details []string) {
fmt.Printf("✓ %s\n", name)
printDetails(details)
}
func reportFail(name string, details []string, err error) {
fmt.Printf("✗ %s\n", name)
printDetails(append(details, "error: "+err.Error()))
}
func reportSkip(name, reason string) {
fmt.Printf("- %s (skipped: %s)\n", name, reason)
fmt.Println()
}
// maxDetailLines caps a single detail entry. Failures from `net use` and
// friends can carry a full usage dump, which buries the actual diagnosis.
const maxDetailLines = 8
func printDetails(details []string) {
for _, d := range details {
lines := strings.Split(strings.TrimRight(d, "\n"), "\n")
truncated := 0
if len(lines) > maxDetailLines {
truncated = len(lines) - maxDetailLines
lines = lines[:maxDetailLines]
}
for _, line := range lines {
fmt.Printf(" %s\n", line)
}
if truncated > 0 {
fmt.Printf(" ... (%d more line(s))\n", truncated)
}
}
fmt.Println()
}
func enabledLabel(enabled bool) string {
if enabled {
return "enabled"
}
return "disabled"
}

View File

@ -1,220 +0,0 @@
package check
import (
"errors"
"fmt"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/httpClient"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
const masterPlaylistBody = `#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=5500000,RESOLUTION=1920x1080
1080p/chunk.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3200000,RESOLUTION=1280x720
720p/chunk.m3u8
`
const mediaPlaylistBody = `#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:10
#EXTINF:4.0,
seg1.ts
#EXTINF:4.0,
seg2.ts
`
// playlistServer serves body at /playlist.m3u8 with the given status, and
// records the request headers so we can assert the check sends the same ones
// the real downloader does.
func playlistServer(t *testing.T, status int, body string) (*httptest.Server, *http.Header) {
t.Helper()
var got http.Header
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got = r.Header.Clone()
w.WriteHeader(status)
fmt.Fprint(w, body)
}))
t.Cleanup(srv.Close)
return srv, &got
}
func joinDetails(details []string) string {
return strings.Join(details, "\n")
}
func TestCheckMasterPlaylistMaster(t *testing.T) {
srv, headers := playlistServer(t, http.StatusOK, masterPlaylistBody)
details, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8")
if err != nil {
t.Fatalf("expected master playlist check to pass, got: %v", err)
}
joined := joinDetails(details)
for _, want := range []string{"master playlist with 2 variant(s)", "1080p (5500000 bps)", "720p (3200000 bps)", "HTTP 200"} {
if !strings.Contains(joined, want) {
t.Errorf("details missing %q:\n%s", want, joined)
}
}
if ua := headers.Get("User-Agent"); !strings.Contains(ua, "Mozilla/5.0") {
t.Errorf("expected downloader User-Agent to be sent, got %q", ua)
}
if ref := headers.Get("Referer"); ref == "" {
t.Error("expected Referer header to be sent")
}
}
func TestCheckMasterPlaylistMedia(t *testing.T) {
srv, _ := playlistServer(t, http.StatusOK, mediaPlaylistBody)
details, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8")
if err != nil {
t.Fatalf("expected media playlist check to pass, got: %v", err)
}
if joined := joinDetails(details); !strings.Contains(joined, "media playlist with 2 segment(s)") {
t.Errorf("expected media playlist segment count in details:\n%s", joined)
}
}
func TestCheckMasterPlaylistNon200(t *testing.T) {
srv, _ := playlistServer(t, http.StatusForbidden, "denied")
details, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8")
if err == nil {
t.Fatal("expected a 403 response to fail the check")
}
if code := httpClient.GetHTTPStatusCode(err); code != http.StatusForbidden {
t.Errorf("expected a structured HTTP 403 error, got code %d from %v", code, err)
}
// Details collected before the failure should still be reported.
if joined := joinDetails(details); !strings.Contains(joined, "HTTP 403") {
t.Errorf("expected partial details on failure:\n%s", joined)
}
}
func TestCheckMasterPlaylistNotAPlaylist(t *testing.T) {
srv, _ := playlistServer(t, http.StatusOK, "this is not a playlist")
if _, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8"); err == nil {
t.Fatal("expected a non-m3u8 body to fail the check")
}
}
func TestCheckMasterPlaylistEmptyMaster(t *testing.T) {
srv, _ := playlistServer(t, http.StatusOK, "#EXTM3U\n")
if _, err := checkMasterPlaylist(srv.URL + "/playlist.m3u8"); err == nil {
t.Fatal("expected a playlist with no variants or segments to fail the check")
}
}
func TestCheckMasterPlaylistInvalidURL(t *testing.T) {
for _, bad := range []string{"", "notaurl", "/relative/path.m3u8", "://missing-scheme"} {
if _, err := checkMasterPlaylist(bad); err == nil {
t.Errorf("expected %q to be rejected as an invalid URL", bad)
}
}
}
func TestCheckMasterPlaylistUnreachable(t *testing.T) {
// Port 9 (discard) refuses connections without a listener.
if _, err := checkMasterPlaylist("http://127.0.0.1:9/playlist.m3u8"); err == nil {
t.Fatal("expected an unreachable host to fail the check")
}
}
func writeFile(path string) error {
return os.WriteFile(path, []byte("blocker"), 0644)
}
func nasConfig(path string) *config.Config {
cfg := &config.Config{}
cfg.NAS.EnableTransfer = true
cfg.NAS.OutputPath = path
cfg.NAS.Timeout = 10 * time.Second
cfg.NAS.RetryLimit = 3
return cfg
}
func TestCheckNASLocalPath(t *testing.T) {
target := filepath.Join(t.TempDir(), "streams")
details, err := checkNAS(nasConfig(target))
if err != nil {
t.Fatalf("expected a writable local path to pass the NAS check, got: %v", err)
}
joined := joinDetails(details)
for _, want := range []string{"configured: " + target, "strategy: local", "write probe succeeded"} {
if !strings.Contains(joined, want) {
t.Errorf("details missing %q:\n%s", want, joined)
}
}
// The probe must not leave its test file behind.
if matches, _ := filepath.Glob(filepath.Join(target, ".connection_test")); len(matches) != 0 {
t.Errorf("connection test file was not cleaned up: %v", matches)
}
}
func TestCheckNASUnusableTarget(t *testing.T) {
// A regular file cannot be turned into the destination directory.
blocker := filepath.Join(t.TempDir(), "not-a-dir")
if err := writeFile(blocker); err != nil {
t.Fatalf("failed to set up test file: %v", err)
}
details, err := checkNAS(nasConfig(filepath.Join(blocker, "streams")))
if err == nil {
t.Fatal("expected an unusable NAS target to fail the check")
}
if joined := joinDetails(details); !strings.Contains(joined, "configured: ") {
t.Errorf("expected path detail on failure:\n%s", joined)
}
}
func TestProbeAsyncTimesOut(t *testing.T) {
start := time.Now()
result := <-probeAsync(50*time.Millisecond, func() ([]string, error) {
time.Sleep(5 * time.Second)
return nil, nil
})
if result.err == nil {
t.Fatal("expected a hung probe to report a timeout")
}
if !strings.Contains(result.err.Error(), "timed out") {
t.Errorf("expected a timeout error, got: %v", result.err)
}
if elapsed := time.Since(start); elapsed > time.Second {
t.Errorf("probe took %s, expected it to abandon the hung call promptly", elapsed)
}
}
func TestProbeAsyncPassesThroughResults(t *testing.T) {
sentinel := errors.New("probe failed")
if result := <-probeAsync(time.Second, func() ([]string, error) { return []string{"ok"}, nil }); result.err != nil || len(result.details) != 1 {
t.Errorf("expected success to pass through, got details=%v err=%v", result.details, result.err)
}
if result := <-probeAsync(time.Second, func() ([]string, error) { return nil, sentinel }); !errors.Is(result.err, sentinel) {
t.Errorf("expected probe error to pass through, got: %v", result.err)
}
}
func TestFormatDuration(t *testing.T) {
if got := formatDuration(100 * time.Microsecond); got != "<1ms" {
t.Errorf("sub-millisecond duration = %q, want \"<1ms\"", got)
}
if got := formatDuration(1500 * time.Millisecond); got != "1.5s" {
t.Errorf("1.5s duration = %q", got)
}
}

View File

@ -4,11 +4,8 @@ import (
"context" "context"
"log" "log"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/media" "m3u8-downloader/pkg/media"
"m3u8-downloader/pkg/transfer" "m3u8-downloader/pkg/transfer"
"m3u8-downloader/pkg/utils"
"os" "os"
"os/signal" "os/signal"
"sync" "sync"
@ -17,8 +14,6 @@ import (
) )
func Download(masterURL string, eventName string, debug bool) { func Download(masterURL string, eventName string, debug bool) {
appLog := logging.MustGetFactory().App
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
defer cancel() defer cancel()
@ -28,34 +23,16 @@ func Download(masterURL string, eventName string, debug bool) {
go func() { go func() {
<-sigChan <-sigChan
log.Println("Shutting down...") log.Println("Shutting down...")
appLog.Info("shutdown signal received")
cancel() cancel()
}() }()
cfg := constants.MustGetConfig()
if cfg.Health.Enabled {
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
var wg sync.WaitGroup var wg sync.WaitGroup
var transferService *transfer.TransferService var transferService *transfer.TransferService
if cfg.NAS.EnableTransfer { if constants.EnableNASTransfer {
ts, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName) ts, err := transfer.NewTrasferService(constants.NASOutputPath, eventName)
if err != nil { if err != nil {
log.Printf("Failed to create transfer service: %v", err) log.Printf("Failed to create transfer service: %v", err)
log.Println("Continuing without transfer service...") log.Println("Continuing without transfer service...")
appLog.Error("failed to create transfer service, continuing without it", logging.KeyError, err.Error())
} else { } else {
transferService = ts transferService = ts
wg.Add(1) wg.Add(1)
@ -63,33 +40,24 @@ func Download(masterURL string, eventName string, debug bool) {
defer wg.Done() defer wg.Done()
if err := transferService.Start(ctx); err != nil && err != context.Canceled { if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err) log.Printf("Transfer service error: %v", err)
appLog.Error("transfer service error", logging.KeyError, err.Error())
} }
}() }()
log.Println("Transfer service started.") log.Println("Transfer service started.")
appLog.Info("transfer service started")
} }
} }
// One writer for the whole run: every variant records into it, and it is manifestWriter := media.NewManifestWriter(eventName)
// the one written to disk at the end.
manifest := media.NewManifestWriter(eventName)
eventPath := cfg.GetEventPath(eventName) variants, err := media.GetAllVariants(masterURL, constants.LocalOutputDirPath+"/"+eventName, manifestWriter)
if err := utils.EnsureDir(eventPath); err != nil {
appLog.Error("failed to create event directory", logging.KeyError, err.Error())
log.Fatalf("Failed to create event directory: %v", err)
}
variants, err := media.GetAllVariants(masterURL, eventPath)
if err != nil { if err != nil {
appLog.Error("failed to get variants", logging.KeyError, err.Error())
log.Fatalf("Failed to get variants: %v", err) log.Fatalf("Failed to get variants: %v", err)
} }
log.Printf("Found %d variants", len(variants)) log.Printf("Found %d variants", len(variants))
sem := make(chan struct{}, constants.WorkerCount*len(variants)) sem := make(chan struct{}, constants.WorkerCount*len(variants))
manifest := media.NewManifestWriter(eventName)
for _, variant := range variants { for _, variant := range variants {
// Debug mode only tracks one variant for easier debugging // Debug mode only tracks one variant for easier debugging
if debug { if debug {
@ -106,7 +74,6 @@ func Download(masterURL string, eventName string, debug bool) {
wg.Wait() wg.Wait()
log.Println("All variant downloaders finished.") log.Println("All variant downloaders finished.")
appLog.Info("all variant downloaders finished")
if transferService != nil { if transferService != nil {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second) shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
@ -115,9 +82,7 @@ func Download(masterURL string, eventName string, debug bool) {
} }
log.Println("All Services shut down.") log.Println("All Services shut down.")
appLog.Info("all services shut down")
manifest.WriteManifest() manifestWriter.WriteManifest()
log.Println("Manifest written.") log.Println("Manifest written.")
appLog.Info("manifest written", logging.KeyEvent, eventName, "segments", manifest.Len())
} }

View File

@ -4,38 +4,23 @@ import (
"bufio" "bufio"
"flag" "flag"
"fmt" "fmt"
"m3u8-downloader/cmd/check"
"m3u8-downloader/cmd/downloader" "m3u8-downloader/cmd/downloader"
"m3u8-downloader/cmd/processor" "m3u8-downloader/cmd/processor"
"m3u8-downloader/cmd/transfer" "m3u8-downloader/cmd/transfer"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"os" "os"
"strings" "strings"
"time"
) )
func main() { func main() {
url := flag.String("url", "", "M3U8 playlist URL") url := flag.String("url", "", "M3U8 playlist URL")
eventName := flag.String("event", "", "Event name") eventName := flag.String("event", time.Now().Format("2006-01-02"), "Event name")
debug := flag.Bool("debug", false, "Enable debug mode") debug := flag.Bool("debug", false, "Enable debug mode")
transferOnly := flag.Bool("transfer", false, "Transfer-only mode: transfer existing files without downloading") transferOnly := flag.Bool("transfer", false, "Transfer-only mode: transfer existing files without downloading")
processOnly := flag.Bool("process", false, "Process-only mode: process existing files without downloading") processOnly := flag.Bool("process", false, "Process-only mode: process existing files without downloading")
checkOnly := flag.Bool("check", false, "Connection-check mode: test the stream URL and NAS connectivity, then exit")
flag.Parse() flag.Parse()
// Must happen before the first logging.MustGetFactory() call anywhere in
// the program, since the logging factory reads its level from this
// config singleton once and caches it.
if *debug {
constants.MustGetConfig().Logging.Level = "debug"
}
logging.MustGetFactory().App.Info("m3u8-downloader starting", "debug", *debug)
if *checkOnly {
os.Exit(check.Run(*url))
}
if *transferOnly { if *transferOnly {
transfer.RunTransferOnly(*eventName) transfer.RunTransferOnly(*eventName)
return return

View File

@ -3,40 +3,16 @@ package processor
import ( import (
"context" "context"
"log" "log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/processing" "m3u8-downloader/pkg/processing"
"time"
) )
func Process(eventName string) { func Process(eventName string) {
appLog := logging.MustGetFactory().App //log.Printf("Starting processing for event: %s", eventName)
log.Printf("Starting processing for event: %s", eventName) ps, err := processing.NewProcessingService(eventName)
appLog.Info("starting processing", logging.KeyEvent, eventName)
cfg := constants.MustGetConfig()
if cfg.Health.Enabled {
healthSrv, err := health.Start(context.Background(), &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
ps, err := processing.NewProcessingService(eventName, cfg)
if err != nil { if err != nil {
appLog.Error("failed to create processing service", logging.KeyError, err.Error())
log.Fatalf("Failed to create processing service: %v", err) log.Fatalf("Failed to create processing service: %v", err)
} }
if err := ps.Start(context.Background()); err != nil { if err := ps.Start(context.Background()); err != nil {
appLog.Error("processing service failed", logging.KeyError, err.Error())
log.Fatalf("Failed to run processing service: %v", err) log.Fatalf("Failed to run processing service: %v", err)
} }
} }

View File

@ -1,77 +1,17 @@
package transfer package transfer
import ( import (
"bufio"
"context" "context"
"fmt"
"log" "log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/transfer" "m3u8-downloader/pkg/transfer"
"m3u8-downloader/pkg/utils"
"os" "os"
"os/signal" "os/signal"
"strconv"
"strings"
"syscall" "syscall"
"time" "time"
) )
func getEventDirs(cfg *config.Config) ([]string, error) {
dirs, err := os.ReadDir(cfg.Paths.LocalOutput)
if err != nil {
return nil, fmt.Errorf("failed to read directory: %w", err)
}
var eventDirs []string
for _, dir := range dirs {
if dir.IsDir() {
eventDirs = append(eventDirs, dir.Name())
}
}
return eventDirs, nil
}
func RunTransferOnly(eventName string) { func RunTransferOnly(eventName string) {
appLog := logging.MustGetFactory().App
cfg := constants.MustGetConfig()
// Check if NAS transfer is enabled
if !cfg.NAS.EnableTransfer {
appLog.Error("NAS transfer disabled, cannot run transfer-only mode")
log.Fatal("NAS transfer is disabled in configuration. Please enable it to use transfer-only mode.")
}
if eventName == "" {
events, err := getEventDirs(cfg)
if err != nil {
log.Fatalf("Failed to get event directories: %v", err)
}
if len(events) == 0 {
log.Fatal("No events found")
}
if len(events) > 1 {
fmt.Println("Multiple events found, please select one:")
for i, event := range events {
fmt.Printf("%d. %s\n", i+1, event)
}
reader := bufio.NewReader(os.Stdin)
input, _ := reader.ReadString('\n')
input = strings.TrimSpace(input)
index, err := strconv.Atoi(input)
if err != nil {
log.Fatalf("Failed to parse input: %v", err)
}
if index < 1 || index > len(events) {
log.Fatal("Invalid input")
}
eventName = events[index-1]
} else {
eventName = events[0]
}
}
log.Printf("Starting transfer-only mode for event: %s", eventName) log.Printf("Starting transfer-only mode for event: %s", eventName)
// Setup context and signal handling // Setup context and signal handling
@ -83,50 +23,35 @@ func RunTransferOnly(eventName string) {
go func() { go func() {
<-sigChan <-sigChan
log.Println("Shutting down transfer service...") log.Println("Shutting down transfer service...")
appLog.Info("shutdown signal received")
cancel() cancel()
}() }()
if cfg.Health.Enabled { // Check if NAS transfer is enabled
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry()) if !constants.EnableNASTransfer {
if err != nil { log.Fatal("NAS transfer is disabled in constants. Please enable it to use transfer-only mode.")
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
} }
// Verify local event directory exists // Verify local event directory exists
localEventPath := cfg.GetEventPath(eventName) localEventPath := constants.LocalOutputDirPath + "/" + eventName
if !utils.PathExists(localEventPath) { if _, err := os.Stat(localEventPath); os.IsNotExist(err) {
appLog.Error("local event directory does not exist", logging.KeyEvent, eventName)
log.Fatalf("Local event directory does not exist: %s", localEventPath) log.Fatalf("Local event directory does not exist: %s", localEventPath)
} }
// Create transfer service // Create transfer service
transferService, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName) transferService, err := transfer.NewTrasferService(constants.NASOutputPath, eventName)
if err != nil { if err != nil {
appLog.Error("failed to create transfer service", logging.KeyError, err.Error())
log.Fatalf("Failed to create transfer service: %v", err) log.Fatalf("Failed to create transfer service: %v", err)
} }
// Find and queue existing files // Find and queue existing files
if err := transferService.QueueExistingFiles(localEventPath); err != nil { if err := transferService.QueueExistingFiles(localEventPath); err != nil {
appLog.Error("failed to queue existing files", logging.KeyError, err.Error())
log.Fatalf("Failed to queue existing files: %v", err) log.Fatalf("Failed to queue existing files: %v", err)
} }
// Start transfer service // Start transfer service
log.Println("Starting transfer service...") log.Println("Starting transfer service...")
appLog.Info("starting transfer service", logging.KeyEvent, eventName)
if err := transferService.Start(ctx); err != nil && err != context.Canceled { if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err) log.Printf("Transfer service error: %v", err)
appLog.Error("transfer service error", logging.KeyError, err.Error())
} }
// Graceful shutdown // Graceful shutdown
@ -135,5 +60,4 @@ func RunTransferOnly(eventName string) {
transferService.Shutdown(shutdownCtx) transferService.Shutdown(shutdownCtx)
log.Println("Transfer-only mode completed.") log.Println("Transfer-only mode completed.")
appLog.Info("transfer-only mode completed", logging.KeyEvent, eventName)
} }

View File

@ -1,305 +0,0 @@
package config
import (
"fmt"
"os"
"path/filepath"
"strconv"
"time"
)
type Config struct {
Core CoreConfig
HTTP HTTPConfig
NAS NASConfig
Processing ProcessingConfig
Transfer TransferConfig
Cleanup CleanupConfig
Paths PathsConfig
Logging LoggingConfig
Health HealthConfig
}
type CoreConfig struct {
WorkerCount int
RefreshDelay time.Duration
}
type HTTPConfig struct {
UserAgent string
Referer string
}
type NASConfig struct {
EnableTransfer bool
// OutputPath is the destination as configured: a UNC path, an smb:// URL,
// or a directory on this host. How it is reached depends on the platform;
// see pkg/nas.
OutputPath string
Username string
Password string
Timeout time.Duration
RetryLimit int
// AutoMount lets the recorder mount an SMB share itself when the host has
// not already mounted it. Linux only, and requires root.
AutoMount bool
// MountPoint overrides where AutoMount attaches the share.
MountPoint string
}
type ProcessingConfig struct {
Enabled bool
AutoProcess bool
WorkerCount int
FFmpegPath string
}
type TransferConfig struct {
WorkerCount int
RetryLimit int
Timeout time.Duration
FileSettlingDelay time.Duration
QueueSize int
BatchSize int
}
type CleanupConfig struct {
AfterTransfer bool
BatchSize int
RetainHours int
}
type PathsConfig struct {
BaseDir string
LocalOutput string
ProcessOutput string
ManifestDir string
PersistenceFile string
}
type LoggingConfig struct {
Dir string
Level string
}
type HealthConfig struct {
Enabled bool
Addr string
}
var defaultConfig = Config{
Core: CoreConfig{
WorkerCount: 4,
RefreshDelay: 3 * time.Second,
},
HTTP: HTTPConfig{
UserAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
Referer: "https://www.flomarching.com",
},
NAS: NASConfig{
EnableTransfer: true,
OutputPath: "\\\\HomeLabNAS\\dci\\streams",
Username: "NASAdmin",
Password: "s3tkY6tzA&KN6M",
Timeout: 30 * time.Second,
RetryLimit: 3,
AutoMount: false,
MountPoint: "",
},
Processing: ProcessingConfig{
Enabled: true,
AutoProcess: true,
WorkerCount: 2,
FFmpegPath: "ffmpeg",
},
Transfer: TransferConfig{
WorkerCount: 2,
RetryLimit: 3,
Timeout: 30 * time.Second,
FileSettlingDelay: 5 * time.Second,
QueueSize: 100000,
BatchSize: 1000,
},
Cleanup: CleanupConfig{
AfterTransfer: true,
BatchSize: 1000,
RetainHours: 0,
},
Paths: PathsConfig{
BaseDir: "data",
LocalOutput: "data",
ProcessOutput: "out",
ManifestDir: "data",
PersistenceFile: "transfer_queue.json",
},
Logging: LoggingConfig{
Dir: "data/logs",
Level: "info",
},
Health: HealthConfig{
Enabled: true,
Addr: ":8090",
},
}
func Load() (*Config, error) {
cfg := defaultConfig
if err := cfg.loadFromEnvironment(); err != nil {
return nil, fmt.Errorf("failed to load environment config: %w", err)
}
if err := cfg.resolveAndValidatePaths(); err != nil {
return nil, fmt.Errorf("path validation failed: %w", err)
}
return &cfg, nil
}
func (c *Config) loadFromEnvironment() error {
if val := os.Getenv("WORKER_COUNT"); val != "" {
if parsed, err := strconv.Atoi(val); err == nil {
c.Core.WorkerCount = parsed
}
}
if val := os.Getenv("REFRESH_DELAY_SECONDS"); val != "" {
if parsed, err := strconv.Atoi(val); err == nil {
c.Core.RefreshDelay = time.Duration(parsed) * time.Second
}
}
if val := os.Getenv("NAS_OUTPUT_PATH"); val != "" {
c.NAS.OutputPath = val
}
if val := os.Getenv("NAS_USERNAME"); val != "" {
c.NAS.Username = val
}
if val := os.Getenv("NAS_PASSWORD"); val != "" {
c.NAS.Password = val
}
if val := os.Getenv("ENABLE_NAS_TRANSFER"); val != "" {
c.NAS.EnableTransfer = val == "true"
}
if val := os.Getenv("NAS_AUTO_MOUNT"); val != "" {
c.NAS.AutoMount = val == "true"
}
if val := os.Getenv("NAS_MOUNT_POINT"); val != "" {
c.NAS.MountPoint = val
}
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
c.Paths.LocalOutput = val
// The manifest describes the event directory and is documented as
// sitting beside it, so it follows the output directory. Without this
// it stays in the default data/ while the segments move elsewhere.
c.Paths.ManifestDir = val
}
// Applied after LOCAL_OUTPUT_DIR so an explicit setting wins.
if val := os.Getenv("MANIFEST_DIR"); val != "" {
c.Paths.ManifestDir = val
}
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {
c.Paths.ProcessOutput = val
}
if val := os.Getenv("FFMPEG_PATH"); val != "" {
c.Processing.FFmpegPath = val
}
if val := os.Getenv("LOG_DIR"); val != "" {
c.Logging.Dir = val
}
if val := os.Getenv("LOG_LEVEL"); val != "" {
c.Logging.Level = val
}
if val := os.Getenv("HEALTH_ENABLED"); val != "" {
c.Health.Enabled = val == "true"
}
if val := os.Getenv("HEALTH_ADDR"); val != "" {
c.Health.Addr = val
}
return nil
}
func (c *Config) resolveAndValidatePaths() error {
cwd, err := os.Getwd()
if err != nil {
return fmt.Errorf("failed to get working directory: %w", err)
}
// Only join with cwd if path is not already absolute
if !filepath.IsAbs(c.Paths.BaseDir) {
c.Paths.BaseDir = filepath.Join(cwd, c.Paths.BaseDir)
}
if !filepath.IsAbs(c.Paths.LocalOutput) {
c.Paths.LocalOutput = filepath.Join(cwd, c.Paths.LocalOutput)
}
if !filepath.IsAbs(c.Paths.ProcessOutput) {
c.Paths.ProcessOutput = filepath.Join(cwd, c.Paths.ProcessOutput)
}
if !filepath.IsAbs(c.Paths.ManifestDir) {
c.Paths.ManifestDir = filepath.Join(cwd, c.Paths.ManifestDir)
}
if !filepath.IsAbs(c.Paths.PersistenceFile) {
c.Paths.PersistenceFile = filepath.Join(c.Paths.BaseDir, c.Paths.PersistenceFile)
}
if !filepath.IsAbs(c.Logging.Dir) {
c.Logging.Dir = filepath.Join(cwd, c.Logging.Dir)
}
requiredDirs := []string{
c.Paths.BaseDir,
c.Paths.LocalOutput,
c.Paths.ProcessOutput,
c.Paths.ManifestDir,
c.Logging.Dir,
}
for _, dir := range requiredDirs {
if err := os.MkdirAll(dir, 0755); err != nil {
return fmt.Errorf("failed to create directory %s: %w", dir, err)
}
}
if c.NAS.EnableTransfer && c.NAS.OutputPath == "" {
return fmt.Errorf("NAS output path is required when transfer is enabled")
}
if c.Processing.Enabled && c.Processing.FFmpegPath == "" {
return fmt.Errorf("FFmpeg path is required when processing is enabled")
}
return nil
}
func (c *Config) GetEventPath(eventName string) string {
return filepath.Join(c.Paths.LocalOutput, eventName)
}
func (c *Config) GetManifestPath(eventName string) string {
return filepath.Join(c.Paths.ManifestDir, eventName+".json")
}
func (c *Config) GetNASEventPath(eventName string) string {
return filepath.Join(c.NAS.OutputPath, eventName)
}
func (c *Config) GetProcessOutputPath(eventName string) string {
return filepath.Join(c.Paths.ProcessOutput, eventName)
}
func (c *Config) GetQualityPath(eventName, quality string) string {
return filepath.Join(c.GetEventPath(eventName), quality)
}

View File

@ -1,181 +0,0 @@
package config
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestConfig_Load(t *testing.T) {
// Save original env vars
originalVars := map[string]string{
"WORKER_COUNT": os.Getenv("WORKER_COUNT"),
"NAS_USERNAME": os.Getenv("NAS_USERNAME"),
"LOCAL_OUTPUT_DIR": os.Getenv("LOCAL_OUTPUT_DIR"),
"ENABLE_NAS_TRANSFER": os.Getenv("ENABLE_NAS_TRANSFER"),
}
defer func() {
// Restore original env vars
for key, value := range originalVars {
if value == "" {
os.Unsetenv(key)
} else {
os.Setenv(key, value)
}
}
}()
// Test default config load
cfg, err := Load()
if err != nil {
t.Fatalf("Load() failed: %v", err)
}
// Verify defaults
if cfg.Core.WorkerCount != 4 {
t.Errorf("Expected WorkerCount=4, got %d", cfg.Core.WorkerCount)
}
if cfg.Core.RefreshDelay != 3*time.Second {
t.Errorf("Expected RefreshDelay=3s, got %v", cfg.Core.RefreshDelay)
}
if !cfg.NAS.EnableTransfer {
t.Errorf("Expected NAS.EnableTransfer=true, got false")
}
// Test environment variable override
os.Setenv("WORKER_COUNT", "8")
os.Setenv("NAS_USERNAME", "testuser")
os.Setenv("ENABLE_NAS_TRANSFER", "false")
os.Setenv("LOCAL_OUTPUT_DIR", "custom_data")
cfg2, err := Load()
if err != nil {
t.Fatalf("Load() with env vars failed: %v", err)
}
if cfg2.Core.WorkerCount != 8 {
t.Errorf("Expected WorkerCount=8 from env, got %d", cfg2.Core.WorkerCount)
}
if cfg2.NAS.Username != "testuser" {
t.Errorf("Expected NAS.Username='testuser' from env, got %s", cfg2.NAS.Username)
}
if cfg2.NAS.EnableTransfer {
t.Errorf("Expected NAS.EnableTransfer=false from env, got true")
}
if !strings.Contains(cfg2.Paths.LocalOutput, "custom_data") {
t.Errorf("Expected LocalOutput to contain 'custom_data', got %s", cfg2.Paths.LocalOutput)
}
}
func TestConfig_PathMethods(t *testing.T) {
cfg, err := Load()
if err != nil {
t.Fatalf("Load() failed: %v", err)
}
testEvent := "test-event"
testQuality := "1080p"
// Test GetEventPath
eventPath := cfg.GetEventPath(testEvent)
if !strings.Contains(eventPath, testEvent) {
t.Errorf("GetEventPath should contain event name, got %s", eventPath)
}
// Test GetManifestPath
manifestPath := cfg.GetManifestPath(testEvent)
if !strings.Contains(manifestPath, testEvent) {
t.Errorf("GetManifestPath should contain event name, got %s", manifestPath)
}
if !strings.HasSuffix(manifestPath, ".json") {
t.Errorf("GetManifestPath should end with .json, got %s", manifestPath)
}
// Test GetNASEventPath
nasPath := cfg.GetNASEventPath(testEvent)
if !strings.Contains(nasPath, testEvent) {
t.Errorf("GetNASEventPath should contain event name, got %s", nasPath)
}
// Test GetProcessOutputPath
processPath := cfg.GetProcessOutputPath(testEvent)
if !strings.Contains(processPath, testEvent) {
t.Errorf("GetProcessOutputPath should contain event name, got %s", processPath)
}
// Test GetQualityPath
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
if !strings.Contains(qualityPath, testEvent) {
t.Errorf("GetQualityPath should contain event name, got %s", qualityPath)
}
if !strings.Contains(qualityPath, testQuality) {
t.Errorf("GetQualityPath should contain quality, got %s", qualityPath)
}
}
func TestConfig_PathValidation(t *testing.T) {
// Create a temporary directory for testing
tempDir, err := os.MkdirTemp("", "config_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Set environment variables to use temp directory
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
cfg, err := Load()
if err != nil {
t.Fatalf("Load() failed: %v", err)
}
// Verify directories were created
if _, err := os.Stat(cfg.Paths.LocalOutput); os.IsNotExist(err) {
t.Errorf("LocalOutput directory should have been created: %s", cfg.Paths.LocalOutput)
}
if _, err := os.Stat(cfg.Paths.ProcessOutput); os.IsNotExist(err) {
t.Errorf("ProcessOutput directory should have been created: %s", cfg.Paths.ProcessOutput)
}
}
func TestConfig_ValidationErrors(t *testing.T) {
// Save original env vars
originalNASPath := os.Getenv("NAS_OUTPUT_PATH")
originalFFmpegPath := os.Getenv("FFMPEG_PATH")
defer func() {
if originalNASPath == "" {
os.Unsetenv("NAS_OUTPUT_PATH")
} else {
os.Setenv("NAS_OUTPUT_PATH", originalNASPath)
}
if originalFFmpegPath == "" {
os.Unsetenv("FFMPEG_PATH")
} else {
os.Setenv("FFMPEG_PATH", originalFFmpegPath)
}
}()
// Note: Validation tests are limited because the default config
// has working defaults. We can test that Load() works with valid configs.
// Test that Load works with proper paths set
tempDir2, err := os.MkdirTemp("", "config_validation_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir2)
os.Setenv("NAS_OUTPUT_PATH", "\\\\test\\path")
os.Setenv("LOCAL_OUTPUT_DIR", tempDir2)
cfg, err := Load()
if err != nil {
t.Errorf("Load() should work with valid config: %v", err)
}
if cfg == nil {
t.Error("Config should not be nil")
}
}

View File

@ -1,51 +1,34 @@
package constants package constants
import ( import "time"
"m3u8-downloader/pkg/config"
"sync"
)
var (
globalConfig *config.Config
configOnce sync.Once
configError error
)
func GetConfig() (*config.Config, error) {
configOnce.Do(func() {
globalConfig, configError = config.Load()
})
return globalConfig, configError
}
func MustGetConfig() *config.Config {
cfg, err := GetConfig()
if err != nil {
panic("Failed to load configuration: " + err.Error())
}
return cfg
}
const ( const (
WorkerCount = 4 WorkerCount = 4
RefreshDelay = 3 RefreshDelay = 3 * time.Second
HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
REFERRER = "https://www.flomarching.com" REFERRER = "https://www.flomarching.com"
LocalOutputDirPath = "../data/"
DefaultNASOutputPath = "\\\\HomeLabNAS\\dci\\streams" EnableNASTransfer = false
DefaultNASUsername = "NASAdmin" NASOutputPath = "\\\\HomeLabNAS\\dci\\streams"
NASUsername = "NASAdmin"
NASPassword = "s3tkY6tzA&KN6M"
TransferWorkerCount = 2
TransferRetryLimit = 3
TransferTimeout = 30 * time.Second
FileSettlingDelay = 5 * time.Second
PersistencePath = "../data/transfer_queue.json"
TransferQueueSize = 100000
BatchSize = 1000
ManifestPath = "../data"
DefaultTransferWorkerCount = 2 CleanupAfterTransfer = false
DefaultTransferRetryLimit = 3 CleanupBatchSize = 1000
DefaultTransferTimeout = 30 RetainLocalHours = 0
DefaultFileSettlingDelay = 5
DefaultTransferQueueSize = 100000
DefaultBatchSize = 1000
DefaultCleanupBatchSize = 1000 ProcessOutputPath = "../out"
DefaultRetainLocalHours = 0 AutoProcess = true
ProcessingEnabled = true
DefaultProcessWorkerCount = 2 ProcessWorkerCount = 2
DefaultFFmpegPath = "ffmpeg"
) )

View File

@ -1,240 +0,0 @@
package constants
import (
"os"
"strings"
"testing"
"time"
)
func TestGetConfig(t *testing.T) {
// Test successful config loading
cfg, err := GetConfig()
if err != nil {
t.Fatalf("GetConfig() failed: %v", err)
}
if cfg == nil {
t.Fatal("GetConfig() returned nil config")
}
// Test that subsequent calls return the same instance (singleton)
cfg2, err := GetConfig()
if err != nil {
t.Fatalf("Second GetConfig() call failed: %v", err)
}
// Both should be the same instance due to sync.Once
if cfg != cfg2 {
t.Error("GetConfig() should return the same instance (singleton)")
}
}
func TestMustGetConfig(t *testing.T) {
// This should not panic with valid environment
cfg := MustGetConfig()
if cfg == nil {
t.Fatal("MustGetConfig() returned nil")
}
// Verify it returns a properly initialized config
if cfg.Core.WorkerCount <= 0 {
t.Errorf("Expected positive WorkerCount, got %d", cfg.Core.WorkerCount)
}
if cfg.Core.RefreshDelay <= 0 {
t.Errorf("Expected positive RefreshDelay, got %v", cfg.Core.RefreshDelay)
}
}
func TestMustGetConfig_Panic(t *testing.T) {
// We can't easily test the panic scenario without breaking the singleton,
// but we can test that MustGetConfig works normally
defer func() {
if r := recover(); r != nil {
t.Errorf("MustGetConfig() panicked unexpectedly: %v", r)
}
}()
cfg := MustGetConfig()
if cfg == nil {
t.Fatal("MustGetConfig() returned nil without panicking")
}
}
func TestConfigSingleton(t *testing.T) {
// Reset the singleton for this test (this is a bit hacky but necessary for testing)
// We'll create multiple goroutines to test concurrent access
configs := make(chan interface{}, 10)
// Launch multiple goroutines to call GetConfig concurrently
for i := 0; i < 10; i++ {
go func() {
cfg, _ := GetConfig()
configs <- cfg
}()
}
// Collect all configs
var allConfigs []interface{}
for i := 0; i < 10; i++ {
allConfigs = append(allConfigs, <-configs)
}
// All should be the same instance
firstConfig := allConfigs[0]
for i, cfg := range allConfigs {
if cfg != firstConfig {
t.Errorf("Config %d is different from first config", i)
}
}
}
func TestConstants_Values(t *testing.T) {
// Test that constants have expected values
if WorkerCount != 4 {
t.Errorf("Expected WorkerCount=4, got %d", WorkerCount)
}
if RefreshDelay != 3 {
t.Errorf("Expected RefreshDelay=3, got %d", RefreshDelay)
}
// Test HTTP constants
if HTTPUserAgent == "" {
t.Error("HTTPUserAgent should not be empty")
}
if !strings.Contains(HTTPUserAgent, "Mozilla") {
t.Error("HTTPUserAgent should contain 'Mozilla'")
}
if REFERRER != "https://www.flomarching.com" {
t.Errorf("Expected REFERRER='https://www.flomarching.com', got '%s'", REFERRER)
}
// Test default NAS constants
if DefaultNASOutputPath != "\\\\HomeLabNAS\\dci\\streams" {
t.Errorf("Expected DefaultNASOutputPath='\\\\HomeLabNAS\\dci\\streams', got '%s'", DefaultNASOutputPath)
}
if DefaultNASUsername != "NASAdmin" {
t.Errorf("Expected DefaultNASUsername='NASAdmin', got '%s'", DefaultNASUsername)
}
// Test transfer constants
if DefaultTransferWorkerCount != 2 {
t.Errorf("Expected DefaultTransferWorkerCount=2, got %d", DefaultTransferWorkerCount)
}
if DefaultTransferRetryLimit != 3 {
t.Errorf("Expected DefaultTransferRetryLimit=3, got %d", DefaultTransferRetryLimit)
}
if DefaultTransferTimeout != 30 {
t.Errorf("Expected DefaultTransferTimeout=30, got %d", DefaultTransferTimeout)
}
if DefaultFileSettlingDelay != 5 {
t.Errorf("Expected DefaultFileSettlingDelay=5, got %d", DefaultFileSettlingDelay)
}
if DefaultTransferQueueSize != 100000 {
t.Errorf("Expected DefaultTransferQueueSize=100000, got %d", DefaultTransferQueueSize)
}
if DefaultBatchSize != 1000 {
t.Errorf("Expected DefaultBatchSize=1000, got %d", DefaultBatchSize)
}
// Test cleanup constants
if DefaultCleanupBatchSize != 1000 {
t.Errorf("Expected DefaultCleanupBatchSize=1000, got %d", DefaultCleanupBatchSize)
}
if DefaultRetainLocalHours != 0 {
t.Errorf("Expected DefaultRetainLocalHours=0, got %d", DefaultRetainLocalHours)
}
// Test processing constants
if DefaultProcessWorkerCount != 2 {
t.Errorf("Expected DefaultProcessWorkerCount=2, got %d", DefaultProcessWorkerCount)
}
if DefaultFFmpegPath != "ffmpeg" {
t.Errorf("Expected DefaultFFmpegPath='ffmpeg', got '%s'", DefaultFFmpegPath)
}
}
func TestConfig_Integration(t *testing.T) {
cfg := MustGetConfig()
// Test that config values match or override constants appropriately
if cfg.Core.WorkerCount != WorkerCount && os.Getenv("WORKER_COUNT") == "" {
t.Errorf("Config WorkerCount (%d) should match constant (%d) when no env override", cfg.Core.WorkerCount, WorkerCount)
}
if cfg.Core.RefreshDelay != time.Duration(RefreshDelay)*time.Second && os.Getenv("REFRESH_DELAY_SECONDS") == "" {
t.Errorf("Config RefreshDelay (%v) should match constant (%v) when no env override", cfg.Core.RefreshDelay, time.Duration(RefreshDelay)*time.Second)
}
// Test HTTP settings
if cfg.HTTP.UserAgent != HTTPUserAgent {
t.Errorf("Config UserAgent (%s) should match constant (%s)", cfg.HTTP.UserAgent, HTTPUserAgent)
}
if cfg.HTTP.Referer != REFERRER {
t.Errorf("Config Referer (%s) should match constant (%s)", cfg.HTTP.Referer, REFERRER)
}
}
func TestConfig_PathMethods(t *testing.T) {
cfg := MustGetConfig()
testEvent := "test-event-123"
testQuality := "1080p"
// Test GetEventPath
eventPath := cfg.GetEventPath(testEvent)
if !strings.Contains(eventPath, testEvent) {
t.Errorf("GetEventPath should contain event name '%s', got: %s", testEvent, eventPath)
}
// Test GetManifestPath
manifestPath := cfg.GetManifestPath(testEvent)
if !strings.Contains(manifestPath, testEvent) {
t.Errorf("GetManifestPath should contain event name '%s', got: %s", testEvent, manifestPath)
}
if !strings.HasSuffix(manifestPath, ".json") {
t.Errorf("GetManifestPath should end with '.json', got: %s", manifestPath)
}
// Test GetNASEventPath
nasPath := cfg.GetNASEventPath(testEvent)
if !strings.Contains(nasPath, testEvent) {
t.Errorf("GetNASEventPath should contain event name '%s', got: %s", testEvent, nasPath)
}
// Test GetProcessOutputPath
processPath := cfg.GetProcessOutputPath(testEvent)
if !strings.Contains(processPath, testEvent) {
t.Errorf("GetProcessOutputPath should contain event name '%s', got: %s", testEvent, processPath)
}
// Test GetQualityPath
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
if !strings.Contains(qualityPath, testEvent) {
t.Errorf("GetQualityPath should contain event name '%s', got: %s", testEvent, qualityPath)
}
if !strings.Contains(qualityPath, testQuality) {
t.Errorf("GetQualityPath should contain quality '%s', got: %s", testQuality, qualityPath)
}
}
func TestConfig_DefaultValues(t *testing.T) {
cfg := MustGetConfig()
// Test that default values are reasonable
if cfg.Transfer.QueueSize != DefaultTransferQueueSize {
t.Errorf("Expected transfer queue size %d, got %d", DefaultTransferQueueSize, cfg.Transfer.QueueSize)
}
if cfg.Transfer.BatchSize != DefaultBatchSize {
t.Errorf("Expected transfer batch size %d, got %d", DefaultBatchSize, cfg.Transfer.BatchSize)
}
if cfg.Processing.WorkerCount != DefaultProcessWorkerCount {
t.Errorf("Expected processing worker count %d, got %d", DefaultProcessWorkerCount, cfg.Processing.WorkerCount)
}
if cfg.Cleanup.BatchSize != DefaultCleanupBatchSize {
t.Errorf("Expected cleanup batch size %d, got %d", DefaultCleanupBatchSize, cfg.Cleanup.BatchSize)
}
}

View File

@ -1,323 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>m3u8-downloader status</title>
<style>
:root {
--bg: #0f1115;
--panel: #161922;
--border: #262b38;
--text: #e6e8ee;
--muted: #8b909e;
--green: #3ecf8e;
--red: #f2545b;
--amber: #f2b705;
--blue: #4f8ef7;
}
* { box-sizing: border-box; }
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: var(--bg);
color: var(--text);
}
header {
display: flex;
align-items: center;
justify-content: space-between;
flex-wrap: wrap;
gap: 12px;
padding: 16px 24px;
border-bottom: 1px solid var(--border);
}
header h1 {
font-size: 18px;
margin: 0;
font-weight: 600;
}
.header-meta {
display: flex;
align-items: center;
gap: 16px;
color: var(--muted);
font-size: 13px;
}
.badge {
display: inline-flex;
align-items: center;
gap: 6px;
padding: 4px 12px;
border-radius: 999px;
font-size: 13px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.03em;
}
.badge-dot { width: 8px; height: 8px; border-radius: 50%; }
.badge.healthy { background: rgba(62,207,142,0.15); color: var(--green); }
.badge.healthy .badge-dot { background: var(--green); }
.badge.degraded { background: rgba(242,84,91,0.15); color: var(--red); }
.badge.degraded .badge-dot { background: var(--red); }
.badge.unknown { background: rgba(139,144,158,0.15); color: var(--muted); }
.badge.unknown .badge-dot { background: var(--muted); }
#conn-banner {
display: none;
background: var(--red);
color: #fff;
text-align: center;
padding: 8px;
font-size: 13px;
font-weight: 600;
}
#conn-banner.show { display: block; }
main {
padding: 24px;
display: grid;
grid-template-columns: repeat(auto-fit, minmax(340px, 1fr));
gap: 20px;
max-width: 1400px;
margin: 0 auto;
}
.card {
background: var(--panel);
border: 1px solid var(--border);
border-radius: 10px;
padding: 18px 20px;
}
.card h2 {
font-size: 14px;
text-transform: uppercase;
letter-spacing: 0.05em;
color: var(--muted);
margin: 0 0 14px 0;
display: flex;
align-items: center;
justify-content: space-between;
}
table { width: 100%; border-collapse: collapse; font-size: 13px; }
th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); }
th { color: var(--muted); font-weight: 500; font-size: 12px; }
tr.failed-row { background: rgba(242,84,91,0.08); }
tr.inactive-row { opacity: 0.5; }
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 6px; }
.dot.on { background: var(--green); }
.dot.off { background: var(--red); }
.stat-grid {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 10px;
margin-bottom: 14px;
}
.stat {
background: rgba(255,255,255,0.03);
border-radius: 8px;
padding: 10px 12px;
}
.stat .value { font-size: 20px; font-weight: 600; }
.stat .label { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: 0.03em; }
.stat.failed .value { color: var(--red); }
.state-pill {
display: inline-block;
padding: 3px 10px;
border-radius: 999px;
font-size: 12px;
font-weight: 600;
text-transform: uppercase;
}
.state-idle { background: rgba(139,144,158,0.15); color: var(--muted); }
.state-scanning, .state-running { background: rgba(79,142,247,0.15); color: var(--blue); }
.state-completed { background: rgba(62,207,142,0.15); color: var(--green); }
.state-failed { background: rgba(242,84,91,0.15); color: var(--red); }
.kv { display: flex; justify-content: space-between; padding: 4px 0; font-size: 13px; border-bottom: 1px solid var(--border); }
.kv:last-child { border-bottom: none; }
.kv .k { color: var(--muted); }
.error-list { margin: 10px 0 0 0; padding: 0; list-style: none; }
.error-list li {
font-size: 12px;
padding: 6px 8px;
margin-bottom: 4px;
background: rgba(242,84,91,0.08);
border-left: 3px solid var(--red);
border-radius: 4px;
color: var(--text);
}
.error-list .err-time { color: var(--muted); margin-right: 6px; }
.empty-note { color: var(--muted); font-size: 13px; font-style: italic; }
</style>
</head>
<body>
<div id="conn-banner">Lost connection to status endpoint - retrying...</div>
<header>
<h1>m3u8-downloader</h1>
<div class="header-meta">
<span id="overall-badge" class="badge unknown"><span class="badge-dot"></span>unknown</span>
<span id="uptime">uptime: -</span>
<span id="last-updated">last updated: -</span>
</div>
</header>
<main>
<section class="card">
<h2>Download</h2>
<table>
<thead>
<tr><th>Resolution</th><th>Active</th><th>OK</th><th>Failed</th><th>Last activity</th></tr>
</thead>
<tbody id="variant-rows"></tbody>
</table>
<div id="download-empty" class="empty-note" style="display:none;">No variants reporting yet.</div>
<ul id="download-errors" class="error-list"></ul>
</section>
<section class="card">
<h2>Transfer</h2>
<div class="kv"><span class="k">NAS connection</span><span><span id="nas-dot" class="dot off"></span><span id="nas-text">unknown</span></span></div>
<div class="stat-grid" style="margin-top:12px;">
<div class="stat"><div class="value" id="t-added">-</div><div class="label">Added</div></div>
<div class="stat"><div class="value" id="t-completed">-</div><div class="label">Completed</div></div>
<div class="stat failed"><div class="value" id="t-failed">-</div><div class="label">Failed</div></div>
<div class="stat"><div class="value" id="t-pending">-</div><div class="label">Pending</div></div>
<div class="stat"><div class="value" id="t-queue">-</div><div class="label">Queue size</div></div>
<div class="stat"><div class="value" id="t-cleanup">-</div><div class="label">Cleanup pending</div></div>
</div>
<div class="kv"><span class="k">Transferred</span><span id="t-bytes">-</span></div>
<ul id="transfer-errors" class="error-list"></ul>
</section>
<section class="card">
<h2>Processing</h2>
<div class="kv"><span class="k">State</span><span id="p-state" class="state-pill state-idle">idle</span></div>
<div class="kv"><span class="k">Event</span><span id="p-event">-</span></div>
<div class="kv"><span class="k">Segments found</span><span id="p-segments">-</span></div>
<div class="kv"><span class="k">Started</span><span id="p-started">-</span></div>
<div class="kv"><span class="k">Finished</span><span id="p-finished">-</span></div>
<ul id="processing-errors" class="error-list"></ul>
</section>
</main>
<script>
const POLL_INTERVAL_MS = 3000;
const connBanner = document.getElementById('conn-banner');
function fmtBytes(n) {
if (!n || n <= 0) return '0 B';
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
let i = 0;
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
return n.toFixed(i === 0 ? 0 : 1) + ' ' + units[i];
}
function fmtDuration(sec) {
sec = Math.floor(sec || 0);
const h = Math.floor(sec / 3600);
const m = Math.floor((sec % 3600) / 60);
const s = sec % 60;
const parts = [];
if (h) parts.push(h + 'h');
if (m || h) parts.push(m + 'm');
parts.push(s + 's');
return parts.join(' ');
}
function fmtTime(iso) {
if (!iso) return '-';
const d = new Date(iso);
if (isNaN(d.getTime()) || d.getFullYear() < 1971) return '-';
return d.toLocaleTimeString();
}
function renderErrors(listEl, errors) {
listEl.innerHTML = '';
(errors || []).slice().reverse().forEach(e => {
const li = document.createElement('li');
const t = document.createElement('span');
t.className = 'err-time';
t.textContent = fmtTime(e.time);
li.appendChild(t);
li.appendChild(document.createTextNode(e.message));
listEl.appendChild(li);
});
}
function render(data) {
const badge = document.getElementById('overall-badge');
badge.className = 'badge ' + (data.status === 'healthy' ? 'healthy' : 'degraded');
badge.innerHTML = '<span class="badge-dot"></span>' + data.status;
document.getElementById('uptime').textContent = 'uptime: ' + fmtDuration(data.uptime_seconds);
document.getElementById('last-updated').textContent = 'last updated: ' + new Date().toLocaleTimeString();
// Download
const variants = (data.download && data.download.variants) || [];
const rows = document.getElementById('variant-rows');
rows.innerHTML = '';
document.getElementById('download-empty').style.display = variants.length ? 'none' : 'block';
variants.forEach(v => {
const tr = document.createElement('tr');
if (v.segments_failed > 0) tr.classList.add('failed-row');
if (!v.active) tr.classList.add('inactive-row');
tr.innerHTML =
'<td>' + v.resolution + '</td>' +
'<td><span class="dot ' + (v.active ? 'on' : 'off') + '"></span>' + (v.active ? 'active' : 'stopped') + '</td>' +
'<td>' + v.segments_ok + '</td>' +
'<td>' + v.segments_failed + '</td>' +
'<td>' + fmtTime(v.last_activity) + '</td>';
rows.appendChild(tr);
});
renderErrors(document.getElementById('download-errors'), data.download && data.download.recent_errors);
// Transfer
const t = data.transfer || {};
const nasDot = document.getElementById('nas-dot');
nasDot.className = 'dot ' + (t.nas_connected ? 'on' : 'off');
document.getElementById('nas-text').textContent = t.nas_connected ? 'connected' : 'disconnected';
document.getElementById('t-added').textContent = t.added ?? '-';
document.getElementById('t-completed').textContent = t.completed ?? '-';
document.getElementById('t-failed').textContent = t.failed ?? '-';
document.getElementById('t-pending').textContent = t.pending ?? '-';
document.getElementById('t-queue').textContent = t.queue_size ?? '-';
document.getElementById('t-cleanup').textContent = t.cleanup_pending ?? '-';
document.getElementById('t-bytes').textContent = fmtBytes(t.bytes_transferred);
renderErrors(document.getElementById('transfer-errors'), t.recent_errors);
// Processing
const p = data.processing || {};
const stateEl = document.getElementById('p-state');
const state = p.state || 'idle';
stateEl.textContent = state;
stateEl.className = 'state-pill state-' + state;
document.getElementById('p-event').textContent = p.event_name || '-';
document.getElementById('p-segments').textContent = p.segments_found ?? '-';
document.getElementById('p-started').textContent = fmtTime(p.started_at);
document.getElementById('p-finished').textContent = fmtTime(p.finished_at);
renderErrors(document.getElementById('processing-errors'), p.recent_errors);
}
async function poll() {
try {
const resp = await fetch('/status', { cache: 'no-store' });
if (!resp.ok) throw new Error('status ' + resp.status);
const data = await resp.json();
connBanner.classList.remove('show');
render(data);
} catch (e) {
connBanner.classList.add('show');
}
}
poll();
setInterval(poll, POLL_INTERVAL_MS);
</script>
</body>
</html>

View File

@ -1,352 +0,0 @@
package health
import (
"sort"
"sync"
"time"
)
const maxRecentErrors = 10
type ErrorEntry struct {
Time time.Time `json:"time"`
Message string `json:"message"`
}
type VariantStatus struct {
Resolution string `json:"resolution"`
Bandwidth uint32 `json:"bandwidth"`
Active bool `json:"active"`
SegmentsOK int `json:"segments_ok"`
SegmentsFailed int `json:"segments_failed"`
LastActivity time.Time `json:"last_activity"`
LastError string `json:"last_error,omitempty"`
}
// StatsProvider is satisfied by *transfer.QueueStats without pkg/health
// needing to import pkg/transfer (which would create an import cycle, since
// pkg/transfer calls into pkg/health to report status).
type StatsProvider interface {
GetStats() (added, completed, failed, pending int, bytes int64)
}
type downloadState struct {
variants map[string]*VariantStatus
recentErrors []ErrorEntry
}
type transferState struct {
statsProvider StatsProvider
queueSizeFn func() int
cleanupPendingFn func() int
nasConnectedFn func() bool
nasConnected bool
recentErrors []ErrorEntry
}
type processingState struct {
state string // idle | scanning | running | completed | failed
eventName string
segmentsFound int
startedAt time.Time
finishedAt time.Time
lastError string
recentErrors []ErrorEntry
}
// Registry is the process-wide health/metrics store. All subsystems report
// into it via the Record* methods; the health HTTP server reads it via
// Snapshot(). Safe for concurrent use.
type Registry struct {
mu sync.RWMutex
startedAt time.Time
download downloadState
transfer transferState
processing processingState
}
var (
globalRegistry *Registry
registryOnce sync.Once
)
// MustGetRegistry returns the process-wide registry, mirroring the
// constants.MustGetConfig() / logging.MustGetFactory() singleton style.
func MustGetRegistry() *Registry {
registryOnce.Do(func() {
globalRegistry = New()
})
return globalRegistry
}
func New() *Registry {
return &Registry{
startedAt: time.Now(),
download: downloadState{
variants: make(map[string]*VariantStatus),
},
processing: processingState{
state: "idle",
},
}
}
func appendBounded(errs []ErrorEntry, e ErrorEntry) []ErrorEntry {
errs = append(errs, e)
if len(errs) > maxRecentErrors {
errs = errs[len(errs)-maxRecentErrors:]
}
return errs
}
func errString(err error) string {
if err == nil {
return ""
}
return err.Error()
}
// --- Download ---
func (r *Registry) RecordVariantStarted(resolution string, bandwidth uint32) {
r.mu.Lock()
defer r.mu.Unlock()
r.download.variants[resolution] = &VariantStatus{
Resolution: resolution,
Bandwidth: bandwidth,
Active: true,
LastActivity: time.Now(),
}
}
func (r *Registry) RecordVariantStopped(resolution string) {
r.mu.Lock()
defer r.mu.Unlock()
if v, ok := r.download.variants[resolution]; ok {
v.Active = false
}
}
func (r *Registry) RecordSegmentDownloaded(resolution string) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.SegmentsOK++
v.LastActivity = time.Now()
}
func (r *Registry) RecordSegmentFailed(resolution string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.SegmentsFailed++
v.LastActivity = time.Now()
v.LastError = errString(err)
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
Time: time.Now(),
Message: resolution + ": " + errString(err),
})
}
func (r *Registry) RecordPlaylistError(resolution string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.LastError = errString(err)
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
Time: time.Now(),
Message: resolution + " playlist: " + errString(err),
})
}
func (r *Registry) ensureVariantLocked(resolution string) *VariantStatus {
v, ok := r.download.variants[resolution]
if !ok {
v = &VariantStatus{Resolution: resolution, Active: true}
r.download.variants[resolution] = v
}
return v
}
// --- Transfer ---
// AttachTransfer wires the registry to the live counters/state already
// maintained by pkg/transfer instead of duplicating them. Called once from
// TransferService construction.
func (r *Registry) AttachTransfer(stats StatsProvider, queueSizeFn, cleanupPendingFn func() int, nasConnectedFn func() bool) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.statsProvider = stats
r.transfer.queueSizeFn = queueSizeFn
r.transfer.cleanupPendingFn = cleanupPendingFn
r.transfer.nasConnectedFn = nasConnectedFn
}
func (r *Registry) RecordNASConnectivity(connected bool) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.nasConnected = connected
}
// RecordTransferError captures failures QueueStats' cumulative counters
// don't individually retain (pre-queue failures, cleanup failures) so they
// still surface in the bounded recent-errors list.
func (r *Registry) RecordTransferError(sourcePath string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.recentErrors = appendBounded(r.transfer.recentErrors, ErrorEntry{
Time: time.Now(),
Message: sourcePath + ": " + errString(err),
})
}
// --- Processing ---
func (r *Registry) RecordProcessingStageChange(state, eventName string) {
r.mu.Lock()
defer r.mu.Unlock()
r.processing.state = state
if eventName != "" {
r.processing.eventName = eventName
}
switch state {
case "scanning":
r.processing.startedAt = time.Now()
r.processing.finishedAt = time.Time{}
r.processing.lastError = ""
case "completed", "failed":
r.processing.finishedAt = time.Now()
}
}
func (r *Registry) RecordProcessingSegmentsFound(count int) {
r.mu.Lock()
defer r.mu.Unlock()
r.processing.segmentsFound = count
}
func (r *Registry) RecordProcessingError(err error) {
r.mu.Lock()
defer r.mu.Unlock()
msg := errString(err)
r.processing.lastError = msg
r.processing.recentErrors = appendBounded(r.processing.recentErrors, ErrorEntry{
Time: time.Now(),
Message: msg,
})
}
// --- Snapshot ---
type DownloadSnapshot struct {
Variants []*VariantStatus `json:"variants"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type TransferSnapshot struct {
NASConnected bool `json:"nas_connected"`
QueueSize int `json:"queue_size"`
CleanupPending int `json:"cleanup_pending"`
Added int `json:"added"`
Completed int `json:"completed"`
Failed int `json:"failed"`
Pending int `json:"pending"`
BytesTransferred int64 `json:"bytes_transferred"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type ProcessingSnapshot struct {
State string `json:"state"`
EventName string `json:"event_name,omitempty"`
SegmentsFound int `json:"segments_found"`
StartedAt *time.Time `json:"started_at,omitempty"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
LastError string `json:"last_error,omitempty"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type Snapshot struct {
Status string `json:"status"`
UptimeSec float64 `json:"uptime_seconds"`
Download DownloadSnapshot `json:"download"`
Transfer TransferSnapshot `json:"transfer"`
Processing ProcessingSnapshot `json:"processing"`
}
func (r *Registry) Snapshot() Snapshot {
r.mu.RLock()
defer r.mu.RUnlock()
variants := make([]*VariantStatus, 0, len(r.download.variants))
for _, v := range r.download.variants {
vCopy := *v
variants = append(variants, &vCopy)
}
sort.Slice(variants, func(i, j int) bool { return variants[i].Resolution < variants[j].Resolution })
dl := DownloadSnapshot{
Variants: variants,
RecentErrors: append([]ErrorEntry(nil), r.download.recentErrors...),
}
tr := TransferSnapshot{
NASConnected: r.transfer.nasConnected,
RecentErrors: append([]ErrorEntry(nil), r.transfer.recentErrors...),
}
if r.transfer.statsProvider != nil {
added, completed, failed, pending, bytes := r.transfer.statsProvider.GetStats()
tr.Added, tr.Completed, tr.Failed, tr.Pending, tr.BytesTransferred = added, completed, failed, pending, bytes
}
if r.transfer.queueSizeFn != nil {
tr.QueueSize = r.transfer.queueSizeFn()
}
if r.transfer.cleanupPendingFn != nil {
tr.CleanupPending = r.transfer.cleanupPendingFn()
}
if r.transfer.nasConnectedFn != nil {
tr.NASConnected = r.transfer.nasConnectedFn()
}
proc := ProcessingSnapshot{
State: r.processing.state,
EventName: r.processing.eventName,
SegmentsFound: r.processing.segmentsFound,
LastError: r.processing.lastError,
RecentErrors: append([]ErrorEntry(nil), r.processing.recentErrors...),
}
if !r.processing.startedAt.IsZero() {
t := r.processing.startedAt
proc.StartedAt = &t
}
if !r.processing.finishedAt.IsZero() {
t := r.processing.finishedAt
proc.FinishedAt = &t
}
snap := Snapshot{
UptimeSec: time.Since(r.startedAt).Seconds(),
Download: dl,
Transfer: tr,
Processing: proc,
}
snap.Status = computeStatus(dl, tr, proc)
return snap
}
// computeStatus gives one glanceable field: "degraded" the moment anything
// worth investigating shows up, "healthy" otherwise. Transfer is only
// considered for degradation once it's actually in use (Attached + has seen
// activity), so a download-only run with NAS transfer disabled doesn't
// falsely report degraded.
func computeStatus(dl DownloadSnapshot, tr TransferSnapshot, proc ProcessingSnapshot) string {
if len(dl.RecentErrors) > 0 || len(tr.RecentErrors) > 0 || len(proc.RecentErrors) > 0 {
return "degraded"
}
if !tr.NASConnected && (tr.Added > 0 || tr.QueueSize > 0) {
return "degraded"
}
if proc.State == "failed" {
return "degraded"
}
return "healthy"
}

View File

@ -1,60 +0,0 @@
package health
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/logging"
"net"
"net/http"
)
//go:embed dashboard.html
var dashboardHTML []byte
// Start binds the health/status HTTP server and returns immediately once the
// listener is bound, so callers see bind failures (e.g. port already in use)
// synchronously rather than losing them in a background goroutine. The
// returned *http.Server should be shut down by the caller via Shutdown(ctx).
//
// Security note: cfg.Addr is expected to be a bind-all address (e.g. ":8090")
// so status can be checked from other devices on the local network during a
// live event. There is no authentication or TLS on these endpoints - this is
// only safe on a trusted local network, never exposed to the open internet.
func Start(ctx context.Context, cfg *config.HealthConfig, reg *Registry) (*http.Server, error) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Write(dashboardHTML)
})
mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(reg.Snapshot()); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
})
ln, err := net.Listen("tcp", cfg.Addr)
if err != nil {
return nil, fmt.Errorf("failed to bind health server to %s: %w", cfg.Addr, err)
}
srv := &http.Server{Handler: mux}
go func() {
if err := srv.Serve(ln); err != nil && err != http.ErrServerClosed {
logging.MustGetFactory().App.Error("health server stopped unexpectedly", logging.KeyError, err.Error())
}
}()
return srv, nil
}

View File

@ -5,50 +5,6 @@ import (
"fmt" "fmt"
) )
// HTTPError represents an HTTP error with status code and message
type HTTPError struct {
StatusCode int
Message string
}
// Error returns the string representation of the HTTP error
func (e *HTTPError) Error() string {
return fmt.Sprintf("HTTP %d: %s", e.StatusCode, e.Message)
}
// Is implements error comparison for errors.Is
func (e *HTTPError) Is(target error) bool {
var httpErr *HTTPError
if errors.As(target, &httpErr) {
return e.StatusCode == httpErr.StatusCode
}
return false
}
// NewHTTPError creates a new HTTP error
func NewHTTPError(statusCode int, message string) error {
return &HTTPError{
StatusCode: statusCode,
Message: message,
}
}
// IsHTTPError checks if an error is an HTTP error
func IsHTTPError(err error) bool {
var httpErr *HTTPError
return errors.As(err, &httpErr)
}
// GetHTTPStatusCode extracts the status code from an HTTP error
func GetHTTPStatusCode(err error) int {
var httpErr *HTTPError
if errors.As(err, &httpErr) {
return httpErr.StatusCode
}
return 0
}
// Legacy support for existing code
type HttpError struct { type HttpError struct {
Code int Code int
} }

View File

@ -1,318 +0,0 @@
package httpClient
import (
"fmt"
"net/http"
"strings"
"testing"
)
func TestHTTPError_Error(t *testing.T) {
tests := []struct {
name string
statusCode int
message string
want string
}{
{
name: "basic http error",
statusCode: 404,
message: "Not Found",
want: "HTTP 404: Not Found",
},
{
name: "server error",
statusCode: 500,
message: "Internal Server Error",
want: "HTTP 500: Internal Server Error",
},
{
name: "unauthorized error",
statusCode: 401,
message: "Unauthorized",
want: "HTTP 401: Unauthorized",
},
{
name: "empty message",
statusCode: 400,
message: "",
want: "HTTP 400: ",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := &HTTPError{
StatusCode: tt.statusCode,
Message: tt.message,
}
got := err.Error()
if got != tt.want {
t.Errorf("HTTPError.Error() = %q, want %q", got, tt.want)
}
})
}
}
func TestHTTPError_Is(t *testing.T) {
err404 := &HTTPError{StatusCode: 404, Message: "Not Found"}
err500 := &HTTPError{StatusCode: 500, Message: "Server Error"}
otherErr404 := &HTTPError{StatusCode: 404, Message: "Different message"}
regularError := fmt.Errorf("regular error")
tests := []struct {
name string
err error
target error
want bool
}{
{
name: "same error instance",
err: err404,
target: err404,
want: true,
},
{
name: "different HTTP errors with same status",
err: err404,
target: otherErr404,
want: true,
},
{
name: "different HTTP errors with different status",
err: err404,
target: err500,
want: false,
},
{
name: "HTTP error vs regular error",
err: err404,
target: regularError,
want: false,
},
{
name: "regular error vs HTTP error",
err: regularError,
target: err404,
want: false,
},
{
name: "nil target",
err: err404,
target: nil,
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var got bool
if httpErr, ok := tt.err.(*HTTPError); ok {
got = httpErr.Is(tt.target)
} else {
got = false // Non-HTTP errors return false
}
if got != tt.want {
t.Errorf("HTTPError.Is() = %v, want %v", got, tt.want)
}
})
}
}
func TestIsHTTPError(t *testing.T) {
httpErr := &HTTPError{StatusCode: 404, Message: "Not Found"}
regularErr := fmt.Errorf("regular error")
tests := []struct {
name string
err error
want bool
}{
{
name: "http error",
err: httpErr,
want: true,
},
{
name: "regular error",
err: regularErr,
want: false,
},
{
name: "nil error",
err: nil,
want: false,
},
{
name: "wrapped http error",
err: fmt.Errorf("wrapped: %w", httpErr),
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := IsHTTPError(tt.err)
if got != tt.want {
t.Errorf("IsHTTPError() = %v, want %v", got, tt.want)
}
})
}
}
func TestGetHTTPStatusCode(t *testing.T) {
tests := []struct {
name string
err error
want int
}{
{
name: "http error 404",
err: &HTTPError{StatusCode: 404, Message: "Not Found"},
want: 404,
},
{
name: "http error 500",
err: &HTTPError{StatusCode: 500, Message: "Server Error"},
want: 500,
},
{
name: "wrapped http error",
err: fmt.Errorf("wrapped: %w", &HTTPError{StatusCode: 403, Message: "Forbidden"}),
want: 403,
},
{
name: "regular error",
err: fmt.Errorf("regular error"),
want: 0,
},
{
name: "nil error",
err: nil,
want: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := GetHTTPStatusCode(tt.err)
if got != tt.want {
t.Errorf("GetHTTPStatusCode() = %v, want %v", got, tt.want)
}
})
}
}
func TestNewHTTPError(t *testing.T) {
statusCode := 404
message := "Page not found"
err := NewHTTPError(statusCode, message)
// Check type
httpErr, ok := err.(*HTTPError)
if !ok {
t.Fatalf("NewHTTPError should return *HTTPError, got %T", err)
}
// Check fields
if httpErr.StatusCode != statusCode {
t.Errorf("Expected StatusCode=%d, got %d", statusCode, httpErr.StatusCode)
}
if httpErr.Message != message {
t.Errorf("Expected Message=%q, got %q", message, httpErr.Message)
}
// Check error string
expectedErrorString := fmt.Sprintf("HTTP %d: %s", statusCode, message)
if httpErr.Error() != expectedErrorString {
t.Errorf("Expected error string=%q, got %q", expectedErrorString, httpErr.Error())
}
}
func TestHTTPError_StatusCodeChecks(t *testing.T) {
tests := []struct {
name string
statusCode int
isClient bool
isServer bool
}{
{"200 OK", 200, false, false},
{"400 Bad Request", 400, true, false},
{"401 Unauthorized", 401, true, false},
{"404 Not Found", 404, true, false},
{"499 Client Error", 499, true, false},
{"500 Server Error", 500, false, true},
{"502 Bad Gateway", 502, false, true},
{"599 Server Error", 599, false, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := &HTTPError{StatusCode: tt.statusCode, Message: "test"}
isClient := err.StatusCode >= 400 && err.StatusCode < 500
isServer := err.StatusCode >= 500 && err.StatusCode < 600
if isClient != tt.isClient {
t.Errorf("Status %d: expected isClient=%v, got %v", tt.statusCode, tt.isClient, isClient)
}
if isServer != tt.isServer {
t.Errorf("Status %d: expected isServer=%v, got %v", tt.statusCode, tt.isServer, isServer)
}
})
}
}
func TestHTTPError_Integration(t *testing.T) {
// Test that HTTPError integrates well with standard error handling
err := NewHTTPError(http.StatusNotFound, "Resource not found")
// Should be able to use with errors.Is
target := &HTTPError{StatusCode: http.StatusNotFound}
if !err.(*HTTPError).Is(target) {
t.Error("HTTPError should match target with same status code")
}
// Should be detectable as HTTPError
if !IsHTTPError(err) {
t.Error("Should be detectable as HTTPError")
}
// Should return correct status code
if GetHTTPStatusCode(err) != http.StatusNotFound {
t.Error("Should return correct status code")
}
// Should have meaningful string representation
errorString := err.Error()
if !strings.Contains(errorString, "404") {
t.Error("Error string should contain status code")
}
if !strings.Contains(errorString, "Resource not found") {
t.Error("Error string should contain message")
}
}
func TestHTTPError_EdgeCases(t *testing.T) {
// Test with zero status code
err := NewHTTPError(0, "Zero status")
if err.Error() != "HTTP 0: Zero status" {
t.Errorf("Unexpected error string for zero status: %s", err.Error())
}
// Test with very long message
longMessage := strings.Repeat("a", 1000)
err = NewHTTPError(500, longMessage)
if !strings.Contains(err.Error(), longMessage) {
t.Error("Long message should be preserved")
}
// Test status code boundaries
for _, code := range []int{399, 400, 499, 500, 599, 600} {
err := &HTTPError{StatusCode: code}
// Should not panic
_ = err.Error()
}
}

View File

@ -1,72 +0,0 @@
package logging
import (
"log/slog"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"os"
"path/filepath"
"strings"
"sync"
)
// Factory holds one structured logger per pipeline stage, each writing JSON
// lines to its own file under LoggingConfig.Dir. These are additive to the
// existing stdlib log.* calls, not a replacement for them.
type Factory struct {
App *slog.Logger
Download *slog.Logger
Transfer *slog.Logger
Processing *slog.Logger
}
var (
globalFactory *Factory
factoryOnce sync.Once
)
// MustGetFactory returns the process-wide logging factory, building it from
// the config singleton on first use. Mirrors constants.MustGetConfig().
func MustGetFactory() *Factory {
factoryOnce.Do(func() {
globalFactory = New(&constants.MustGetConfig().Logging)
})
return globalFactory
}
func New(cfg *config.LoggingConfig) *Factory {
level := parseLevel(cfg.Level)
return &Factory{
App: newSubsystemLogger(cfg.Dir, "app.log", "app", level),
Download: newSubsystemLogger(cfg.Dir, "download.log", "download", level),
Transfer: newSubsystemLogger(cfg.Dir, "transfer.log", "transfer", level),
Processing: newSubsystemLogger(cfg.Dir, "processing.log", "processing", level),
}
}
// newSubsystemLogger opens (or creates) the subsystem's log file in append
// mode. A logging misconfiguration must never take down an active recording,
// so failure to open the file falls back to stderr instead of panicking.
func newSubsystemLogger(dir, filename, component string, level slog.Level) *slog.Logger {
path := filepath.Join(dir, filename)
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
fallback := slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: level})).With("component", component)
fallback.Error("failed to open log file, falling back to stderr", KeyFile, path, KeyError, err.Error())
return fallback
}
return slog.New(slog.NewJSONHandler(f, &slog.HandlerOptions{Level: level})).With("component", component)
}
func parseLevel(level string) slog.Level {
switch strings.ToLower(strings.TrimSpace(level)) {
case "debug":
return slog.LevelDebug
case "warn", "warning":
return slog.LevelWarn
case "error":
return slog.LevelError
default:
return slog.LevelInfo
}
}

View File

@ -1,13 +0,0 @@
package logging
const (
KeyResolution = "resolution"
KeyBandwidth = "bandwidth"
KeySegment = "segment"
KeySeq = "seq"
KeyFile = "file"
KeyBytes = "bytes"
KeyEvent = "event"
KeyError = "error"
KeyAttempt = "attempt"
)

View File

@ -4,26 +4,14 @@ import (
"encoding/json" "encoding/json"
"log" "log"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/utils"
"os" "os"
"sort" "sort"
"strconv"
"sync"
) )
type ManifestWriter struct { type ManifestWriter struct {
ManifestPath string ManifestPath string
Segments []ManifestItem Segments []ManifestItem
// Index maps a sequence number to its position in Segments. It holds Index map[string]*ManifestItem
// positions rather than pointers because append reallocates the backing
// array, which would leave previously handed-out element pointers aimed at
// a stale copy.
Index map[string]int
// mu guards the fields above: every variant downloads concurrently, and
// each reports its segments as they land.
mu sync.Mutex
} }
type ManifestItem struct { type ManifestItem struct {
@ -32,75 +20,51 @@ type ManifestItem struct {
} }
func NewManifestWriter(eventName string) *ManifestWriter { func NewManifestWriter(eventName string) *ManifestWriter {
cfg := constants.MustGetConfig()
return &ManifestWriter{ return &ManifestWriter{
ManifestPath: cfg.GetManifestPath(eventName), ManifestPath: constants.ManifestPath + "/" + eventName + ".json",
Segments: make([]ManifestItem, 0), Segments: make([]ManifestItem, 0),
Index: make(map[string]int), Index: make(map[string]*ManifestItem),
} }
} }
// AddOrUpdateSegment records that a segment was downloaded, keeping the
// highest-resolution variant seen for each sequence number.
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) { func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
m.mu.Lock()
defer m.mu.Unlock()
if m.Index == nil { if m.Index == nil {
m.Index = make(map[string]int) m.Index = make(map[string]*ManifestItem)
} }
if m.Segments == nil { if m.Segments == nil {
m.Segments = make([]ManifestItem, 0) m.Segments = make([]ManifestItem, 0)
} }
if pos, ok := m.Index[seqNo]; ok { if existing, ok := m.Index[seqNo]; ok {
if ResolutionHeight(resolution) > ResolutionHeight(m.Segments[pos].Resolution) { if resolution > existing.Resolution {
m.Segments[pos].Resolution = resolution existing.Resolution = resolution
} }
return return
} else {
item := ManifestItem{
SeqNo: seqNo,
Resolution: resolution,
}
m.Segments = append(m.Segments, item)
m.Index[seqNo] = &item
} }
m.Segments = append(m.Segments, ManifestItem{SeqNo: seqNo, Resolution: resolution})
m.Index[seqNo] = len(m.Segments) - 1
}
// Len reports how many distinct sequence numbers have been recorded.
func (m *ManifestWriter) Len() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.Segments)
} }
func (m *ManifestWriter) WriteManifest() { func (m *ManifestWriter) WriteManifest() {
// Sort a copy: reordering Segments in place would invalidate Index, and a sort.Slice(m.Segments, func(i, j int) bool {
// recording may still be writing into it. return m.Segments[i].SeqNo < m.Segments[j].SeqNo
m.mu.Lock()
segments := make([]ManifestItem, len(m.Segments))
copy(segments, m.Segments)
m.mu.Unlock()
sort.Slice(segments, func(i, j int) bool {
return lessSeqNo(segments[i].SeqNo, segments[j].SeqNo)
}) })
data, err := json.MarshalIndent(segments, "", " ") data, err := json.MarshalIndent(m.Segments, "", " ")
if err != nil { if err != nil {
log.Printf("Failed to marshal manifest: %v", err) log.Printf("Failed to marshal manifest: %v", err)
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
return
}
if err := utils.ValidateWritablePath(m.ManifestPath); err != nil {
log.Printf("Manifest path validation failed: %v", err)
logging.MustGetFactory().Download.Error("manifest path validation failed", logging.KeyError, err.Error())
return return
} }
file, err := os.Create(m.ManifestPath) file, err := os.Create(m.ManifestPath)
if err != nil { if err != nil {
log.Printf("Failed to create manifest file: %v", err) log.Printf("Failed to create manifest file: %v", err)
logging.MustGetFactory().Download.Error("failed to create manifest file", logging.KeyError, err.Error())
return return
} }
@ -109,19 +73,6 @@ func (m *ManifestWriter) WriteManifest() {
_, err = file.Write(data) _, err = file.Write(data)
if err != nil { if err != nil {
log.Printf("Failed to write manifest file: %v", err) log.Printf("Failed to write manifest file: %v", err)
logging.MustGetFactory().Download.Error("failed to write manifest file", logging.KeyError, err.Error())
return return
} }
} }
// lessSeqNo orders sequence numbers numerically, so segment 9 precedes segment
// 10 instead of sorting after it as the strings would. Values that are not
// numeric fall back to lexicographic order.
func lessSeqNo(a, b string) bool {
ai, aErr := strconv.ParseUint(a, 10, 64)
bi, bErr := strconv.ParseUint(b, 10, 64)
if aErr == nil && bErr == nil {
return ai < bi
}
return a < b
}

View File

@ -1,409 +0,0 @@
package media
import (
"encoding/json"
"os"
"path/filepath"
"strconv"
"sync"
"testing"
)
func TestManifestWriter_NewManifestWriter(t *testing.T) {
// Set up temporary environment for testing
tempDir, err := os.MkdirTemp("", "manifest_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Set environment variable to use temp directory
os.Setenv("LOCAL_OUTPUT_DIR", tempDir)
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
eventName := "test-event"
writer := NewManifestWriter(eventName)
if writer == nil {
t.Fatal("NewManifestWriter() returned nil")
}
if writer.Segments == nil {
t.Error("Segments should be initialized")
}
if writer.Index == nil {
t.Error("Index should be initialized")
}
if len(writer.Segments) != 0 {
t.Errorf("Segments should be empty, got %d items", len(writer.Segments))
}
}
func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "test.json",
Segments: make([]ManifestItem, 0),
Index: make(map[string]int),
}
// Test adding new segment
writer.AddOrUpdateSegment("1001", "1080p")
if len(writer.Segments) != 1 {
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
}
if writer.Segments[0].SeqNo != "1001" {
t.Errorf("Expected SeqNo '1001', got '%s'", writer.Segments[0].SeqNo)
}
if writer.Segments[0].Resolution != "1080p" {
t.Errorf("Expected Resolution '1080p', got '%s'", writer.Segments[0].Resolution)
}
// Test updating existing segment with higher resolution
writer.AddOrUpdateSegment("1001", "1440p")
if len(writer.Segments) != 1 {
t.Errorf("Segments count should remain 1 after update, got %d", len(writer.Segments))
}
if writer.Segments[0].Resolution != "1440p" {
t.Errorf("Expected updated resolution '1440p', got '%s'", writer.Segments[0].Resolution)
}
// Test updating existing segment with lower resolution (should not change)
writer.AddOrUpdateSegment("1001", "720p")
if writer.Segments[0].Resolution != "1440p" {
t.Errorf("Resolution should remain '1440p', got '%s'", writer.Segments[0].Resolution)
}
// Test adding different segment
writer.AddOrUpdateSegment("1002", "720p")
if len(writer.Segments) != 2 {
t.Errorf("Expected 2 segments, got %d", len(writer.Segments))
}
}
func TestManifestWriter_AddOrUpdateSegment_NilFields(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "test.json",
}
// Test with nil fields (should initialize them)
writer.AddOrUpdateSegment("1001", "1080p")
if writer.Segments == nil {
t.Error("Segments should be initialized")
}
if writer.Index == nil {
t.Error("Index should be initialized")
}
if len(writer.Segments) != 1 {
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
}
}
func TestManifestWriter_WriteManifest(t *testing.T) {
tempDir, err := os.MkdirTemp("", "manifest_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
manifestPath := filepath.Join(tempDir, "test-manifest.json")
writer := &ManifestWriter{
ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0),
Index: make(map[string]int),
}
// Add some test segments out of order
writer.AddOrUpdateSegment("1003", "1080p")
writer.AddOrUpdateSegment("1001", "720p")
writer.AddOrUpdateSegment("1002", "1080p")
// Write manifest
writer.WriteManifest()
// Verify file was created
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
t.Fatalf("Manifest file was not created: %s", manifestPath)
}
// Read and verify content
content, err := os.ReadFile(manifestPath)
if err != nil {
t.Fatalf("Failed to read manifest file: %v", err)
}
var segments []ManifestItem
err = json.Unmarshal(content, &segments)
if err != nil {
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
}
// Verify segments are sorted by sequence number
if len(segments) != 3 {
t.Errorf("Expected 3 segments in manifest, got %d", len(segments))
}
expectedOrder := []string{"1001", "1002", "1003"}
for i, segment := range segments {
if segment.SeqNo != expectedOrder[i] {
t.Errorf("Segment %d: expected SeqNo '%s', got '%s'", i, expectedOrder[i], segment.SeqNo)
}
}
// Verify content structure
if segments[0].Resolution != "720p" {
t.Errorf("Expected first segment resolution '720p', got '%s'", segments[0].Resolution)
}
if segments[1].Resolution != "1080p" {
t.Errorf("Expected second segment resolution '1080p', got '%s'", segments[1].Resolution)
}
}
func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
tempDir, err := os.MkdirTemp("", "manifest_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
manifestPath := filepath.Join(tempDir, "empty-manifest.json")
writer := &ManifestWriter{
ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0),
Index: make(map[string]int),
}
// Write empty manifest
writer.WriteManifest()
// Verify file was created
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
t.Fatalf("Manifest file was not created: %s", manifestPath)
}
// Read and verify content is empty array
content, err := os.ReadFile(manifestPath)
if err != nil {
t.Fatalf("Failed to read manifest file: %v", err)
}
var segments []ManifestItem
err = json.Unmarshal(content, &segments)
if err != nil {
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
}
if len(segments) != 0 {
t.Errorf("Expected empty segments array, got %d items", len(segments))
}
}
func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
Index: make(map[string]int),
}
// This should not panic, just fail gracefully
writer.WriteManifest()
// Test passes if no panic occurs
}
func TestManifestItem_JSONSerialization(t *testing.T) {
item := ManifestItem{
SeqNo: "1001",
Resolution: "1080p",
}
// Test marshaling
data, err := json.Marshal(item)
if err != nil {
t.Fatalf("Failed to marshal ManifestItem: %v", err)
}
// Test unmarshaling
var unmarshaled ManifestItem
err = json.Unmarshal(data, &unmarshaled)
if err != nil {
t.Fatalf("Failed to unmarshal ManifestItem: %v", err)
}
if unmarshaled.SeqNo != item.SeqNo {
t.Errorf("SeqNo mismatch: expected '%s', got '%s'", item.SeqNo, unmarshaled.SeqNo)
}
if unmarshaled.Resolution != item.Resolution {
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
}
}
func TestResolutionHeight(t *testing.T) {
cases := map[string]int{
"1080p": 1080,
"720p": 720,
"2160p": 2160,
"1440p": 1440,
"240p": 240,
// Not a <height>p label: must rank below every real resolution rather
// than accidentally outranking them.
"unknown": 0,
"": 0,
"p": 0,
}
for label, want := range cases {
if got := ResolutionHeight(label); got != want {
t.Errorf("ResolutionHeight(%q) = %d, want %d", label, got, want)
}
}
}
// TestAddOrUpdateSegmentKeepsHighestResolution pins the comparison used to
// choose between variants. Comparing the labels as strings — as this once did —
// ranks "720p" above "1080p", quietly downgrading the manifest.
func TestAddOrUpdateSegmentKeepsHighestResolution(t *testing.T) {
cases := []struct {
name string
arriving []string
want string
}{
{"higher replaces lower", []string{"720p", "1080p"}, "1080p"},
{"lower does not replace higher", []string{"1080p", "720p"}, "1080p"},
{"fewer digits must not win", []string{"1080p", "480p"}, "1080p"},
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
{"unknown loses to a real resolution", []string{"unknown", "480p"}, "480p"},
{"unknown does not displace a real resolution", []string{"1080p", "unknown"}, "1080p"},
{"repeated identical resolution", []string{"720p", "720p"}, "720p"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
w := &ManifestWriter{}
for _, res := range tc.arriving {
w.AddOrUpdateSegment("1001", res)
}
if len(w.Segments) != 1 {
t.Fatalf("expected a single entry, got %d", len(w.Segments))
}
if got := w.Segments[0].Resolution; got != tc.want {
t.Errorf("kept %q, want %q (arrival order %v)", got, tc.want, tc.arriving)
}
})
}
}
// TestAddOrUpdateSegmentUpdatesTheStoredEntry guards the aliasing bug: the
// index once held a pointer to a local copy rather than to the stored element,
// so updates were applied to an orphan and never reached the written manifest.
func TestAddOrUpdateSegmentUpdatesTheStoredEntry(t *testing.T) {
w := &ManifestWriter{}
// Enough entries to force at least one append reallocation, which would
// invalidate any pointer held into the backing array.
for i := 0; i < 64; i++ {
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "480p")
}
for i := 0; i < 64; i++ {
w.AddOrUpdateSegment(strconv.Itoa(1000+i), "1080p")
}
if len(w.Segments) != 64 {
t.Fatalf("expected 64 entries, got %d", len(w.Segments))
}
for _, seg := range w.Segments {
if seg.Resolution != "1080p" {
t.Fatalf("segment %s kept %q; the upgrade never reached the stored entry", seg.SeqNo, seg.Resolution)
}
}
}
// TestAddOrUpdateSegmentIsConcurrencySafe reflects real use: every variant
// downloads in its own goroutine and reports segments as they land.
func TestAddOrUpdateSegmentIsConcurrencySafe(t *testing.T) {
w := &ManifestWriter{}
resolutions := []string{"240p", "480p", "720p", "1080p"}
var wg sync.WaitGroup
for _, res := range resolutions {
wg.Add(1)
go func(res string) {
defer wg.Done()
for seq := 0; seq < 200; seq++ {
w.AddOrUpdateSegment(strconv.Itoa(seq), res)
}
}(res)
}
wg.Wait()
if got := w.Len(); got != 200 {
t.Errorf("expected 200 distinct sequence numbers, got %d", got)
}
for _, seg := range w.Segments {
if seg.Resolution != "1080p" {
t.Errorf("segment %s = %q, want the highest variant 1080p", seg.SeqNo, seg.Resolution)
}
}
}
// TestWriteManifestSortsNumerically checks that segment 9 precedes segment 10,
// which string ordering would reverse.
func TestWriteManifestSortsNumerically(t *testing.T) {
dir := t.TempDir()
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
for _, seq := range []string{"100", "9", "1000", "10", "1"} {
w.AddOrUpdateSegment(seq, "1080p")
}
w.WriteManifest()
data, err := os.ReadFile(w.ManifestPath)
if err != nil {
t.Fatalf("failed to read manifest: %v", err)
}
var items []ManifestItem
if err := json.Unmarshal(data, &items); err != nil {
t.Fatalf("manifest is not valid JSON: %v", err)
}
want := []string{"1", "9", "10", "100", "1000"}
if len(items) != len(want) {
t.Fatalf("expected %d entries, got %d", len(want), len(items))
}
for i, seq := range want {
if items[i].SeqNo != seq {
t.Errorf("position %d = %q, want %q (full order: %v)", i, items[i].SeqNo, seq, items)
}
}
}
// TestWriteManifestLeavesIndexUsable ensures writing mid-recording does not
// corrupt the writer: sorting in place would desynchronise Index from Segments.
func TestWriteManifestLeavesIndexUsable(t *testing.T) {
dir := t.TempDir()
w := &ManifestWriter{ManifestPath: filepath.Join(dir, "event.json")}
w.AddOrUpdateSegment("20", "480p")
w.AddOrUpdateSegment("10", "480p")
w.WriteManifest()
// After the write, an upgrade must still land on the right entry.
w.AddOrUpdateSegment("20", "1080p")
pos, ok := w.Index["20"]
if !ok {
t.Fatal("sequence 20 missing from the index after WriteManifest")
}
if got := w.Segments[pos].Resolution; got != "1080p" {
t.Errorf("sequence 20 = %q, want 1080p; the index no longer matches the slice", got)
}
if w.Segments[w.Index["10"]].SeqNo != "10" {
t.Error("index and slice disagree after WriteManifest")
}
}

View File

@ -6,7 +6,6 @@ import (
"io" "io"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/httpClient" "m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@ -34,7 +33,6 @@ func (j SegmentJob) Key() string {
func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error { func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error {
for attempt := 0; attempt < 2; attempt++ { for attempt := 0; attempt < 2; attempt++ {
if attempt > 0 { if attempt > 0 {
logging.MustGetFactory().Download.Debug("retrying segment download", logging.KeyAttempt, attempt, "url", segmentURL)
time.Sleep(300 * time.Millisecond) time.Sleep(300 * time.Millisecond)
} }
req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil) req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil)

View File

@ -7,13 +7,10 @@ import (
"github.com/grafov/m3u8" "github.com/grafov/m3u8"
"log" "log"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/httpClient" "m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"net/http" "net/http"
"net/url" "net/url"
"path" "path"
"strconv"
"strings" "strings"
"time" "time"
) )
@ -25,32 +22,10 @@ type StreamVariant struct {
ID int ID int
Resolution string Resolution string
OutputDir string OutputDir string
Writer *ManifestWriter
} }
// ResolutionHeight converts a label produced by ExtractResolution back into func extractResolution(variant *m3u8.Variant) string {
// the pixel height it stands for, so variants can be ranked by quality.
//
// The height is parsed rather than looked up in a table: labels come from the
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
// 1440p or 2160p that no fixed table would anticipate. Anything not in
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
// therefore ranks below every recognised resolution.
//
// Comparing the labels as strings instead is a trap worth naming: "720p" sorts
// after "1080p", so lexicographic comparison silently prefers lower quality.
func ResolutionHeight(resolution string) int {
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
height, err := strconv.Atoi(trimmed)
if err != nil || height < 0 {
return 0
}
return height
}
// ExtractResolution maps a variant to the label used for its output directory,
// falling back to a bandwidth heuristic when the playlist omits RESOLUTION.
func ExtractResolution(variant *m3u8.Variant) string {
if variant.Resolution != "" { if variant.Resolution != "" {
parts := strings.Split(variant.Resolution, "x") parts := strings.Split(variant.Resolution, "x")
if len(parts) == 2 { if len(parts) == 2 {
@ -71,24 +46,19 @@ func ExtractResolution(variant *m3u8.Variant) string {
} }
} }
// GetAllVariants fetches the master playlist and returns one entry per stream func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) {
// variant. Segments are recorded into the manifest by VariantDownloader, which
// receives it directly, so no writer is threaded through here.
func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error) {
client := &http.Client{} client := &http.Client{}
req, _ := http.NewRequest("GET", masterURL, nil) req, _ := http.NewRequest("GET", masterURL, nil)
req.Header.Set("User-Agent", constants.HTTPUserAgent) req.Header.Set("User-Agent", constants.HTTPUserAgent)
req.Header.Set("Referer", constants.REFERRER) req.Header.Set("Referer", constants.REFERRER)
resp, err := client.Do(req) resp, err := client.Do(req)
if err != nil { if err != nil {
logging.MustGetFactory().Download.Error("failed to fetch master playlist", logging.KeyError, err.Error())
return nil, err return nil, err
} }
defer resp.Body.Close() defer resp.Body.Close()
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true) playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
if err != nil { if err != nil {
logging.MustGetFactory().Download.Error("failed to decode master playlist", logging.KeyError, err.Error())
return nil, err return nil, err
} }
@ -102,6 +72,7 @@ func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error
ID: 0, ID: 0,
Resolution: "unknown", Resolution: "unknown",
OutputDir: path.Join(outputDir, "unknown"), OutputDir: path.Join(outputDir, "unknown"),
Writer: writer,
}}, nil }}, nil
} }
@ -114,7 +85,7 @@ func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error
for i, v := range master.Variants { for i, v := range master.Variants {
vURL, _ := url.Parse(v.URI) vURL, _ := url.Parse(v.URI)
fullURL := base.ResolveReference(vURL).String() fullURL := base.ResolveReference(vURL).String()
resolution := ExtractResolution(v) resolution := extractResolution(v)
outputDir := path.Join(outputDir, resolution) outputDir := path.Join(outputDir, resolution)
variants = append(variants, &StreamVariant{ variants = append(variants, &StreamVariant{
URL: fullURL, URL: fullURL,
@ -125,17 +96,11 @@ func GetAllVariants(masterURL string, outputDir string) ([]*StreamVariant, error
OutputDir: outputDir, OutputDir: outputDir,
}) })
} }
logging.MustGetFactory().Download.Info("discovered variants", "count", len(variants))
return variants, nil return variants, nil
} }
func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}, manifest *ManifestWriter) { func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}, manifest *ManifestWriter) {
log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth) log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth)
dlLog := logging.MustGetFactory().Download.With(logging.KeyResolution, variant.Resolution)
registry := health.MustGetRegistry()
dlLog.Info("variant downloader started", logging.KeyBandwidth, variant.Bandwidth)
registry.RecordVariantStarted(variant.Resolution, variant.Bandwidth)
ticker := time.NewTicker(constants.RefreshDelay) ticker := time.NewTicker(constants.RefreshDelay)
defer ticker.Stop() defer ticker.Stop()
client := &http.Client{} client := &http.Client{}
@ -144,20 +109,16 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
registry.RecordVariantStopped(variant.Resolution)
return return
default: default:
} }
var seq uint64
playlist, err := LoadMediaPlaylist(variant.URL) playlist, err := LoadMediaPlaylist(variant.URL)
seq := playlist.SeqNo
if err != nil { if err != nil {
log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err) log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err)
dlLog.Warn("failed to load media playlist", logging.KeyError, err.Error())
registry.RecordPlaylistError(variant.Resolution, err)
goto waitTick goto waitTick
} }
seq = playlist.SeqNo
for _, seg := range playlist.Segments { for _, seg := range playlist.Segments {
if seg == nil { if seg == nil {
@ -187,11 +148,6 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
if err == nil { if err == nil {
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name) log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
dlLog.Debug("segment downloaded", logging.KeySegment, name)
registry.RecordSegmentDownloaded(variant.Resolution)
if manifest != nil {
manifest.AddOrUpdateSegment(strconv.FormatUint(j.Seq, 10), variant.Resolution)
}
return return
} }
@ -202,27 +158,21 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
if httpClient.IsHTTPStatus(err, 403) { if httpClient.IsHTTPStatus(err, 403) {
log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name) log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name)
dlLog.Warn("segment download failed (403)", logging.KeySegment, name)
} else { } else {
log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err) log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err)
dlLog.Error("segment download failed", logging.KeySegment, name, logging.KeyError, err.Error())
} }
registry.RecordSegmentFailed(variant.Resolution, err)
}(job) }(job)
seq++ seq++
} }
if playlist.Closed { if playlist.Closed {
log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution) log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution)
dlLog.Info("playlist closed")
registry.RecordVariantStopped(variant.Resolution)
return return
} }
waitTick: waitTick:
select { select {
case <-ctx.Done(): case <-ctx.Done():
registry.RecordVariantStopped(variant.Resolution)
return return
case <-ticker.C: case <-ticker.C:
} }

View File

@ -3,18 +3,10 @@ package nas
import "time" import "time"
type NASConfig struct { type NASConfig struct {
// Path is the destination as configured. It may be a UNC path, an
// smb:// URL, or a directory on this host; see ParseLocation.
Path string Path string
Username string Username string
Password string Password string
Timeout time.Duration Timeout time.Duration
RetryLimit int RetryLimit int
VerifySize bool VerifySize bool
// AutoMount allows the recorder to mount an SMB share itself when the
// host has not already mounted it. Linux only, and requires root, so it
// is off by default in favour of a clear "mount it first" error.
AutoMount bool
// MountPoint overrides where AutoMount attaches the share.
MountPoint string
} }

View File

@ -1,98 +0,0 @@
package nas
import (
"context"
"fmt"
"os/exec"
"runtime"
)
// Connector attaches a configured Location to the host filesystem so that
// ordinary file operations reach the NAS.
//
// Windows and Linux disagree about what "connecting to a share" means —
// Windows authenticates a UNC path in place, Linux needs the share mounted
// somewhere in its single filesystem tree — so each host gets its own
// implementation and NASService stays platform-neutral. Selection happens at
// runtime rather than through build tags so that every implementation is
// compiled, vetted and unit-testable on every platform; only the final exec
// call is genuinely host-specific.
type Connector interface {
// Name identifies the strategy in logs and check output.
Name() string
// Attach makes loc reachable and reports where to write.
Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error)
// Detach undoes whatever Attach set up. Implementations must undo only
// what they created: a share this process found already mounted belongs
// to the host, and unmounting it could break unrelated software.
Detach(ctx context.Context, att Attachment) error
}
// Attachment is the result of a successful Attach.
type Attachment struct {
// BasePath is the host-native directory that maps to the configured
// destination. Every file operation is resolved against it.
BasePath string
// Notes explain how the host reached the destination, for -check output
// and startup logs.
Notes []string
// Owned records that this process created the connection or mount, and
// so may tear it down on shutdown.
Owned bool
// mountPoint is the directory this process mounted, when Owned is set by
// the Linux connector.
mountPoint string
// share is the network path this process connected, when Owned is set by
// the Windows connector.
share string
}
// commandRunner executes an external command. Implementations keep argument
// construction in a pure function and call through this, so tests can assert
// exactly what would run without running it.
type commandRunner func(ctx context.Context, name string, args ...string) ([]byte, error)
// execRunner is the production runner: it combines stdout and stderr because
// mount and net both report the useful diagnosis on one or the other.
func execRunner(ctx context.Context, name string, args ...string) ([]byte, error) {
return exec.CommandContext(ctx, name, args...).CombinedOutput()
}
// connectorFor picks the strategy for a destination on a given OS. A
// destination that is already an ordinary directory needs no network step on
// any platform, so it short-circuits before the OS switch.
func connectorFor(loc Location, goos string) Connector {
if !loc.IsRemote() {
return &localConnector{goos: goos}
}
switch goos {
case "windows":
return &windowsConnector{run: execRunner}
case "linux":
return &linuxConnector{run: execRunner, mountsPath: procMountsPath}
default:
return &unsupportedConnector{goos: goos}
}
}
// defaultConnector selects the strategy for the running host.
func defaultConnector(loc Location) Connector {
return connectorFor(loc, runtime.GOOS)
}
// unsupportedConnector fails clearly on platforms this project does not target,
// rather than silently writing a share-shaped directory to local disk.
type unsupportedConnector struct {
goos string
}
func (c *unsupportedConnector) Name() string { return "unsupported" }
func (c *unsupportedConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
return Attachment{}, fmt.Errorf(
"connecting to network share %s is not supported on %s; mount the share and set NAS_OUTPUT_PATH to the mount point",
loc.SMBSource(), c.goos)
}
func (c *unsupportedConnector) Detach(context.Context, Attachment) error { return nil }

View File

@ -1,57 +0,0 @@
package nas
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// localConnector handles destinations that are already part of the host
// filesystem: a plain output directory, or — the normal arrangement on Linux —
// an SMB/NFS share mounted by fstab, autofs or systemd before the recorder
// starts. There is nothing to connect, but there is something to check.
type localConnector struct {
goos string
}
func (c *localConnector) Name() string { return "local" }
func (c *localConnector) Attach(_ context.Context, loc Location, _ NASConfig) (Attachment, error) {
if err := c.validate(loc.Raw); err != nil {
return Attachment{}, err
}
return Attachment{
BasePath: filepath.Clean(loc.Raw),
Notes: []string{"local or already-mounted path (no network authentication)"},
}, nil
}
func (c *localConnector) Detach(context.Context, Attachment) error { return nil }
// validate rejects a Windows-shaped path on a non-Windows host. This looks
// pedantic and is not: backslashes and colons are legal characters in Linux
// filenames, so a stray C:\data or \\server\share would be created verbatim as
// a local directory. Transfers would report success, cleanup would delete the
// local originals, and nothing would ever reach the NAS.
func (c *localConnector) validate(path string) error {
if c.goos == "windows" {
return nil
}
if driveLetter.MatchString(path) {
return fmt.Errorf(
"%q is a Windows drive path but this host is %s; set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams",
path, c.goos)
}
if strings.Contains(path, `\`) {
return fmt.Errorf(
"%q contains backslashes but this host is %s, where they are ordinary filename characters rather than separators; "+
"set NAS_OUTPUT_PATH to a mount point such as /mnt/nas/streams, or to \\\\server\\share\\folder to have the share resolved automatically",
path, c.goos)
}
return nil
}

View File

@ -1,178 +0,0 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
)
// linuxConnector reaches a share the way Linux does: through a mount point in
// the single filesystem tree. There is no per-path authentication, so the
// question is not "can we connect" but "where is this share mounted".
//
// The default and by far the safest answer is that the host already mounted it
// (fstab, autofs, systemd, or `docker run -v`), so the connector looks the
// share up in the mount table. Mounting it here is opt-in because it requires
// root and changes system state that outlives a failed run.
type linuxConnector struct {
run commandRunner
mountsPath string
// geteuid is indirected for tests; production uses os.Geteuid.
geteuid func() int
}
func (c *linuxConnector) Name() string { return "mount" }
func (c *linuxConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
entries, err := readMountTable(c.mountsPath)
if err != nil {
return Attachment{}, err
}
if point, rest, ok := findMount(entries, loc); ok {
return Attachment{
BasePath: filepath.Join(append([]string{point}, rest...)...),
Notes: []string{
"share: " + loc.SMBSource(),
fmt.Sprintf("resolved via existing mount at %s", point),
},
}, nil
}
if !cfg.AutoMount {
return Attachment{}, fmt.Errorf("%s", notMountedMessage(loc, cfg))
}
return c.mount(ctx, loc, cfg)
}
func (c *linuxConnector) Detach(ctx context.Context, att Attachment) error {
// Never unmount a share the host had already mounted; other software may
// be using it, and we did not put it there.
if !att.Owned || att.mountPoint == "" {
return nil
}
if out, err := c.run(ctx, "umount", att.mountPoint); err != nil {
return fmt.Errorf("failed to unmount %s: %v", att.mountPoint, summarize(err, out))
}
return nil
}
// mount performs an opt-in mount.cifs of the share.
func (c *linuxConnector) mount(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
geteuid := c.geteuid
if geteuid == nil {
geteuid = os.Geteuid
}
if geteuid() != 0 {
return Attachment{}, fmt.Errorf(
"NAS_AUTO_MOUNT is set but mounting %s requires root; run as root, or mount the share beforehand and %s",
loc.SMBSource(), suggestMountPointConfig(loc))
}
point := autoMountPoint(loc, cfg)
if err := os.MkdirAll(point, 0755); err != nil {
return Attachment{}, fmt.Errorf("failed to create mount point %s: %w", point, err)
}
credentialsPath, cleanup, err := writeCredentialsFile(cfg)
if err != nil {
return Attachment{}, err
}
defer cleanup()
if cfg.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
}
if out, err := c.run(ctx, "mount", mountArgs(loc, point, credentialsPath)...); err != nil {
return Attachment{}, fmt.Errorf("failed to mount %s at %s: %v", loc.SMBSource(), point, summarize(err, out))
}
return Attachment{
BasePath: filepath.Join(append([]string{point}, loc.Sub...)...),
Owned: true,
mountPoint: point,
Notes: []string{
"share: " + loc.SMBSource(),
"user: " + usernameNote(cfg.Username),
fmt.Sprintf("mounted at %s by this process (will unmount on shutdown)", point),
},
}, nil
}
// mountArgs builds the mount.cifs invocation. Kept pure so it can be asserted
// without mounting anything.
func mountArgs(loc Location, mountPoint, credentialsPath string) []string {
options := "rw"
if credentialsPath != "" {
options += ",credentials=" + credentialsPath
} else {
options += ",guest"
}
return []string{"-t", "cifs", loc.SMBSource(), mountPoint, "-o", options}
}
// writeCredentialsFile stages the NAS credentials in a 0600 file. mount.cifs
// also accepts user=/pass= options, but those land in the process table where
// any local user can read the password; a credentials file does not.
func writeCredentialsFile(cfg NASConfig) (path string, cleanup func(), err error) {
if cfg.Username == "" || cfg.Password == "" {
return "", func() {}, nil
}
dir, err := os.MkdirTemp("", "streamrecorder-cifs-")
if err != nil {
return "", func() {}, fmt.Errorf("failed to stage NAS credentials: %w", err)
}
cleanup = func() { os.RemoveAll(dir) }
path = filepath.Join(dir, "credentials")
content := fmt.Sprintf("username=%s\npassword=%s\n", cfg.Username, cfg.Password)
if err := os.WriteFile(path, []byte(content), 0600); err != nil {
cleanup()
return "", func() {}, fmt.Errorf("failed to write NAS credentials file: %w", err)
}
return path, cleanup, nil
}
// autoMountPoint chooses where to mount the share when the operator has not
// named a location. Everything lands under one predictable directory so a
// leftover mount is easy to find and clean up.
func autoMountPoint(loc Location, cfg NASConfig) string {
if cfg.MountPoint != "" {
return cfg.MountPoint
}
return filepath.Join("/mnt", "streamrecorder", strings.ToLower(loc.Server), strings.ToLower(loc.Share))
}
// notMountedMessage explains the failure in terms of what the operator should
// actually type. A bare "no such mount" would leave them guessing.
func notMountedMessage(loc Location, cfg NASConfig) string {
source := loc.SMBSource()
example := filepath.Join("/mnt", strings.ToLower(loc.Share))
credentials := ""
if cfg.Username != "" {
credentials = fmt.Sprintf(",username=%s,password=***", cfg.Username)
}
return fmt.Sprintf(
"%s is not mounted on this host. Either mount it and %s, for example:\n"+
" sudo mkdir -p %s && sudo mount -t cifs %s %s -o rw%s\n"+
"or set NAS_AUTO_MOUNT=true to have the recorder mount it (requires root)",
source, suggestMountPointConfig(loc), example, source, example, credentials)
}
func suggestMountPointConfig(loc Location) string {
if len(loc.Sub) == 0 {
return "set NAS_OUTPUT_PATH to the mount point"
}
return fmt.Sprintf("set NAS_OUTPUT_PATH to the mount point plus %s", loc.SubPath())
}

View File

@ -1,107 +0,0 @@
package nas
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// windowsConnector reaches a share the way Windows does: authenticate the UNC
// path with `net use`, then use the UNC path directly as a filesystem path.
// Nothing is mounted into a tree, so the resolved base path is the configured
// UNC path itself.
type windowsConnector struct {
run commandRunner
}
func (c *windowsConnector) Name() string { return "net use" }
func (c *windowsConnector) Attach(ctx context.Context, loc Location, cfg NASConfig) (Attachment, error) {
share := loc.ShareUNC()
att := Attachment{
BasePath: loc.UNC(),
share: share,
Notes: []string{"share: " + share, "user: " + usernameNote(cfg.Username)},
}
if cfg.Timeout > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, cfg.Timeout)
defer cancel()
}
out, err := c.run(ctx, "net", netUseArgs(loc, cfg)...)
if err != nil {
// A failure here is not decisive. `net use` also fails when the share
// is already connected — commonly because Explorer or another process
// got there first — in which case the path is perfectly usable. Record
// the reason and let the caller's write probe deliver the verdict,
// rather than refusing to run against a working share.
att.Notes = append(att.Notes,
fmt.Sprintf("net use did not complete (%v); continuing, the share may already be connected", summarize(err, out)))
return att, nil
}
att.Owned = true
att.Notes = append(att.Notes, "connected via net use")
return att, nil
}
func (c *windowsConnector) Detach(ctx context.Context, att Attachment) error {
// Only drop connections this process made. Deleting a mapping the user or
// another application established would break them.
if !att.Owned || att.share == "" {
return nil
}
if out, err := c.run(ctx, "net", "use", att.share, "/delete"); err != nil {
return fmt.Errorf("failed to disconnect %s: %v", att.share, summarize(err, out))
}
return nil
}
// netUseArgs builds the `net use` invocation. Kept pure so the Windows code
// path can be asserted from any host, including the Linux box this is
// developed on, without executing anything.
func netUseArgs(loc Location, cfg NASConfig) []string {
args := []string{"use", loc.ShareUNC()}
if cfg.Username != "" && cfg.Password != "" {
args = append(args, "/user:"+cfg.Username, cfg.Password)
}
return append(args, "/persistent:no")
}
// summarize pairs an exec error with the first line of command output, which is
// where net and mount put the actual diagnosis.
func summarize(err error, out []byte) string {
text := strings.TrimSpace(string(out))
if text == "" {
return err.Error()
}
if idx := strings.IndexByte(text, '\n'); idx >= 0 {
text = text[:idx]
}
return fmt.Sprintf("%v: %s", err, text)
}
func usernameNote(username string) string {
if username == "" {
return "(none, using ambient credentials)"
}
return username
}
// joinBase resolves a relative path against a base that may be a UNC path.
// filepath.Join is correct on Windows and would mangle the leading \\ anywhere
// else, so UNC bases are joined with backslashes explicitly.
func joinBase(base string, elem ...string) string {
if len(elem) == 0 {
return base
}
if strings.HasPrefix(base, `\\`) {
parts := append([]string{strings.TrimRight(base, `\`)}, elem...)
return strings.Join(parts, `\`)
}
return filepath.Join(append([]string{base}, elem...)...)
}

View File

@ -1,353 +0,0 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// recordingRunner captures the commands a connector would run instead of
// running them. This is what lets the Windows path be tested from a Linux
// development machine: the decision logic and the exact argv are asserted
// without ever invoking net.exe.
type recordingRunner struct {
calls [][]string
output []byte
err error
}
func (r *recordingRunner) run(_ context.Context, name string, args ...string) ([]byte, error) {
r.calls = append(r.calls, append([]string{name}, args...))
return r.output, r.err
}
func (r *recordingRunner) lastCall() string {
if len(r.calls) == 0 {
return ""
}
return strings.Join(r.calls[len(r.calls)-1], " ")
}
func remoteLocation(t *testing.T, raw string) Location {
t.Helper()
loc, err := ParseLocation(raw)
if err != nil {
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
}
return loc
}
func TestConnectorSelection(t *testing.T) {
remote := remoteLocation(t, `\\nas\dci\streams`)
local := remoteLocation(t, "/mnt/dci/streams")
cases := []struct {
name string
loc Location
goos string
want string
}{
{"windows share", remote, "windows", "net use"},
{"linux share", remote, "linux", "mount"},
{"darwin share", remote, "darwin", "unsupported"},
{"local path on windows", local, "windows", "local"},
{"local path on linux", local, "linux", "local"},
// A already-mounted path needs no network step even where shares are
// unsupported, so it must not fall through to the error connector.
{"local path on darwin", local, "darwin", "local"},
}
for _, tc := range cases {
if got := connectorFor(tc.loc, tc.goos).Name(); got != tc.want {
t.Errorf("%s: connectorFor = %q, want %q", tc.name, got, tc.want)
}
}
}
func TestWindowsConnectorAttach(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
loc := remoteLocation(t, `\\HomeLabNAS\dci\streams`)
att, err := c.Attach(context.Background(), loc, NASConfig{Username: "NASAdmin", Password: "secret"})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
// The base path is the full UNC path; net use authenticates only the share.
if want := `\\HomeLabNAS\dci\streams`; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if want := `net use \\HomeLabNAS\dci /user:NASAdmin secret /persistent:no`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
if !att.Owned {
t.Error("a successful net use should be owned so it can be released on shutdown")
}
}
func TestWindowsConnectorOmitsCredentialsWhenUnset(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := `net use \\nas\dci /persistent:no`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
}
func TestWindowsConnectorContinuesWhenAlreadyConnected(t *testing.T) {
// net use fails when a connection already exists, but the share is usable;
// refusing to run would be wrong. The write probe is the real verdict.
runner := &recordingRunner{
err: fmt.Errorf("exit status 2"),
output: []byte("The network connection could not be found.\nMore help is available..."),
}
c := &windowsConnector{run: runner.run}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
if err != nil {
t.Fatalf("Attach should tolerate a net use failure, got: %v", err)
}
if att.Owned {
t.Error("a failed net use must not be marked owned, or Detach would drop someone else's connection")
}
notes := strings.Join(att.Notes, "\n")
if !strings.Contains(notes, "net use did not complete") {
t.Errorf("expected the failure to be recorded in notes:\n%s", notes)
}
// Only the first line of output is worth surfacing.
if strings.Contains(notes, "More help is available") {
t.Errorf("notes should not carry the full usage dump:\n%s", notes)
}
}
func TestWindowsConnectorDetachOnlyReleasesOwned(t *testing.T) {
runner := &recordingRunner{}
c := &windowsConnector{run: runner.run}
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: false}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not disconnect a share this process did not connect, ran: %v", runner.calls)
}
if err := c.Detach(context.Background(), Attachment{share: `\\nas\dci`, Owned: true}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if want := `net use \\nas\dci /delete`; runner.lastCall() != want {
t.Errorf("command = %q, want %q", runner.lastCall(), want)
}
}
// writeMountTable stages a /proc/mounts-format file.
func writeMountTable(t *testing.T, lines ...string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "mounts")
body := strings.Join(lines, "\n") + "\n"
if err := os.WriteFile(path, []byte(body), 0644); err != nil {
t.Fatalf("failed to write mount table: %v", err)
}
return path
}
func TestLinuxConnectorResolvesExistingMount(t *testing.T) {
mounts := writeMountTable(t,
"sysfs /sys sysfs rw,nosuid 0 0",
"//homelabnas/dci /mnt/dci cifs rw,relatime,username=nasadmin 0 0",
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\HomeLabNAS\dci\streams`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
// Server and share names are case-insensitive on SMB, so the configured
// \\HomeLabNAS\dci must match a //homelabnas/dci mount.
if want := "/mnt/dci/streams"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if att.Owned {
t.Error("a pre-existing mount must not be owned, or shutdown would unmount it")
}
}
func TestLinuxConnectorPrefersDeepestMount(t *testing.T) {
// Both mounts could serve the destination; the more specific one is the
// only one guaranteed to actually contain it.
mounts := writeMountTable(t,
"//nas/dci /mnt/dci cifs rw 0 0",
"//nas/dci/streams /mnt/streams cifs rw 0 0",
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams\2026`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := "/mnt/streams/2026"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
}
func TestLinuxConnectorMatchesNFSAndEscapedPoints(t *testing.T) {
mounts := writeMountTable(t,
`nas:/dci /mnt/my\040share nfs4 rw 0 0`,
)
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{})
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := "/mnt/my share/streams"; att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
}
func TestLinuxConnectorIgnoresNonNetworkMounts(t *testing.T) {
// A local directory that merely looks like the share must not match.
mounts := writeMountTable(t, "//nas/dci /mnt/dci ext4 rw 0 0")
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
if _, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{}); err == nil {
t.Fatal("expected an ext4 mount not to satisfy a network share")
}
}
func TestLinuxConnectorUnmountedGivesActionableError(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
c := &linuxConnector{run: (&recordingRunner{}).run, mountsPath: mounts}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), NASConfig{Username: "admin"})
if err == nil {
t.Fatal("expected an unmounted share to fail")
}
msg := err.Error()
for _, want := range []string{"//nas/dci", "mount -t cifs", "NAS_AUTO_MOUNT", "NAS_OUTPUT_PATH"} {
if !strings.Contains(msg, want) {
t.Errorf("error should mention %q:\n%s", want, msg)
}
}
// The remedy must not echo the real password.
if strings.Contains(msg, "hunter2") {
t.Error("error message leaked a credential")
}
}
func TestLinuxConnectorAutoMountRequiresRoot(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
runner := &recordingRunner{}
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 1000 }}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{AutoMount: true})
if err == nil || !strings.Contains(err.Error(), "requires root") {
t.Fatalf("expected a root requirement error, got: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not attempt mount without privileges, ran: %v", runner.calls)
}
}
func TestLinuxConnectorAutoMount(t *testing.T) {
mounts := writeMountTable(t, "sysfs /sys sysfs rw 0 0")
runner := &recordingRunner{}
point := filepath.Join(t.TempDir(), "mnt", "dci")
c := &linuxConnector{run: runner.run, mountsPath: mounts, geteuid: func() int { return 0 }}
cfg := NASConfig{AutoMount: true, MountPoint: point, Username: "admin", Password: "hunter2", Timeout: 5 * time.Second}
att, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci\streams`), cfg)
if err != nil {
t.Fatalf("Attach failed: %v", err)
}
if want := filepath.Join(point, "streams"); att.BasePath != want {
t.Errorf("BasePath = %q, want %q", att.BasePath, want)
}
if !att.Owned || att.mountPoint != point {
t.Errorf("a self-created mount must be owned and recorded, got owned=%v point=%q", att.Owned, att.mountPoint)
}
call := runner.lastCall()
if !strings.HasPrefix(call, "mount -t cifs //nas/dci "+point+" -o rw,credentials=") {
t.Errorf("unexpected mount command: %q", call)
}
// The password must reach mount.cifs through a credentials file, never
// through argv where any local user could read it from the process table.
if strings.Contains(call, "hunter2") {
t.Errorf("password leaked into the mount command line: %q", call)
}
// The staged credentials file must not outlive the mount call.
credPath := call[strings.Index(call, "credentials=")+len("credentials="):]
if _, err := os.Stat(credPath); !os.IsNotExist(err) {
t.Errorf("credentials file %s should have been removed, stat err = %v", credPath, err)
}
if err := c.Detach(context.Background(), att); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if want := "umount " + point; runner.lastCall() != want {
t.Errorf("detach command = %q, want %q", runner.lastCall(), want)
}
}
func TestLinuxConnectorDetachLeavesForeignMounts(t *testing.T) {
runner := &recordingRunner{}
c := &linuxConnector{run: runner.run, mountsPath: writeMountTable(t)}
if err := c.Detach(context.Background(), Attachment{BasePath: "/mnt/dci", Owned: false}); err != nil {
t.Fatalf("Detach failed: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("must not unmount a share the host mounted, ran: %v", runner.calls)
}
}
func TestLocalConnectorRejectsWindowsPathsOffWindows(t *testing.T) {
c := &localConnector{goos: "linux"}
for _, raw := range []string{`C:\data\streams`, `data\streams`} {
loc, err := ParseLocation(raw)
if err != nil {
t.Fatalf("ParseLocation(%q) failed: %v", raw, err)
}
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err == nil {
t.Errorf("%q should be rejected on Linux: backslashes are legal filename characters there, "+
"so it would silently create a local directory instead of writing to the NAS", raw)
}
}
}
func TestLocalConnectorAcceptsWindowsPathsOnWindows(t *testing.T) {
c := &localConnector{goos: "windows"}
loc, err := ParseLocation(`C:\data\streams`)
if err != nil {
t.Fatalf("ParseLocation failed: %v", err)
}
if _, err := c.Attach(context.Background(), loc, NASConfig{}); err != nil {
t.Errorf("a drive path is valid on Windows: %v", err)
}
}
func TestUnsupportedConnectorExplainsItself(t *testing.T) {
c := &unsupportedConnector{goos: "plan9"}
_, err := c.Attach(context.Background(), remoteLocation(t, `\\nas\dci`), NASConfig{})
if err == nil {
t.Fatal("expected an error on an unsupported platform")
}
if !strings.Contains(err.Error(), "plan9") || !strings.Contains(err.Error(), "NAS_OUTPUT_PATH") {
t.Errorf("error should name the platform and the remedy: %v", err)
}
}

View File

@ -1,144 +0,0 @@
package nas
import (
"fmt"
"path/filepath"
"regexp"
"strings"
)
// Location is a configured NAS destination parsed into a host-neutral form.
//
// The same destination is spelled differently depending on where you're
// standing: Windows wants \\HomeLabNAS\dci\streams, a Linux host reaches the
// same bytes through a mount point like /mnt/dci/streams, and either host may
// simply be handed a plain local directory. Parsing happens once, up front,
// and platform-specific rendering is left to the Connector implementations.
type Location struct {
// Raw is the destination exactly as configured, for error messages.
Raw string
// Server is the host serving the share; empty for a plain local path.
Server string
// Share is the top-level export name; empty for a plain local path.
Share string
// Sub holds the path components below the share, if any.
Sub []string
}
// driveLetter matches a Windows drive-qualified path such as C:\data.
var driveLetter = regexp.MustCompile(`^[A-Za-z]:[\\/]`)
// ParseLocation accepts every spelling of a destination this project supports:
// Windows UNC (\\server\share\dir), the SMB source form (//server/share/dir),
// a smb:// URL, or a plain local/mounted directory. It does no I/O and makes no
// assumptions about the host OS, so the result is safe to compute anywhere.
func ParseLocation(raw string) (Location, error) {
trimmed := strings.TrimSpace(raw)
if trimmed == "" {
return Location{}, fmt.Errorf("NAS path is empty")
}
loc := Location{Raw: trimmed}
remainder, isRemote := stripRemotePrefix(trimmed)
if !isRemote {
return loc, nil
}
parts := splitPathComponents(remainder)
if len(parts) < 2 {
return Location{}, fmt.Errorf(
"%q names a server but no share; expected a form like \\\\server\\share\\folder", trimmed)
}
loc.Server = parts[0]
loc.Share = parts[1]
loc.Sub = parts[2:]
return loc, nil
}
// stripRemotePrefix reports whether raw names a network share and returns the
// server/share/subpath portion when it does. A leading // is treated as a share
// rather than a local path: POSIX leaves paths beginning with exactly two
// slashes implementation-defined, and nobody configures a NAS destination that
// way by accident.
func stripRemotePrefix(raw string) (string, bool) {
lower := strings.ToLower(raw)
switch {
case strings.HasPrefix(lower, "smb://"):
return raw[len("smb://"):], true
case strings.HasPrefix(lower, "cifs://"):
return raw[len("cifs://"):], true
case strings.HasPrefix(raw, `\\`):
return raw[2:], true
case strings.HasPrefix(raw, "//"):
return raw[2:], true
default:
return raw, false
}
}
// splitPathComponents splits on either separator and drops empty components, so
// that \\server\share\\folder\ and //server/share/folder agree.
func splitPathComponents(path string) []string {
fields := strings.FieldsFunc(path, func(r rune) bool { return r == '/' || r == '\\' })
out := make([]string, 0, len(fields))
for _, f := range fields {
if f != "" {
out = append(out, f)
}
}
return out
}
// IsRemote reports whether reaching this destination requires attaching a
// network share first, as opposed to writing to a directory that is already
// part of the local filesystem.
func (l Location) IsRemote() bool {
return l.Server != "" && l.Share != ""
}
// UNC renders the destination in Windows form: \\server\share\sub.
func (l Location) UNC() string {
if !l.IsRemote() {
return l.Raw
}
return `\\` + strings.Join(append([]string{l.Server, l.Share}, l.Sub...), `\`)
}
// ShareUNC renders just the share in Windows form: \\server\share. This is the
// unit `net use` authenticates against; the subpath below it is an ordinary
// directory on the connected share.
func (l Location) ShareUNC() string {
if !l.IsRemote() {
return ""
}
return `\\` + l.Server + `\` + l.Share
}
// SMBSource renders the share in the //server/share form used by mount tables
// and by mount.cifs. It deliberately omits Sub: that part is a path *within*
// the mounted share, not part of the share's identity.
func (l Location) SMBSource() string {
if !l.IsRemote() {
return ""
}
return "//" + l.Server + "/" + l.Share
}
// SubPath renders the components below the share using the host's separator, or
// "" when the destination is the share root.
func (l Location) SubPath() string {
if len(l.Sub) == 0 {
return ""
}
return filepath.Join(l.Sub...)
}
// String describes the destination for logs and check output.
func (l Location) String() string {
if l.IsRemote() {
return l.UNC()
}
return l.Raw
}

View File

@ -1,123 +0,0 @@
package nas
import (
"strings"
"testing"
)
func TestParseLocationRemoteForms(t *testing.T) {
// Every spelling of the same destination must parse identically, so that
// an operator can configure it the way their platform's documentation
// describes without changing behaviour.
equivalent := []string{
`\\HomeLabNAS\dci\streams`,
`//HomeLabNAS/dci/streams`,
`smb://HomeLabNAS/dci/streams`,
`SMB://HomeLabNAS/dci/streams`,
`cifs://HomeLabNAS/dci/streams`,
`\\HomeLabNAS\dci\streams\`,
`\\HomeLabNAS\\dci\\streams`,
}
for _, raw := range equivalent {
loc, err := ParseLocation(raw)
if err != nil {
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
continue
}
if !loc.IsRemote() {
t.Errorf("ParseLocation(%q) should be remote", raw)
}
if loc.Server != "HomeLabNAS" || loc.Share != "dci" {
t.Errorf("ParseLocation(%q) = server %q share %q, want HomeLabNAS/dci", raw, loc.Server, loc.Share)
}
if len(loc.Sub) != 1 || loc.Sub[0] != "streams" {
t.Errorf("ParseLocation(%q) sub = %v, want [streams]", raw, loc.Sub)
}
}
}
func TestParseLocationRendering(t *testing.T) {
loc, err := ParseLocation(`//HomeLabNAS/dci/streams/2026`)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got, want := loc.UNC(), `\\HomeLabNAS\dci\streams\2026`; got != want {
t.Errorf("UNC() = %q, want %q", got, want)
}
if got, want := loc.ShareUNC(), `\\HomeLabNAS\dci`; got != want {
t.Errorf("ShareUNC() = %q, want %q", got, want)
}
if got, want := loc.SMBSource(), "//HomeLabNAS/dci"; got != want {
t.Errorf("SMBSource() = %q, want %q", got, want)
}
}
func TestParseLocationShareRoot(t *testing.T) {
loc, err := ParseLocation(`\\nas\media`)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(loc.Sub) != 0 {
t.Errorf("expected no subpath, got %v", loc.Sub)
}
if got, want := loc.UNC(), `\\nas\media`; got != want {
t.Errorf("UNC() = %q, want %q", got, want)
}
}
func TestParseLocationLocalPaths(t *testing.T) {
for _, raw := range []string{"/mnt/nas/streams", "data/streams", `C:\data\streams`, "/mnt"} {
loc, err := ParseLocation(raw)
if err != nil {
t.Errorf("ParseLocation(%q) failed: %v", raw, err)
continue
}
if loc.IsRemote() {
t.Errorf("ParseLocation(%q) should not be treated as a network share", raw)
}
if loc.Raw != raw {
t.Errorf("ParseLocation(%q).Raw = %q", raw, loc.Raw)
}
}
}
func TestParseLocationRejectsIncomplete(t *testing.T) {
for _, raw := range []string{"", " ", `\\`, `\\server`, "//server", "smb://server"} {
if _, err := ParseLocation(raw); err == nil {
t.Errorf("ParseLocation(%q) should have failed", raw)
}
}
}
func TestParseLocationTrimsWhitespace(t *testing.T) {
loc, err := ParseLocation(" \\\\nas\\dci ")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if loc.Server != "nas" || loc.Share != "dci" {
t.Errorf("whitespace not trimmed: server %q share %q", loc.Server, loc.Share)
}
}
func TestJoinBaseKeepsUNCSeparators(t *testing.T) {
// filepath.Join would produce \\nas\dci/event on Linux, which is not a
// path Windows would accept if the value were ever written to state.
got := joinBase(`\\nas\dci`, "event", "1080p")
if want := `\\nas\dci\event\1080p`; got != want {
t.Errorf("joinBase UNC = %q, want %q", got, want)
}
if got, want := joinBase(`\\nas\dci\`, "event"), `\\nas\dci\event`; got != want {
t.Errorf("joinBase trailing separator = %q, want %q", got, want)
}
if got := joinBase("/mnt/dci"); got != "/mnt/dci" {
t.Errorf("joinBase with no elements = %q", got)
}
if got, want := joinBase("/mnt/dci", "event"), "/mnt/dci/event"; !strings.HasSuffix(got, want) {
t.Errorf("joinBase local = %q, want suffix %q", got, want)
}
}

View File

@ -1,135 +0,0 @@
package nas
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
)
// procMountsPath is the kernel's view of what is mounted in this process's
// mount namespace, which is the right question to ask inside a container too.
const procMountsPath = "/proc/mounts"
// mountEntry is one line of the mount table.
type mountEntry struct {
Source string
Point string
FSType string
}
// networkFilesystems are the mount types that can plausibly be serving a NAS
// share. Restricting the search avoids matching, say, a local directory that
// merely happens to be named like a server.
var networkFilesystems = map[string]bool{
"cifs": true,
"smb3": true,
"smbfs": true,
"nfs": true,
"nfs4": true,
"fuse.smbnetfs": true,
}
// readMountTable parses /proc/mounts (or any file in its format). Fields are
// whitespace-separated: source, mount point, type, options, dump, pass.
func readMountTable(path string) ([]mountEntry, error) {
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("failed to read mount table %s: %w", path, err)
}
defer f.Close()
var entries []mountEntry
scanner := bufio.NewScanner(f)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) < 3 {
continue
}
entries = append(entries, mountEntry{
Source: unescapeMountField(fields[0]),
Point: unescapeMountField(fields[1]),
FSType: fields[2],
})
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("failed to scan mount table %s: %w", path, err)
}
return entries, nil
}
// unescapeMountField decodes the octal escapes the kernel uses for characters
// that would otherwise break the whitespace-separated format — most often \040
// for a space in a mount point.
func unescapeMountField(field string) string {
if !strings.Contains(field, `\`) {
return field
}
var b strings.Builder
for i := 0; i < len(field); i++ {
if field[i] == '\\' && i+3 < len(field) {
if v, err := strconv.ParseUint(field[i+1:i+4], 8, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(field[i])
}
return b.String()
}
// findMount locates the mount serving loc and returns its mount point plus the
// path components still to be appended below it.
//
// The search runs deepest-first because a share may be mounted at any level:
// //nas/dci might be mounted at /mnt/dci, or //nas/dci/streams mounted directly
// at /mnt/streams. Preferring the longest matching source keeps both working
// and avoids pointing at a parent that the mount does not actually cover.
func findMount(entries []mountEntry, loc Location) (point string, rest []string, ok bool) {
if !loc.IsRemote() {
return "", nil, false
}
for depth := len(loc.Sub); depth >= 0; depth-- {
for _, candidate := range mountSourceForms(loc, depth) {
for _, entry := range entries {
if !networkFilesystems[entry.FSType] {
continue
}
if normalizeMountSource(entry.Source) == candidate {
return entry.Point, loc.Sub[depth:], true
}
}
}
}
return "", nil, false
}
// mountSourceForms renders the share, plus its first `depth` subdirectories, in
// the spellings a mount table might use for it.
func mountSourceForms(loc Location, depth int) []string {
segments := append([]string{loc.Share}, loc.Sub[:depth]...)
tail := strings.Join(segments, "/")
return []string{
normalizeMountSource("//" + loc.Server + "/" + tail), // cifs/smb3
normalizeMountSource(loc.Server + ":/" + tail), // nfs
}
}
// normalizeMountSource makes sources comparable: SMB servers and shares are
// case-insensitive, separators may be either slash, and a trailing slash is not
// meaningful.
func normalizeMountSource(source string) string {
s := strings.ReplaceAll(source, `\`, "/")
s = strings.ToLower(strings.TrimSpace(s))
if len(s) > 1 {
s = strings.TrimRight(s, "/")
}
return s
}

View File

@ -1,96 +0,0 @@
package nas
import (
"os"
"testing"
)
func TestUnescapeMountField(t *testing.T) {
cases := map[string]string{
`/mnt/plain`: "/mnt/plain",
`/mnt/my\040share`: "/mnt/my share",
`/mnt/tab\011here`: "/mnt/tab\there",
`/mnt/back\134slash`: `/mnt/back\slash`,
`//nas/dci`: "//nas/dci",
`/mnt/not\9octal`: `/mnt/not\9octal`,
`/mnt/trailing\`: `/mnt/trailing\`,
}
for in, want := range cases {
if got := unescapeMountField(in); got != want {
t.Errorf("unescapeMountField(%q) = %q, want %q", in, got, want)
}
}
}
func TestNormalizeMountSource(t *testing.T) {
cases := map[string]string{
"//NAS/DCI": "//nas/dci",
"//nas/dci/": "//nas/dci",
`\\nas\dci`: "//nas/dci",
" //nas/dci ": "//nas/dci",
"/": "/",
}
for in, want := range cases {
if got := normalizeMountSource(in); got != want {
t.Errorf("normalizeMountSource(%q) = %q, want %q", in, got, want)
}
}
}
func TestReadMountTableSkipsMalformedLines(t *testing.T) {
path := writeMountTable(t,
"source-only",
"source /point-but-no-type",
"",
"//nas/dci /mnt/dci cifs rw 0 0",
)
entries, err := readMountTable(path)
if err != nil {
t.Fatalf("readMountTable failed: %v", err)
}
if len(entries) != 1 {
t.Fatalf("expected 1 usable entry, got %d: %v", len(entries), entries)
}
if entries[0].Source != "//nas/dci" || entries[0].Point != "/mnt/dci" || entries[0].FSType != "cifs" {
t.Errorf("unexpected entry: %+v", entries[0])
}
}
func TestReadMountTableMissingFile(t *testing.T) {
if _, err := readMountTable("/definitely/not/a/mount/table"); err == nil {
t.Error("expected an error for a missing mount table")
}
}
// TestReadRealMountTable closes the loop between the fixtures above and the
// kernel's actual output: the parser must cope with whatever this host has
// mounted, not just hand-written examples.
func TestReadRealMountTable(t *testing.T) {
if _, err := os.Stat(procMountsPath); err != nil {
t.Skipf("%s is not available on this platform", procMountsPath)
}
entries, err := readMountTable(procMountsPath)
if err != nil {
t.Fatalf("failed to parse the host's %s: %v", procMountsPath, err)
}
if len(entries) == 0 {
t.Fatalf("%s parsed to zero entries, which cannot be right", procMountsPath)
}
foundRoot := false
for _, e := range entries {
if e.Point == "" || e.FSType == "" {
t.Errorf("entry parsed with empty fields: %+v", e)
}
if e.Point == "/" {
foundRoot = true
}
}
if !foundRoot {
t.Error("expected the root filesystem in the host's mount table")
}
}

View File

@ -6,168 +6,31 @@ import (
"io" "io"
"log" "log"
"os" "os"
"sync" "os/exec"
"path/filepath"
"strings"
) )
// NASService writes to a NAS destination without caring how this host reaches
// it. Parsing the destination and attaching it are delegated to a Location and
// a Connector; everything below operates on the resolved base path, which is
// always an ordinary path this host's filesystem understands.
type NASService struct { type NASService struct {
Config NASConfig Config NASConfig
location Location
connector Connector
mu sync.RWMutex
attached Attachment
connected bool connected bool
} }
// New parses the configured destination and selects a connection strategy for func NewNASService(config NASConfig) *NASService {
// the running host. It performs no I/O and does not touch the network, so it is nt := &NASService{
// safe to call before deciding whether a transfer will actually happen; call Config: config,
// Connect to attach the destination. }
func New(config NASConfig) (*NASService, error) {
loc, err := ParseLocation(config.Path) // Establish network connection with credentials before accessing the path
if err := nt.EstablishConnection(); err != nil {
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
}
err := nt.EnsureDirectoryExists(nt.Config.Path)
if err != nil { if err != nil {
return nil, err log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err)
} }
return nt
return &NASService{
Config: config,
location: loc,
connector: defaultConnector(loc),
}, nil
}
// Connect attaches the destination, ensures the target directory exists, and
// proves it is writable. The write probe matters: on both platforms a share can
// be reachable and still reject writes, and finding that out at startup beats
// discovering it mid-recording.
func (nt *NASService) Connect(ctx context.Context) error {
att, err := nt.connector.Attach(ctx, nt.location, nt.Config)
if err != nil {
return err
}
nt.mu.Lock()
nt.attached = att
nt.mu.Unlock()
// From here on the attachment exists, so any failure has to undo it —
// otherwise a mount this process created would outlive the failed startup
// and be left behind on the host.
if err := nt.EnsureDirectoryExists(att.BasePath); err != nil {
nt.abandon(ctx, att)
return fmt.Errorf("%s is reachable but %s is unusable: %w", nt.location, att.BasePath, err)
}
if err := nt.writeProbe(att.BasePath); err != nil {
nt.abandon(ctx, att)
return err
}
nt.mu.Lock()
nt.connected = true
nt.mu.Unlock()
log.Printf("Connected to NAS at %s", att.BasePath)
return nil
}
// abandon releases a half-finished attachment after Connect fails. The
// original error is the one worth reporting, so a teardown failure is only
// logged.
func (nt *NASService) abandon(ctx context.Context, att Attachment) {
// Keep the notes: they explain how far the host got, which is exactly what
// the caller needs to report. Everything else is cleared so the service
// cannot be mistaken for usable and Detach cannot run twice.
nt.mu.Lock()
nt.attached = Attachment{Notes: att.Notes}
nt.connected = false
nt.mu.Unlock()
if err := nt.connector.Detach(ctx, att); err != nil {
log.Printf("Warning: failed to release NAS attachment after a failed connect: %v", err)
}
}
// writeProbe creates and removes a file to confirm the destination accepts
// writes.
func (nt *NASService) writeProbe(basePath string) error {
testFile := joinBase(basePath, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("%s is not writable: %w", basePath, err)
}
f.Close()
if err := os.Remove(testFile); err != nil {
log.Printf("Warning: failed to remove connection test file %s: %v", testFile, err)
}
return nil
}
// Disconnect releases only what Connect established. A share that was already
// mounted or already connected when the recorder started is left exactly as it
// was found.
func (nt *NASService) Disconnect(ctx context.Context) error {
nt.mu.Lock()
att := nt.attached
nt.attached = Attachment{}
nt.connected = false
nt.mu.Unlock()
if err := nt.connector.Detach(ctx, att); err != nil {
return err
}
if att.Owned {
log.Printf("Released NAS connection to %s", nt.location)
}
return nil
}
// LocalPath is the resolved base directory on this host, valid after Connect.
func (nt *NASService) LocalPath() string {
nt.mu.RLock()
defer nt.mu.RUnlock()
return nt.attached.BasePath
}
// Join resolves elements against the NAS base path using whatever separator
// this host's base path requires.
func (nt *NASService) Join(elem ...string) string {
return joinBase(nt.LocalPath(), elem...)
}
// Location returns the parsed destination.
func (nt *NASService) Location() Location { return nt.location }
// ConnectionInfo describes how this host reaches the destination, for check
// output and startup logs. Safe to call before Connect, when it reports only
// what is known from configuration.
func (nt *NASService) ConnectionInfo() []string {
nt.mu.RLock()
defer nt.mu.RUnlock()
info := []string{
"configured: " + nt.location.String(),
"strategy: " + nt.connector.Name(),
}
info = append(info, nt.attached.Notes...)
if nt.attached.BasePath != "" && nt.attached.BasePath != nt.location.Raw {
info = append(info, "resolved path: "+nt.attached.BasePath)
}
return info
}
func (nt *NASService) IsConnected() bool {
nt.mu.RLock()
defer nt.mu.RUnlock()
return nt.connected
} }
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error { func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
@ -226,10 +89,90 @@ func (nt *NASService) EnsureDirectoryExists(path string) error {
return nil return nil
} }
// FileExists checks whether a file already exists on the NAS, optionally func (nt *NASService) EstablishConnection() error {
// verifying its size. destinationPath is relative to the NAS base path. // Extract the network path (\\server\share) from the full path
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
// Local path, no authentication needed
return nil
}
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
// Use Windows net use command to establish authenticated connection
var cmd *exec.Cmd
if nt.Config.Username != "" && nt.Config.Password != "" {
cmd = exec.Command("net", "use", networkPath, nt.Config.Password, "/user:"+nt.Config.Username, "/persistent:no")
} else {
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
}
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
}
log.Printf("Network connection established successfully")
return nil
}
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
// Extract \\server\share from paths like \\server\share\folder\subfolder
if !strings.HasPrefix(fullPath, "\\\\") {
return "" // Not a UNC path
}
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
if len(parts) < 2 {
return "" // Invalid UNC path
}
return "\\\\" + parts[0] + "\\" + parts[1]
}
func (nt *NASService) TestConnection() error {
testFile := filepath.Join(nt.Config.Path, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("Failed to create test file: %w", err)
}
f.Close()
os.Remove(testFile)
nt.connected = true
log.Printf("Connected to NAS at %s", nt.Config.Path)
return nil
}
func (nt *NASService) IsConnected() bool {
return nt.connected
}
// Disconnect removes the network connection
func (nt *NASService) Disconnect() error {
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
return nil // Local path, nothing to disconnect
}
cmd := exec.Command("net", "use", networkPath, "/delete")
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
// Don't return error since this is cleanup
} else {
log.Printf("Disconnected from network path: %s", networkPath)
}
nt.connected = false
return nil
}
// FileExists checks if a file already exists on the NAS and optionally verifies size
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) { func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
fullDestPath := nt.Join(destinationPath) fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
destInfo, err := os.Stat(fullDestPath) destInfo, err := os.Stat(fullDestPath)
if err != nil { if err != nil {
@ -249,9 +192,9 @@ func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bo
return true, nil return true, nil
} }
// GetFileSize returns the size of a file on the NAS, relative to the base path. // GetFileSize returns the size of a file on the NAS
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) { func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
fullDestPath := nt.Join(destinationPath) fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
destInfo, err := os.Stat(fullDestPath) destInfo, err := os.Stat(fullDestPath)
if err != nil { if err != nil {

View File

@ -1,308 +0,0 @@
package nas
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
)
// fakeConnector stands in for a platform strategy so the service's own
// behaviour — probing, path resolution, teardown — can be tested without a NAS.
type fakeConnector struct {
basePath string
attachErr error
owned bool
attachCount int
detachCount int
detached Attachment
}
func (f *fakeConnector) Name() string { return "fake" }
func (f *fakeConnector) Attach(context.Context, Location, NASConfig) (Attachment, error) {
f.attachCount++
if f.attachErr != nil {
return Attachment{}, f.attachErr
}
return Attachment{BasePath: f.basePath, Owned: f.owned, Notes: []string{"fake attachment"}}, nil
}
func (f *fakeConnector) Detach(_ context.Context, att Attachment) error {
f.detachCount++
f.detached = att
return nil
}
func newTestService(t *testing.T, configuredPath string, conn Connector) *NASService {
t.Helper()
svc, err := New(NASConfig{Path: configuredPath})
if err != nil {
t.Fatalf("New(%q) failed: %v", configuredPath, err)
}
svc.connector = conn
return svc
}
func TestNewRejectsUnusableDestination(t *testing.T) {
if _, err := New(NASConfig{Path: ""}); err == nil {
t.Error("an empty NAS path should be rejected")
}
if _, err := New(NASConfig{Path: `\\server`}); err == nil {
t.Error("a server without a share should be rejected")
}
}
func TestNewDoesNoIO(t *testing.T) {
// Constructing the service must not touch the network or filesystem, so
// that modes which never transfer can build one harmlessly.
svc, err := New(NASConfig{Path: `\\nas\dci\streams`})
if err != nil {
t.Fatalf("New failed: %v", err)
}
if svc.IsConnected() {
t.Error("service should not report connected before Connect")
}
if svc.LocalPath() != "" {
t.Errorf("no path should be resolved before Connect, got %q", svc.LocalPath())
}
}
func TestConnectResolvesAndProbes(t *testing.T) {
base := filepath.Join(t.TempDir(), "streams")
conn := &fakeConnector{basePath: base}
svc := newTestService(t, `\\nas\dci\streams`, conn)
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
if !svc.IsConnected() {
t.Error("service should report connected")
}
if svc.LocalPath() != base {
t.Errorf("LocalPath = %q, want %q", svc.LocalPath(), base)
}
// Connect creates the destination directory if it is missing.
if info, err := os.Stat(base); err != nil || !info.IsDir() {
t.Errorf("Connect should have created %s: %v", base, err)
}
// The write probe must clean up after itself.
if _, err := os.Stat(filepath.Join(base, ".connection_test")); !os.IsNotExist(err) {
t.Errorf("probe file should have been removed, stat err = %v", err)
}
}
func TestConnectSurfacesAttachFailure(t *testing.T) {
conn := &fakeConnector{attachErr: fmt.Errorf("share is not mounted")}
svc := newTestService(t, `\\nas\dci`, conn)
err := svc.Connect(context.Background())
if err == nil || !strings.Contains(err.Error(), "not mounted") {
t.Fatalf("expected the connector's error to surface, got: %v", err)
}
if svc.IsConnected() {
t.Error("a failed attach must not leave the service marked connected")
}
}
func TestConnectFailsWhenDestinationUnwritable(t *testing.T) {
// A file where the directory should be: reachable, but unusable.
blocker := filepath.Join(t.TempDir(), "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, "/tmp/whatever", &fakeConnector{basePath: filepath.Join(blocker, "streams")})
if err := svc.Connect(context.Background()); err == nil {
t.Fatal("expected an unusable destination to fail the connect")
}
if svc.IsConnected() {
t.Error("service should not report connected after a failed probe")
}
}
func TestFailedProbeReleasesTheAttachment(t *testing.T) {
// A mount created by Attach must not outlive a failed Connect, or a failed
// startup leaves a mount behind on the host.
blocker := filepath.Join(t.TempDir(), "blocker")
if err := os.WriteFile(blocker, []byte("x"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
conn := &fakeConnector{basePath: filepath.Join(blocker, "streams"), owned: true}
svc := newTestService(t, `\\nas\dci`, conn)
if err := svc.Connect(context.Background()); err == nil {
t.Fatal("expected Connect to fail")
}
if conn.detachCount != 1 {
t.Errorf("Detach called %d times after a failed connect, want 1", conn.detachCount)
}
if !conn.detached.Owned {
t.Error("Detach should have been told the attachment was owned")
}
if svc.LocalPath() != "" {
t.Errorf("a released attachment should expose no path, got %q", svc.LocalPath())
}
// The diagnosis of how far it got must survive for the caller to report.
if info := strings.Join(svc.ConnectionInfo(), "\n"); !strings.Contains(info, "fake attachment") {
t.Errorf("connection notes should survive a failed connect:\n%s", info)
}
// A later Disconnect must not detach a second time.
if err := svc.Disconnect(context.Background()); err != nil {
t.Fatalf("Disconnect failed: %v", err)
}
if conn.detachCount != 2 || conn.detached.Owned {
t.Errorf("Disconnect after release should be a no-op teardown, detachCount=%d owned=%v",
conn.detachCount, conn.detached.Owned)
}
}
func TestJoinUsesResolvedPath(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
// Crucially the join is against the resolved path, not the configured UNC
// path — that is the whole point of the indirection.
if got, want := svc.Join("event", "1080p"), filepath.Join(base, "event", "1080p"); got != want {
t.Errorf("Join = %q, want %q", got, want)
}
}
func TestFileExistsAndGetFileSizeUseResolvedPath(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci`, &fakeConnector{basePath: base})
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
rel := filepath.Join("event", "seg.ts")
if err := os.MkdirAll(filepath.Join(base, "event"), 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := os.WriteFile(filepath.Join(base, rel), []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
exists, err := svc.FileExists(rel, 10)
if err != nil || !exists {
t.Errorf("FileExists = %v, %v; want true, nil", exists, err)
}
// A size mismatch means a partial transfer, which should not count.
exists, err = svc.FileExists(rel, 99)
if err != nil || exists {
t.Errorf("FileExists with wrong size = %v, %v; want false, nil", exists, err)
}
exists, err = svc.FileExists(filepath.Join("event", "missing.ts"), 0)
if err != nil || exists {
t.Errorf("FileExists for missing file = %v, %v; want false, nil", exists, err)
}
size, err := svc.GetFileSize(rel)
if err != nil || size != 10 {
t.Errorf("GetFileSize = %d, %v; want 10, nil", size, err)
}
}
func TestDisconnectDelegatesAndResets(t *testing.T) {
conn := &fakeConnector{basePath: t.TempDir(), owned: true}
svc := newTestService(t, `\\nas\dci`, conn)
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
if err := svc.Disconnect(context.Background()); err != nil {
t.Fatalf("Disconnect failed: %v", err)
}
if conn.detachCount != 1 {
t.Errorf("Detach called %d times, want 1", conn.detachCount)
}
if !conn.detached.Owned {
t.Error("Detach should receive the attachment so it knows what it owns")
}
if svc.IsConnected() {
t.Error("service should not report connected after Disconnect")
}
}
func TestConnectionInfoDescribesResolution(t *testing.T) {
base := t.TempDir()
svc := newTestService(t, `\\nas\dci\streams`, &fakeConnector{basePath: base})
// Before connecting, only configuration is known.
before := strings.Join(svc.ConnectionInfo(), "\n")
if !strings.Contains(before, `\\nas\dci\streams`) || !strings.Contains(before, "fake") {
t.Errorf("pre-connect info should name the destination and strategy:\n%s", before)
}
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Connect failed: %v", err)
}
after := strings.Join(svc.ConnectionInfo(), "\n")
if !strings.Contains(after, "resolved path: "+base) {
t.Errorf("post-connect info should report where the destination resolved to:\n%s", after)
}
}
func TestVerifyTransferDetectsSizeMismatch(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "src.ts")
dst := filepath.Join(dir, "dst.ts")
if err := os.WriteFile(src, []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := os.WriteFile(dst, []byte("012"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
if err := svc.VerifyTransfer(src, dst); err == nil {
t.Error("a truncated destination should fail verification")
}
if err := os.WriteFile(dst, []byte("0123456789"), 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
if err := svc.VerifyTransfer(src, dst); err != nil {
t.Errorf("matching sizes should verify: %v", err)
}
}
func TestCopyFile(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "src.ts")
dst := filepath.Join(dir, "dst.ts")
payload := []byte("segment payload")
if err := os.WriteFile(src, payload, 0644); err != nil {
t.Fatalf("setup failed: %v", err)
}
svc := newTestService(t, dir, &fakeConnector{basePath: dir})
if err := svc.CopyFile(context.Background(), src, dst); err != nil {
t.Fatalf("CopyFile failed: %v", err)
}
got, err := os.ReadFile(dst)
if err != nil {
t.Fatalf("failed to read copy: %v", err)
}
if string(got) != string(payload) {
t.Errorf("copied contents = %q, want %q", got, payload)
}
if err := svc.CopyFile(context.Background(), filepath.Join(dir, "missing.ts"), dst); err == nil {
t.Error("copying a missing source should fail")
}
}

View File

@ -5,61 +5,48 @@ import (
"context" "context"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/config" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas" "m3u8-downloader/pkg/nas"
"m3u8-downloader/pkg/utils"
"os" "os"
"os/exec" "os/exec"
"path/filepath"
"regexp" "regexp"
"runtime"
"sort"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
) )
type ProcessingService struct { type ProcessingService struct {
config *config.Config config *ProcessConfig
eventName string nas *nas.NASService
nas *nas.NASService
} }
func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingService, error) { func NewProcessingService(eventName string) (*ProcessingService, error) {
if cfg == nil { config := &ProcessConfig{
return nil, fmt.Errorf("configuration is required") WorkerCount: constants.ProcessWorkerCount,
SourcePath: constants.NASOutputPath + "/" + eventName,
DestinationPath: constants.ProcessOutputPath,
Enabled: constants.ProcessingEnabled,
EventName: eventName,
} }
nasConfig := nas.NASConfig{ nasConfig := &nas.NASConfig{
Path: cfg.NAS.OutputPath, Path: constants.NASOutputPath,
Username: cfg.NAS.Username, Username: constants.NASUsername,
Password: cfg.NAS.Password, Password: constants.NASPassword,
Timeout: cfg.NAS.Timeout, Timeout: constants.TransferTimeout,
RetryLimit: cfg.NAS.RetryLimit, RetryLimit: constants.TransferRetryLimit,
VerifySize: true, VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
} }
nasService, err := nas.New(nasConfig) nasService := nas.NewNASService(*nasConfig)
if err != nil {
logging.MustGetFactory().Processing.Error("invalid NAS destination", logging.KeyError, err.Error())
return nil, fmt.Errorf("invalid NAS destination: %w", err)
}
connectCtx, cancel := context.WithTimeout(context.Background(), cfg.NAS.Timeout) if err := nasService.TestConnection(); err != nil {
defer cancel() return nil, fmt.Errorf("Failed to connect to NAS: %w", err)
if err := nasService.Connect(connectCtx); err != nil {
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
} }
return &ProcessingService{ return &ProcessingService{
config: cfg, config: config,
eventName: eventName, nas: nasService,
nas: nasService,
}, nil }, nil
} }
@ -67,20 +54,11 @@ func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
return nil, nil return nil, nil
} }
// nasEventPath is the event's directory on the NAS as reached from this host.
// It goes through the NAS service rather than config.GetNASEventPath because
// the configured path may be a UNC path that this host reaches via a mount
// point instead.
func (ps *ProcessingService) nasEventPath() string {
return ps.nas.Join(ps.eventName)
}
func (ps *ProcessingService) GetEventDirs() ([]string, error) { func (ps *ProcessingService) GetEventDirs() ([]string, error) {
if ps.eventName == "" { if ps.config.EventName == "" {
sourcePath := ps.nas.LocalPath() dirs, err := os.ReadDir(ps.config.SourcePath)
dirs, err := os.ReadDir(sourcePath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err) return nil, fmt.Errorf("Failed to read directory: %w", err)
} }
var eventDirs []string var eventDirs []string
for _, dir := range dirs { for _, dir := range dirs {
@ -90,37 +68,23 @@ func (ps *ProcessingService) GetEventDirs() ([]string, error) {
} }
return eventDirs, nil return eventDirs, nil
} else { } else {
return []string{ps.eventName}, nil return []string{ps.config.EventName}, nil
} }
} }
func (ps *ProcessingService) Start(ctx context.Context) error { func (ps *ProcessingService) Start(ctx context.Context) error {
procLog := logging.MustGetFactory().Processing if !ps.config.Enabled {
registry := health.MustGetRegistry()
if !ps.config.Processing.Enabled {
log.Println("Processing service disabled") log.Println("Processing service disabled")
return nil return nil
} }
registry.RecordProcessingStageChange("scanning", ps.eventName) if ps.config.EventName == "" {
// fail centralizes the log/registry bookkeeping for every error return
// site below instead of repeating it at each one.
fail := func(err error) error {
procLog.Error("processing failed", logging.KeyEvent, ps.eventName, logging.KeyError, err.Error())
registry.RecordProcessingStageChange("failed", ps.eventName)
registry.RecordProcessingError(err)
return err
}
if ps.eventName == "" {
events, err := ps.GetEventDirs() events, err := ps.GetEventDirs()
if err != nil { if err != nil {
return fail(fmt.Errorf("failed to get event directories: %w", err)) return fmt.Errorf("Failed to get event directories: %w", err)
} }
if len(events) == 0 { if len(events) == 0 {
return fail(fmt.Errorf("no events found")) return fmt.Errorf("No events found")
} }
if len(events) > 1 { if len(events) > 1 {
fmt.Println("Multiple events found, please select one:") fmt.Println("Multiple events found, please select one:")
@ -132,23 +96,21 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
input = strings.TrimSpace(input) input = strings.TrimSpace(input)
index, err := strconv.Atoi(input) index, err := strconv.Atoi(input)
if err != nil { if err != nil {
return fail(fmt.Errorf("failed to parse input: %w", err)) return fmt.Errorf("Failed to parse input: %w", err)
} }
if index < 1 || index > len(events) { if index < 1 || index > len(events) {
return fail(fmt.Errorf("invalid input")) return fmt.Errorf("Invalid input")
} }
ps.eventName = events[index-1] ps.config.EventName = events[index-1]
} else { } else {
ps.eventName = events[0] ps.config.EventName = events[0]
} }
registry.RecordProcessingStageChange("scanning", ps.eventName)
} }
procLog.Info("processing started", logging.KeyEvent, ps.eventName)
//Get all present resolutions //Get all present resolutions
dirs, err := ps.GetResolutions() dirs, err := ps.GetResolutions()
if err != nil { if err != nil {
return fail(fmt.Errorf("Failed to get resolutions: %w", err)) return fmt.Errorf("Failed to get resolutions: %w", err)
} }
//Spawn a worker per resolution //Spawn a worker per resolution
@ -166,37 +128,24 @@ func (ps *ProcessingService) Start(ctx context.Context) error {
segments, err := ps.AggregateSegmentInfo(ch) segments, err := ps.AggregateSegmentInfo(ch)
if err != nil { if err != nil {
return fail(fmt.Errorf("Failed to aggregate segment info: %w", err)) return fmt.Errorf("Failed to aggregate segment info: %w", err)
}
registry.RecordProcessingSegmentsFound(len(segments))
aggFile, err := ps.WriteConcatFile(segments)
if err != nil {
return fail(fmt.Errorf("Failed to write concat file: %w", err))
} }
// Feed info to ffmpeg to stitch files together ps.WriteConcatFile(segments)
outPath := ps.config.GetProcessOutputPath(ps.eventName)
if err := utils.EnsureDir(outPath); err != nil {
return fail(fmt.Errorf("failed to create output directory: %w", err))
}
registry.RecordProcessingStageChange("running", ps.eventName) //Feed info to ffmpeg to stitch files together
concatErr := ps.RunFFmpeg(aggFile, outPath) concatErr := ps.RunFFmpeg(ps.config.SourcePath, ps.config.DestinationPath)
if concatErr != nil { if concatErr != nil {
return fail(concatErr) return concatErr
} }
procLog.Info("processing completed", logging.KeyEvent, ps.eventName, "segments", len(segments))
registry.RecordProcessingStageChange("completed", ps.eventName)
return nil return nil
} }
func (ps *ProcessingService) GetResolutions() ([]string, error) { func (ps *ProcessingService) GetResolutions() ([]string, error) {
eventPath := ps.nasEventPath() dirs, err := os.ReadDir(ps.config.SourcePath)
dirs, err := os.ReadDir(eventPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err) return nil, fmt.Errorf("Failed to read source directory: %w", err)
} }
re := regexp.MustCompile(`^\d+p$`) re := regexp.MustCompile(`^\d+p$`)
@ -214,19 +163,15 @@ func (ps *ProcessingService) GetResolutions() ([]string, error) {
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) { func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
defer wg.Done() defer wg.Done()
resolutionPath := utils.SafeJoin(ps.nasEventPath(), resolution) files, err := os.ReadDir(ps.config.SourcePath + "/" + resolution)
files, err := os.ReadDir(resolutionPath)
if err != nil { if err != nil {
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err) log.Printf("Failed to read resolution directory: %v", err)
return return
} }
for _, file := range files { for _, file := range files {
if !file.IsDir() { if !file.IsDir() {
if !strings.HasSuffix(strings.ToLower(file.Name()), ".ts") { no, err := strconv.Atoi(file.Name()[7:11])
continue
}
no, err := strconv.Atoi(file.Name()[6:10])
if err != nil { if err != nil {
log.Printf("Failed to parse segment number: %v", err) log.Printf("Failed to parse segment number: %v", err)
continue continue
@ -243,14 +188,20 @@ func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) { func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
segmentMap := make(map[int]SegmentInfo) segmentMap := make(map[int]SegmentInfo)
// Each sequence number arrives once per downloaded variant; keep the rank := map[string]int{
// highest-resolution copy of each, since that is the one worth stitching "1080p": 1,
// into the final video. "720p": 2,
"540p": 3,
"480p": 4,
"450p": 5,
"360p": 6,
"270p": 7,
"240p": 8,
}
for segment := range ch { for segment := range ch {
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
current, exists := segmentMap[segment.SeqNo] current, exists := segmentMap[segment.SeqNo]
if !exists || resolutionHeight(segment.Resolution) > resolutionHeight(current.Resolution) { if !exists || rank[segment.Resolution] > rank[current.Resolution] {
segmentMap[segment.SeqNo] = segment segmentMap[segment.SeqNo] = segment
} }
} }
@ -258,134 +209,30 @@ func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[in
return segmentMap, nil return segmentMap, nil
} }
// resolutionHeight converts a resolution label such as "1080p" into the pixel func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) error {
// height it stands for, so variants can be ranked by quality. f, err := os.Create(ps.config.DestinationPath + "/concat.txt")
//
// The height is parsed rather than looked up in a table: labels come from the
// playlist's own RESOLUTION attribute, so a stream can legitimately offer
// 1440p or 2160p that no fixed table would anticipate. Anything not in
// <height>p form — "unknown", from a playlist with no variants — returns 0 and
// therefore loses to every recognised resolution.
func resolutionHeight(resolution string) int {
trimmed := strings.TrimSuffix(strings.ToLower(strings.TrimSpace(resolution)), "p")
height, err := strconv.Atoi(trimmed)
if err != nil || height < 0 {
return 0
}
return height
}
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
concatPath := ps.config.GetProcessOutputPath(ps.eventName)
if err := utils.EnsureDir(concatPath); err != nil {
return "", fmt.Errorf("failed to create directories for concat path: %w", err)
}
concatFilePath := utils.SafeJoin(concatPath, ps.eventName+".txt")
f, err := os.Create(concatFilePath)
if err != nil { if err != nil {
return "", fmt.Errorf("failed to create concat file: %w", err) return fmt.Errorf("Failed to create concat file: %w", err)
} }
defer f.Close() defer f.Close()
// Sort keys to preserve order for _, segment := range segmentMap {
keys := make([]int, 0, len(segmentMap)) _, err = f.WriteString("file '" + ps.config.SourcePath + "/" + segment.Resolution + "/" + segment.Name + "'\n")
for k := range segmentMap {
keys = append(keys, k)
}
sort.Ints(keys)
for _, seq := range keys {
segment := segmentMap[seq]
filePath := utils.SafeJoin(ps.nasEventPath(), segment.Resolution, segment.Name)
line := fmt.Sprintf("file '%s'\n", filePath)
if _, err := f.WriteString(line); err != nil {
return "", fmt.Errorf("failed to write to concat file: %w", err)
}
}
return concatFilePath, nil
}
func (ps *ProcessingService) getFFmpegPath() (string, error) {
// First try the configured path
configuredPath := ps.config.Processing.FFmpegPath
if configuredPath != "" {
// Check if it's just the command name or a full path
if filepath.IsAbs(configuredPath) {
return configuredPath, nil
}
// Try to find it in PATH
if fullPath, err := exec.LookPath(configuredPath); err == nil {
return fullPath, nil
}
}
// Fallback: try local bin directory
var baseDir string
exePath, err := os.Executable()
if err == nil {
baseDir = filepath.Dir(exePath)
} else {
baseDir, err = os.Getwd()
if err != nil { if err != nil {
return "", err return fmt.Errorf("Failed to write to concat file: %w", err)
} }
} }
ffmpeg := utils.SafeJoin(baseDir, "bin", "ffmpeg") return nil
if runtime.GOOS == "windows" {
ffmpeg += ".exe"
}
if utils.PathExists(ffmpeg) {
return ffmpeg, nil
}
// Try current working directory
cwd, err := os.Getwd()
if err != nil {
return "", err
}
ffmpeg = utils.SafeJoin(cwd, "bin", "ffmpeg")
if runtime.GOOS == "windows" {
ffmpeg += ".exe"
}
if utils.PathExists(ffmpeg) {
return ffmpeg, nil
}
return "", fmt.Errorf("FFmpeg not found. Please install FFmpeg or set FFMPEG_PATH environment variable")
} }
func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error { func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error {
fmt.Println("Running ffmpeg...") cmd := exec.Command("ffmpeg", "-f", "concat", "-safe", "0", "-i", inputPath, "-c", "copy", outputPath)
fileOutPath := utils.SafeJoin(outputPath, ps.eventName+".mp4")
fmt.Println("Input path:", inputPath)
fmt.Println("Output path:", fileOutPath)
logging.MustGetFactory().Processing.Info("running ffmpeg", "input", inputPath, "output", fileOutPath)
path, err := ps.getFFmpegPath()
if err != nil {
logging.MustGetFactory().Processing.Error("failed to find ffmpeg", logging.KeyError, err.Error())
return fmt.Errorf("failed to find FFmpeg: %w", err)
}
cmd := exec.Command(path, "-f", "concat", "-safe", "0", "-i", inputPath, "-c", "copy", fileOutPath)
cmd.Stdout = os.Stdout cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr cmd.Stderr = os.Stderr
err := cmd.Run()
if err := cmd.Run(); err != nil { if err != nil {
logging.MustGetFactory().Processing.Error("ffmpeg run failed", logging.KeyError, err.Error()) return fmt.Errorf("Failed to run ffmpeg: %w", err)
return fmt.Errorf("failed to run ffmpeg: %w", err)
} }
fmt.Println("FFmpeg completed successfully")
logging.MustGetFactory().Processing.Info("ffmpeg completed successfully", "output", fileOutPath)
return nil return nil
} }

View File

@ -1,472 +0,0 @@
package processing
import (
"context"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/nas"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
)
// connectedNAS attaches a NAS service to a local directory. Reaching the NAS
// is platform-specific, but an already-mounted or plain directory resolves the
// same way everywhere, which is what makes this usable as a test double.
func connectedNAS(t *testing.T, path string) *nas.NASService {
t.Helper()
svc, err := nas.New(nas.NASConfig{Path: path, Timeout: 5 * time.Second})
if err != nil {
t.Fatalf("Failed to create NAS service for %s: %v", path, err)
}
if err := svc.Connect(context.Background()); err != nil {
t.Fatalf("Failed to connect NAS service to %s: %v", path, err)
}
return svc
}
func createTestConfig(tempDir string) *config.Config {
return &config.Config{
Core: config.CoreConfig{
WorkerCount: 2,
RefreshDelay: 1 * time.Second,
},
NAS: config.NASConfig{
OutputPath: filepath.Join(tempDir, "nas"),
Username: "testuser",
Password: "testpass",
Timeout: 10 * time.Second,
RetryLimit: 2,
EnableTransfer: false, // Disable to avoid NAS connection
},
Processing: config.ProcessingConfig{
Enabled: true,
AutoProcess: true,
WorkerCount: 1,
FFmpegPath: "echo", // Use echo command for testing
},
Paths: config.PathsConfig{
LocalOutput: filepath.Join(tempDir, "data"),
ProcessOutput: filepath.Join(tempDir, "out"),
ManifestDir: filepath.Join(tempDir, "data"),
PersistenceFile: filepath.Join(tempDir, "queue.json"),
},
}
}
func TestNewProcessingService_Success(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
cfg.NAS.EnableTransfer = false // Disable NAS to avoid connection
// We can't test actual NAS connection, so we'll skip the constructor test
// that requires NAS connectivity. Instead, test the configuration handling.
if cfg.Processing.FFmpegPath != "echo" {
t.Errorf("Expected FFmpegPath='echo', got '%s'", cfg.Processing.FFmpegPath)
}
}
func TestNewProcessingService_NilConfig(t *testing.T) {
_, err := NewProcessingService("test-event", nil)
if err == nil {
t.Error("Expected error for nil config")
}
if !strings.Contains(err.Error(), "configuration is required") {
t.Errorf("Expected 'configuration is required' error, got: %v", err)
}
}
func TestProcessingService_GetEventDirs(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
// Create mock NAS directory structure
nasDir := cfg.NAS.OutputPath
os.MkdirAll(filepath.Join(nasDir, "event1"), 0755)
os.MkdirAll(filepath.Join(nasDir, "event2"), 0755)
os.MkdirAll(filepath.Join(nasDir, "event3"), 0755)
// Create a file (should be ignored)
os.WriteFile(filepath.Join(nasDir, "not_a_dir.txt"), []byte("test"), 0644)
ps := &ProcessingService{
config: cfg,
eventName: "", // Empty to test directory discovery
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
dirs, err := ps.GetEventDirs()
if err != nil {
t.Fatalf("GetEventDirs() failed: %v", err)
}
if len(dirs) != 3 {
t.Errorf("Expected 3 event directories, got %d", len(dirs))
}
expectedDirs := []string{"event1", "event2", "event3"}
for _, expected := range expectedDirs {
found := false
for _, actual := range dirs {
if actual == expected {
found = true
break
}
}
if !found {
t.Errorf("Expected to find directory '%s' in results: %v", expected, dirs)
}
}
}
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
// Scoped to a temp dir rather than /tmp directly: attaching the NAS
// service creates the destination, and tests should not leave one behind.
cfg := createTestConfig(t.TempDir())
eventName := "specific-event"
ps := &ProcessingService{
config: cfg,
eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
dirs, err := ps.GetEventDirs()
if err != nil {
t.Fatalf("GetEventDirs() failed: %v", err)
}
if len(dirs) != 1 {
t.Errorf("Expected 1 directory, got %d", len(dirs))
}
if dirs[0] != eventName {
t.Errorf("Expected directory '%s', got '%s'", eventName, dirs[0])
}
}
func TestProcessingService_GetResolutions(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
eventName := "test-event"
// Create mock event directory with quality subdirectories
eventPath := filepath.Join(cfg.NAS.OutputPath, eventName)
os.MkdirAll(filepath.Join(eventPath, "1080p"), 0755)
os.MkdirAll(filepath.Join(eventPath, "720p"), 0755)
os.MkdirAll(filepath.Join(eventPath, "480p"), 0755)
os.MkdirAll(filepath.Join(eventPath, "not_resolution"), 0755) // Should be ignored
os.WriteFile(filepath.Join(eventPath, "file.txt"), []byte("test"), 0644) // Should be ignored
ps := &ProcessingService{
config: cfg,
eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
resolutions, err := ps.GetResolutions()
if err != nil {
t.Fatalf("GetResolutions() failed: %v", err)
}
expectedResolutions := []string{"1080p", "720p", "480p"}
if len(resolutions) != len(expectedResolutions) {
t.Errorf("Expected %d resolutions, got %d: %v", len(expectedResolutions), len(resolutions), resolutions)
}
for _, expected := range expectedResolutions {
found := false
for _, actual := range resolutions {
if actual == expected {
found = true
break
}
}
if !found {
t.Errorf("Expected to find resolution '%s' in results: %v", expected, resolutions)
}
}
}
func TestProcessingService_AggregateSegmentInfo(t *testing.T) {
ps := &ProcessingService{}
// Create test channel with segments
ch := make(chan SegmentInfo, 5)
// Add segments with different qualities for same sequence
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"}
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "1080p"} // Higher quality, should win
ch <- SegmentInfo{Name: "seg_1002.ts", SeqNo: 1002, Resolution: "480p"}
ch <- SegmentInfo{Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"}
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "540p"} // Lower than 1080p, should not replace
close(ch)
segmentMap, err := ps.AggregateSegmentInfo(ch)
if err != nil {
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
}
// Should have 3 unique sequence numbers
if len(segmentMap) != 3 {
t.Errorf("Expected 3 unique segments, got %d", len(segmentMap))
}
// Check sequence 1001 has the highest quality (1080p)
seg1001, exists := segmentMap[1001]
if !exists {
t.Fatal("Segment 1001 should exist")
}
if seg1001.Resolution != "1080p" {
t.Errorf("Expected segment 1001 to have resolution '1080p', got '%s'", seg1001.Resolution)
}
// Check sequence 1002 has 480p
seg1002, exists := segmentMap[1002]
if !exists {
t.Fatal("Segment 1002 should exist")
}
if seg1002.Resolution != "480p" {
t.Errorf("Expected segment 1002 to have resolution '480p', got '%s'", seg1002.Resolution)
}
// Check sequence 1003 has 1080p
seg1003, exists := segmentMap[1003]
if !exists {
t.Fatal("Segment 1003 should exist")
}
if seg1003.Resolution != "1080p" {
t.Errorf("Expected segment 1003 to have resolution '1080p', got '%s'", seg1003.Resolution)
}
}
func TestProcessingService_WriteConcatFile(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
eventName := "test-event"
ps := &ProcessingService{
config: cfg,
eventName: eventName,
nas: connectedNAS(t, cfg.NAS.OutputPath),
}
// Create test segment map
segmentMap := map[int]SegmentInfo{
1003: {Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"},
1001: {Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"},
1002: {Name: "seg_1002.ts", SeqNo: 1002, Resolution: "1080p"},
}
concatFilePath, err := ps.WriteConcatFile(segmentMap)
if err != nil {
t.Fatalf("WriteConcatFile() failed: %v", err)
}
// Verify file was created
if _, err := os.Stat(concatFilePath); os.IsNotExist(err) {
t.Fatalf("Concat file was not created: %s", concatFilePath)
}
// Read and verify content
content, err := os.ReadFile(concatFilePath)
if err != nil {
t.Fatalf("Failed to read concat file: %v", err)
}
contentStr := string(content)
lines := strings.Split(strings.TrimSpace(contentStr), "\n")
if len(lines) != 3 {
t.Errorf("Expected 3 lines in concat file, got %d", len(lines))
}
// Verify segments are sorted by sequence number
expectedOrder := []string{"seg_1001.ts", "seg_1002.ts", "seg_1003.ts"}
for i, line := range lines {
if !strings.Contains(line, expectedOrder[i]) {
t.Errorf("Line %d should contain '%s', got: %s", i, expectedOrder[i], line)
}
if !strings.HasPrefix(line, "file '") {
t.Errorf("Line %d should start with 'file ', got: %s", i, line)
}
}
}
func TestProcessingService_getFFmpegPath(t *testing.T) {
cfg := createTestConfig("/tmp")
tests := []struct {
name string
ffmpegPath string
shouldFind bool
expectedError string
}{
{
name: "echo command (should be found in PATH)",
ffmpegPath: "echo",
shouldFind: true,
},
{
name: "absolute path test",
ffmpegPath: func() string {
if runtime.GOOS == "windows" {
return "C:\\Windows\\System32\\cmd.exe"
}
return "/bin/echo"
}(),
shouldFind: true,
},
{
name: "nonexistent command",
ffmpegPath: "nonexistent_ffmpeg_command_12345",
shouldFind: false,
expectedError: "FFmpeg not found",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
testCfg := *cfg
testCfg.Processing.FFmpegPath = tt.ffmpegPath
ps := &ProcessingService{
config: &testCfg,
eventName: "test",
}
path, err := ps.getFFmpegPath()
if tt.shouldFind {
if err != nil {
t.Errorf("Expected to find FFmpeg, but got error: %v", err)
}
if path == "" {
t.Error("Expected non-empty path")
}
} else {
if err == nil {
t.Error("Expected error for nonexistent FFmpeg")
}
if tt.expectedError != "" && !strings.Contains(err.Error(), tt.expectedError) {
t.Errorf("Expected error containing '%s', got: %v", tt.expectedError, err)
}
}
})
}
}
func TestSegmentInfo_Structure(t *testing.T) {
segment := SegmentInfo{
Name: "test_segment.ts",
SeqNo: 1001,
Resolution: "1080p",
}
if segment.Name != "test_segment.ts" {
t.Errorf("Expected Name='test_segment.ts', got '%s'", segment.Name)
}
if segment.SeqNo != 1001 {
t.Errorf("Expected SeqNo=1001, got %d", segment.SeqNo)
}
if segment.Resolution != "1080p" {
t.Errorf("Expected Resolution='1080p', got '%s'", segment.Resolution)
}
}
func TestProcessJob_Structure(t *testing.T) {
job := ProcessJob{
EventName: "test-event",
}
if job.EventName != "test-event" {
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
}
}
func TestResolutionHeight(t *testing.T) {
cases := map[string]int{
"1080p": 1080,
"720p": 720,
"2160p": 2160, // beyond anything a fixed quality table would list
"1440p": 1440,
"240p": 240,
// Not in <height>p form: must rank below every real resolution rather
// than accidentally outranking them.
"unknown": 0,
"": 0,
"p": 0,
"-720p": 0,
}
for label, want := range cases {
if got := resolutionHeight(label); got != want {
t.Errorf("resolutionHeight(%q) = %d, want %d", label, got, want)
}
}
}
// TestAggregateSegmentInfoPrefersHighestResolution covers the variants a
// hardcoded rank table could not express: resolutions above 1080p, and the
// "unknown" label produced for a playlist with no variants.
func TestAggregateSegmentInfoPrefersHighestResolution(t *testing.T) {
cases := []struct {
name string
arriving []string
want string
}{
{"1080p beats 720p regardless of order", []string{"720p", "1080p"}, "1080p"},
{"1080p beats 720p when it arrives first", []string{"1080p", "720p"}, "1080p"},
{"4k beats 1080p", []string{"1080p", "2160p"}, "2160p"},
{"1440p beats 1080p", []string{"1080p", "1440p"}, "1440p"},
{"a known resolution beats unknown", []string{"unknown", "480p"}, "480p"},
{"unknown does not displace a known resolution", []string{"1080p", "unknown"}, "1080p"},
{"unknown alone is still kept", []string{"unknown"}, "unknown"},
{"lowest only", []string{"240p"}, "240p"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ps := &ProcessingService{}
ch := make(chan SegmentInfo, len(tc.arriving))
for _, res := range tc.arriving {
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: res}
}
close(ch)
segments, err := ps.AggregateSegmentInfo(ch)
if err != nil {
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
}
if len(segments) != 1 {
t.Fatalf("expected a single sequence number, got %d", len(segments))
}
if got := segments[1001].Resolution; got != tc.want {
t.Errorf("kept %q, want %q (arrival order: %v)", got, tc.want, tc.arriving)
}
})
}
}

View File

@ -3,3 +3,11 @@ package processing
type ProcessJob struct { type ProcessJob struct {
EventName string EventName string
} }
type ProcessConfig struct {
WorkerCount int
SourcePath string
DestinationPath string
Enabled bool
EventName string
}

View File

@ -2,20 +2,13 @@ package transfer
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/logging"
"os" "os"
"sync" "sync"
"time" "time"
) )
// errRetained reports that a file is still inside its retention window. It is
// not a failure: the file simply has to be reconsidered on a later pass rather
// than being discarded from the pending list.
var errRetained = errors.New("file is within the retention period")
type CleanupService struct { type CleanupService struct {
config CleanupConfig config CleanupConfig
pendingFiles []string pendingFiles []string
@ -48,7 +41,6 @@ func (cs *CleanupService) Start(ctx context.Context) error {
} }
log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize) log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize)
logging.MustGetFactory().Transfer.Info("cleanup service started", "retention", cs.config.RetentionPeriod.String(), "batch_size", cs.config.BatchSize)
ticker := time.NewTicker(cs.config.CheckInterval) ticker := time.NewTicker(cs.config.CheckInterval)
defer ticker.Stop() defer ticker.Stop()
@ -88,62 +80,36 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
log.Printf("Processing %d files for cleanup", len(batch)) log.Printf("Processing %d files for cleanup", len(batch))
var cleanedCount int var cleanedCount int
var failures []error var errors []error
// requeue collects files this pass must not drop: ones still inside their
// retention window, and — if the context is cancelled mid-batch — the
// remainder that was never looked at. They were already removed from
// pendingFiles above, so without this they would never be cleaned at all.
var requeue []string
for i, filePath := range batch { for _, filePath := range batch {
select { select {
case <-ctx.Done(): case <-ctx.Done():
requeue = append(requeue, batch[i:]...)
cs.requeue(requeue)
return ctx.Err() return ctx.Err()
default: default:
} }
err := cs.cleanupFile(filePath) if err := cs.cleanupFile(filePath); err != nil {
switch { errors = append(errors, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
case errors.Is(err, errRetained): } else {
requeue = append(requeue, filePath)
case err != nil:
failures = append(failures, fmt.Errorf("Failed to cleanup file %s: %w", filePath, err))
default:
cleanedCount++ cleanedCount++
} }
} }
cs.requeue(requeue) log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
log.Printf("Cleanup batch completed (cleaned: %d, retained: %d, errors: %d)", cleanedCount, len(requeue), len(failures)) if len(errors) > 0 {
logging.MustGetFactory().Transfer.Info("cleanup batch completed", for i, err := range errors {
"cleaned", cleanedCount, "retained", len(requeue), "errors", len(failures))
if len(failures) > 0 {
for i, err := range failures {
if i >= 3 { if i >= 3 {
log.Printf("... and %d more errors", len(failures)-3) log.Printf("... and %d more errors", len(errors)-3)
break break
} }
log.Printf("Error: %v", err) log.Printf("Error: %v", err)
logging.MustGetFactory().Transfer.Warn("cleanup error", logging.KeyError, err.Error())
} }
} }
return nil return nil
}
// requeue returns files to the pending list for a later pass.
func (cs *CleanupService) requeue(files []string) {
if len(files) == 0 {
return
}
cs.mu.Lock()
defer cs.mu.Unlock()
cs.pendingFiles = append(cs.pendingFiles, files...)
} }
func (cs *CleanupService) cleanupFile(filePath string) error { func (cs *CleanupService) cleanupFile(filePath string) error {
@ -158,7 +124,7 @@ func (cs *CleanupService) cleanupFile(filePath string) error {
if cs.config.RetentionPeriod > 0 { if cs.config.RetentionPeriod > 0 {
if time.Since(info.ModTime()) < cs.config.RetentionPeriod { if time.Since(info.ModTime()) < cs.config.RetentionPeriod {
log.Printf("File too new to cleanup: %s", filePath) log.Printf("File too new to cleanup: %s", filePath)
return errRetained return nil
} }
} }
@ -192,18 +158,6 @@ func (cs *CleanupService) ForceCleanupAll(ctx context.Context) error {
return err return err
} }
// Stop when a pass makes no progress. Files still inside their
// retention window are requeued by design, so requiring the list to
// reach zero would spin here until the shutdown timeout expired.
cs.mu.Lock()
remaining := len(cs.pendingFiles)
cs.mu.Unlock()
if remaining >= pendingCount {
log.Printf("Force cleanup stopping with %d file(s) still pending (within retention period)", remaining)
break
}
select { select {
case <-ctx.Done(): case <-ctx.Done():
return ctx.Err() return ctx.Err()

View File

@ -1,190 +0,0 @@
package transfer
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
)
func writeAgedFile(t *testing.T, dir, name string, age time.Duration) string {
t.Helper()
path := filepath.Join(dir, name)
if err := os.WriteFile(path, []byte("segment"), 0644); err != nil {
t.Fatalf("failed to write %s: %v", path, err)
}
modTime := time.Now().Add(-age)
if err := os.Chtimes(path, modTime, modTime); err != nil {
t.Fatalf("failed to age %s: %v", path, err)
}
return path
}
func TestCleanupRemovesTransferredFiles(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
path := writeAgedFile(t, dir, "seg_1001.ts", 0)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Errorf("file should have been removed, stat err = %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("pending count = %d, want 0", got)
}
}
// TestCleanupRetainsThenRemoves is the regression test for a file inside its
// retention window being dropped from the pending list: it was reported as
// cleaned, never deleted, and never reconsidered, so local files accumulated
// forever whenever RetainHours was set.
func TestCleanupRetainsThenRemoves(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{
Enabled: true,
RetentionPeriod: time.Hour,
BatchSize: 10,
CheckInterval: time.Second,
})
fresh := writeAgedFile(t, dir, "fresh.ts", time.Minute)
old := writeAgedFile(t, dir, "old.ts", 2*time.Hour)
for _, p := range []string{fresh, old} {
if err := cs.ScheduleCleanup(p); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(old); !os.IsNotExist(err) {
t.Errorf("a file past its retention period should be removed, stat err = %v", err)
}
if _, err := os.Stat(fresh); err != nil {
t.Errorf("a file inside its retention period should survive: %v", err)
}
if got := cs.GetPendingCount(); got != 1 {
t.Fatalf("the retained file must stay queued for a later pass, pending = %d", got)
}
// Once it ages past the window, the next pass must actually remove it.
past := time.Now().Add(-2 * time.Hour)
if err := os.Chtimes(fresh, past, past); err != nil {
t.Fatalf("failed to age file: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(fresh); !os.IsNotExist(err) {
t.Errorf("file should have been removed once past retention, stat err = %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("pending count = %d, want 0", got)
}
}
// TestForceCleanupAllTerminatesWithRetainedFiles guards shutdown: retained
// files are requeued by design, so a loop demanding an empty list would spin
// until the shutdown timeout expired.
func TestForceCleanupAllTerminatesWithRetainedFiles(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{
Enabled: true,
RetentionPeriod: time.Hour,
BatchSize: 10,
CheckInterval: time.Second,
})
if err := cs.ScheduleCleanup(writeAgedFile(t, dir, "fresh.ts", time.Minute)); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
done := make(chan error, 1)
go func() { done <- cs.ForceCleanupAll(context.Background()) }()
select {
case err := <-done:
if err != nil {
t.Fatalf("ForceCleanupAll failed: %v", err)
}
case <-time.After(3 * time.Second):
t.Fatal("ForceCleanupAll did not terminate with a retained file pending")
}
if got := cs.GetPendingCount(); got != 1 {
t.Errorf("the retained file should still be pending, got %d", got)
}
}
func TestCleanupMissingFileIsNotAnError(t *testing.T) {
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
// Already gone — a file transferred and removed by something else.
if err := cs.ScheduleCleanup(filepath.Join(t.TempDir(), "gone.ts")); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("a missing file should not be requeued, pending = %d", got)
}
}
func TestCleanupDisabledSchedulesNothing(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: false, BatchSize: 10, CheckInterval: time.Second})
path := writeAgedFile(t, dir, "seg.ts", time.Hour)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
if got := cs.GetPendingCount(); got != 0 {
t.Errorf("cleanup is disabled, nothing should be queued, got %d", got)
}
if err := cs.ExecuteCleanup(context.Background()); err != nil {
t.Fatalf("ExecuteCleanup failed: %v", err)
}
if _, err := os.Stat(path); err != nil {
t.Errorf("file must survive when cleanup is disabled: %v", err)
}
}
// TestCleanupRequeuesOnCancellation covers shutdown arriving mid-batch: the
// files already pulled off the pending list must go back, not vanish.
func TestCleanupRequeuesOnCancellation(t *testing.T) {
dir := t.TempDir()
cs := NewCleanupService(CleanupConfig{Enabled: true, BatchSize: 10, CheckInterval: time.Second})
for i := 0; i < 5; i++ {
path := writeAgedFile(t, dir, fmt.Sprintf("seg_%d.ts", i), 0)
if err := cs.ScheduleCleanup(path); err != nil {
t.Fatalf("ScheduleCleanup failed: %v", err)
}
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := cs.ExecuteCleanup(ctx); err == nil {
t.Fatal("expected the cancelled context to be reported")
}
if got := cs.GetPendingCount(); got != 5 {
t.Errorf("all files should be requeued after cancellation, pending = %d, want 5", got)
}
}

View File

@ -4,14 +4,13 @@ import (
"context" "context"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas" "m3u8-downloader/pkg/nas"
"os" "os"
"path/filepath" "path/filepath"
) )
func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error { func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
destPath := nt.Join(item.DestinationPath) destPath := filepath.Join(nt.Config.Path, item.DestinationPath)
destDir := filepath.Dir(destPath) destDir := filepath.Dir(destPath)
if err := nt.EnsureDirectoryExists(destDir); err != nil { if err := nt.EnsureDirectoryExists(destDir); err != nil {
@ -33,7 +32,6 @@ func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) e
} }
log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath) log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath)
logging.MustGetFactory().Transfer.Debug("nas copy completed", logging.KeyFile, item.SourcePath, "dest", destPath)
return nil return nil
} }

View File

@ -6,8 +6,6 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas" "m3u8-downloader/pkg/nas"
"os" "os"
"sync" "sync"
@ -76,10 +74,7 @@ func (tq *TransferQueue) Add(item TransferItem) error {
defer tq.mu.Unlock() defer tq.mu.Unlock()
if tq.items.Len() >= tq.config.MaxQueueSize { if tq.items.Len() >= tq.config.MaxQueueSize {
err := fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize) return fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
logging.MustGetFactory().Transfer.Error("queue full, dropping file", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
return err
} }
heap.Push(tq.items, &item) heap.Push(tq.items, &item)
@ -169,20 +164,13 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
return return
} }
maxRetries := tq.config.RetryLimit maxRetries := 3
if maxRetries < 1 {
maxRetries = 1
}
for attempt := 1; attempt <= maxRetries; attempt++ { for attempt := 1; attempt <= maxRetries; attempt++ {
// Back off before a retry, never before the first attempt. The guard if attempt > 0 {
// was `attempt > 0`, which is always true, so every single transfer —
// including healthy ones — paid a one-second delay before it began,
// throttling the whole pipeline to the worker count per second.
if attempt > 1 {
item.Status = StatusRetrying item.Status = StatusRetrying
backoff := time.Duration(attempt*attempt) * time.Second backoff := time.Duration(attempt*attempt) * time.Second
log.Printf("Backing off for %s before retrying (attempt %d/%d)", backoff, attempt, maxRetries) log.Printf("Backing off for %d seconds before retrying (attempt %d/%d)", backoff, attempt, maxRetries)
select { select {
case <-time.After(backoff): case <-time.After(backoff):
@ -199,11 +187,9 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
if tq.cleanup != nil { if tq.cleanup != nil {
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil { if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
log.Printf("Failed to add file to cleanup list: %v", err) log.Printf("Failed to add file to cleanup list: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to schedule cleanup", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
} }
} }
log.Printf("File transfer completed: %s", item.SourcePath) log.Printf("File transfer completed: %s", item.SourcePath)
logging.MustGetFactory().Transfer.Info("file transfer completed", logging.KeyFile, item.SourcePath, logging.KeyBytes, item.FileSize)
return return
} }
@ -211,14 +197,11 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
item.RetryCount++ item.RetryCount++
log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err) log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err)
logging.MustGetFactory().Transfer.Warn("file transfer attempt failed", logging.KeyFile, item.SourcePath, logging.KeyAttempt, item.RetryCount, logging.KeyError, err.Error())
if attempt == maxRetries { if attempt == maxRetries {
item.Status = StatusFailed item.Status = StatusFailed
tq.stats.IncrementFailed() tq.stats.IncrementFailed()
log.Printf("Transfer permanently failed for file: %s", item.SourcePath) log.Printf("Transfer permanently failed for file: %s", item.SourcePath)
logging.MustGetFactory().Transfer.Error("transfer permanently failed", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
return return
} }
} }
@ -242,12 +225,10 @@ func (tq *TransferQueue) SaveState() error {
"timestamp": time.Now(), "timestamp": time.Now(),
}, "", " ") }, "", " ")
if err != nil { if err != nil {
logging.MustGetFactory().Transfer.Error("failed to marshal queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to marshal queue state: %w", err) return fmt.Errorf("Failed to marshal queue state: %w", err)
} }
if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil { if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil {
logging.MustGetFactory().Transfer.Error("failed to save queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to save queue state: %w", err) return fmt.Errorf("Failed to save queue state: %w", err)
} }
@ -260,7 +241,6 @@ func (tq *TransferQueue) LoadState() error {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return nil return nil
} }
logging.MustGetFactory().Transfer.Error("failed to load queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to load queue state: %w", err) return fmt.Errorf("Failed to load queue state: %w", err)
} }

View File

@ -5,10 +5,7 @@ import (
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
nas2 "m3u8-downloader/pkg/nas" nas2 "m3u8-downloader/pkg/nas"
"m3u8-downloader/pkg/utils"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@ -25,64 +22,47 @@ type TransferService struct {
} }
func NewTrasferService(outputDir string, eventName string) (*TransferService, error) { func NewTrasferService(outputDir string, eventName string) (*TransferService, error) {
cfg := constants.MustGetConfig()
nasConfig := nas2.NASConfig{ nasConfig := nas2.NASConfig{
Path: outputDir, Path: outputDir,
Username: cfg.NAS.Username, Username: constants.NASUsername,
Password: cfg.NAS.Password, Password: constants.NASPassword,
Timeout: cfg.NAS.Timeout, Timeout: constants.TransferTimeout,
RetryLimit: cfg.NAS.RetryLimit, RetryLimit: constants.TransferRetryLimit,
VerifySize: true, VerifySize: true,
AutoMount: cfg.NAS.AutoMount,
MountPoint: cfg.NAS.MountPoint,
}
nas, err := nas2.New(nasConfig)
if err != nil {
logging.MustGetFactory().Transfer.Error("invalid NAS destination", logging.KeyError, err.Error())
return nil, fmt.Errorf("invalid NAS destination: %w", err)
} }
nas := nas2.NewNASService(nasConfig)
connectCtx, cancelConnect := context.WithTimeout(context.Background(), cfg.NAS.Timeout) if err := nas.TestConnection(); err != nil {
defer cancelConnect() return nil, fmt.Errorf("Failed to connect to NAS: %w", err)
if err := nas.Connect(connectCtx); err != nil {
logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
} }
logging.MustGetFactory().Transfer.Info("connected to NAS",
"path", nas.LocalPath(), "configured", outputDir, "how", strings.Join(nas.ConnectionInfo(), "; "))
health.MustGetRegistry().RecordNASConnectivity(true)
cleanupConfig := CleanupConfig{ cleanupConfig := CleanupConfig{
Enabled: cfg.Cleanup.AfterTransfer, Enabled: constants.CleanupAfterTransfer,
RetentionPeriod: time.Duration(cfg.Cleanup.RetainHours) * time.Hour, RetentionPeriod: time.Duration(constants.RetainLocalHours) * time.Hour,
BatchSize: cfg.Cleanup.BatchSize, BatchSize: constants.CleanupBatchSize,
CheckInterval: cfg.Transfer.FileSettlingDelay, CheckInterval: 1 * time.Minute,
} }
cleanup := NewCleanupService(cleanupConfig) cleanup := NewCleanupService(cleanupConfig)
queueConfig := QueueConfig{ queueConfig := QueueConfig{
WorkerCount: cfg.Transfer.WorkerCount, WorkerCount: constants.TransferWorkerCount,
PersistencePath: cfg.Paths.PersistenceFile, PersistencePath: constants.PersistencePath,
MaxQueueSize: cfg.Transfer.QueueSize, MaxQueueSize: constants.TransferQueueSize,
BatchSize: cfg.Transfer.BatchSize, BatchSize: constants.BatchSize,
RetryLimit: cfg.Transfer.RetryLimit,
} }
queue := NewTransferQueue(queueConfig, nas, cleanup) queue := NewTransferQueue(queueConfig, nas, cleanup)
// Create local output directory if it doesn't exist // Create local output directory if it doesn't exist
localOutputPath := cfg.GetEventPath(eventName) localOutputPath := constants.LocalOutputDirPath + "/" + eventName
if err := utils.EnsureDir(localOutputPath); err != nil { if err := os.MkdirAll(localOutputPath, 0755); err != nil {
return nil, fmt.Errorf("failed to create local output directory: %w", err) return nil, fmt.Errorf("Failed to create local output directory: %w", err)
} }
watcher, err := NewFileWatcher(localOutputPath, eventName, queue, cfg.Transfer.FileSettlingDelay) watcher, err := NewFileWatcher(localOutputPath, queue)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create file watcher: %w", err) return nil, fmt.Errorf("Failed to create file watcher: %w", err)
} }
health.MustGetRegistry().AttachTransfer(queue.stats, queue.GetQueueSize, cleanup.GetPendingCount, nas.IsConnected)
return &TransferService{ return &TransferService{
watcher: watcher, watcher: watcher,
queue: queue, queue: queue,
@ -126,7 +106,6 @@ func (ts *TransferService) Start(ctx context.Context) error {
}() }()
log.Println("Transfer service started") log.Println("Transfer service started")
logging.MustGetFactory().Transfer.Info("transfer service started")
wg.Wait() wg.Wait()
return nil return nil
@ -146,9 +125,6 @@ func (ts *TransferService) reportStats(ctx context.Context) {
cleanupPending := ts.cleanup.GetPendingCount() cleanupPending := ts.cleanup.GetPendingCount()
log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending) log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending)
logging.MustGetFactory().Transfer.Info("transfer stats",
"added", added, "completed", completed, "failed", failed, "pending", pending,
logging.KeyBytes, bytes, "queue_size", queueSize, "cleanup_pending", cleanupPending)
} }
} }
} }
@ -165,21 +141,17 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
} }
// Disconnect from NAS // Disconnect from NAS
if err := ts.nas.Disconnect(ctx); err != nil { if err := ts.nas.Disconnect(); err != nil {
log.Printf("Warning: failed to disconnect from NAS: %v", err) log.Printf("Warning: failed to disconnect from NAS: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
} }
health.MustGetRegistry().RecordNASConnectivity(false)
log.Println("Transfer service shut down") log.Println("Transfer service shut down")
logging.MustGetFactory().Transfer.Info("transfer service shut down")
return nil return nil
} }
// QueueExistingFiles scans a directory for .ts files and queues them for transfer // QueueExistingFiles scans a directory for .ts files and queues them for transfer
func (ts *TransferService) QueueExistingFiles(localEventPath string) error { func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
cfg := constants.MustGetConfig()
log.Printf("Scanning for existing files in: %s", localEventPath) log.Printf("Scanning for existing files in: %s", localEventPath)
var fileCount, alreadyTransferred, scheduledForCleanup int var fileCount, alreadyTransferred, scheduledForCleanup int
@ -202,8 +174,6 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
relPath, err := filepath.Rel(localEventPath, path) relPath, err := filepath.Rel(localEventPath, path)
if err != nil { if err != nil {
log.Printf("Failed to get relative path for %s: %v", path, err) log.Printf("Failed to get relative path for %s: %v", path, err)
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, path, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(path, err)
return nil return nil
} }
@ -220,7 +190,7 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
alreadyTransferred++ alreadyTransferred++
// Schedule for cleanup if cleanup is enabled // Schedule for cleanup if cleanup is enabled
if cfg.Cleanup.AfterTransfer { if constants.CleanupAfterTransfer {
if err := ts.cleanup.ScheduleCleanup(path); err != nil { if err := ts.cleanup.ScheduleCleanup(path); err != nil {
log.Printf("Failed to schedule cleanup for already-transferred file %s: %v", path, err) log.Printf("Failed to schedule cleanup for already-transferred file %s: %v", path, err)
} else { } else {
@ -244,8 +214,6 @@ func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
// Add to queue // Add to queue
if err := ts.queue.Add(item); err != nil { if err := ts.queue.Add(item); err != nil {
log.Printf("Failed to queue file %s: %v", path, err) log.Printf("Failed to queue file %s: %v", path, err)
logging.MustGetFactory().Transfer.Error("failed to queue file", logging.KeyFile, path, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(path, err)
} else { } else {
log.Printf("Queued file: %s (%s, %d bytes)", path, resolution, info.Size()) log.Printf("Queued file: %s (%s, %d bytes)", path, resolution, info.Size())
fileCount++ fileCount++

View File

@ -49,9 +49,6 @@ type QueueConfig struct {
PersistencePath string PersistencePath string
MaxQueueSize int MaxQueueSize int
BatchSize int BatchSize int
// RetryLimit is the total number of attempts made per file before it is
// marked failed.
RetryLimit int
} }
type CleanupConfig struct { type CleanupConfig struct {

View File

@ -4,8 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/health" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"math/rand" "math/rand"
"os" "os"
"path/filepath" "path/filepath"
@ -17,12 +16,7 @@ import (
) )
type FileWatcher struct { type FileWatcher struct {
outputDir string outputDir string
// eventName namespaces the NAS destination. It is passed in rather than
// derived from outputDir so that an unnamed event resolves to the NAS root
// — matching where the downloader put the files — instead of picking up
// the local output directory's own name.
eventName string
queue *TransferQueue queue *TransferQueue
watcher *fsnotify.Watcher watcher *fsnotify.Watcher
settingDelay time.Duration settingDelay time.Duration
@ -30,17 +24,16 @@ type FileWatcher struct {
mu sync.Mutex mu sync.Mutex
} }
func NewFileWatcher(outputDir, eventName string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) { func NewFileWatcher(outputDir string, queue *TransferQueue) (*FileWatcher, error) {
watcher, err := fsnotify.NewWatcher() watcher, err := fsnotify.NewWatcher()
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &FileWatcher{ return &FileWatcher{
outputDir: outputDir, outputDir: outputDir,
eventName: eventName,
queue: queue, queue: queue,
watcher: watcher, watcher: watcher,
settingDelay: settlingDelay, settingDelay: constants.FileSettlingDelay,
pendingFiles: make(map[string]*time.Timer), pendingFiles: make(map[string]*time.Timer),
}, nil }, nil
} }
@ -53,13 +46,11 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
} }
log.Printf("Starting file watcher on %s", fw.outputDir) log.Printf("Starting file watcher on %s", fw.outputDir)
logging.MustGetFactory().Transfer.Info("file watcher started", "dir", fw.outputDir)
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
log.Println("File watcher shutting down...") log.Println("File watcher shutting down...")
logging.MustGetFactory().Transfer.Info("file watcher shutting down")
return ctx.Err() return ctx.Err()
case event, ok := <-fw.watcher.Events: case event, ok := <-fw.watcher.Events:
@ -73,7 +64,6 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
return fmt.Errorf("Watcher errors channel closed") return fmt.Errorf("Watcher errors channel closed")
} }
log.Printf("Watcher error: %v", err) log.Printf("Watcher error: %v", err)
logging.MustGetFactory().Transfer.Error("watcher error", logging.KeyError, err.Error())
} }
} }
} }
@ -98,18 +88,6 @@ func (fw *FileWatcher) addWatchRecursive(root string) error {
} }
func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) { func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
// Directories must be handled before the .ts filter below. fsnotify is not
// recursive, so a resolution directory created after the watcher starts —
// which is the normal case, since the transfer service starts before the
// downloader creates 1080p/ and friends — has to be picked up here or
// nothing inside it is ever seen.
if event.Op&fsnotify.Create == fsnotify.Create {
if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
fw.watchNewDirectory(event.Name)
return
}
}
if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") { if !strings.HasSuffix(strings.ToLower(event.Name), ".ts") {
return return
} }
@ -122,33 +100,15 @@ func (fw *FileWatcher) handleFileEvent(event fsnotify.Event) {
case event.Op&fsnotify.Remove == fsnotify.Remove: case event.Op&fsnotify.Remove == fsnotify.Remove:
fw.cancelPendingTransfer(event.Name) fw.cancelPendingTransfer(event.Name)
} }
}
// watchNewDirectory starts watching a directory that appeared after startup, if event.Op&fsnotify.Create == fsnotify.Create {
// and sweeps whatever is already inside it. if info, err := os.Stat(event.Name); err == nil && info.IsDir() {
// if err := fw.watcher.Add(event.Name); err != nil {
// The sweep is not redundant: the downloader creates a resolution directory and log.Printf("Failed to watch directory %s: %v", event.Name, err)
// writes segments into it immediately, so files can land between the mkdir and } else {
// the watch being registered. Those writes generate no event we would see, and log.Printf("Watching directory %s", event.Name)
// without the sweep the first segments of every recording would be lost. }
func (fw *FileWatcher) watchNewDirectory(dir string) {
if err := fw.addWatchRecursive(dir); err != nil {
log.Printf("Failed to watch new directory %s: %v", dir, err)
logging.MustGetFactory().Transfer.Warn("failed to watch new directory", "dir", dir, logging.KeyError, err.Error())
return
}
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil // keep sweeping the rest
} }
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
fw.scheduleTransfer(path)
}
return nil
})
if err != nil {
log.Printf("Failed to sweep new directory %s: %v", dir, err)
} }
} }
@ -185,8 +145,6 @@ func (fw *FileWatcher) processFile(filePath string) {
info, err := os.Stat(filePath) info, err := os.Stat(filePath)
if err != nil { if err != nil {
log.Printf("Failed to stat file %s: %v", filePath, err) log.Printf("Failed to stat file %s: %v", filePath, err)
logging.MustGetFactory().Transfer.Warn("failed to stat file", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
return return
} }
@ -195,18 +153,13 @@ func (fw *FileWatcher) processFile(filePath string) {
relPath, err := filepath.Rel(fw.outputDir, filePath) relPath, err := filepath.Rel(fw.outputDir, filePath)
if err != nil { if err != nil {
log.Printf("Failed to get relative path for file %s: %v", filePath, err) log.Printf("Failed to get relative path for file %s: %v", filePath, err)
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
return return
} }
// The destination is namespaced by event, mirroring the local layout and
// matching what QueueExistingFiles produces for the same file — otherwise
// live transfers and catch-up transfers would write to different places.
item := TransferItem{ item := TransferItem{
ID: generateID(), ID: generateID(),
SourcePath: filePath, SourcePath: filePath,
DestinationPath: filepath.Join(fw.eventName, relPath), DestinationPath: relPath,
Resolution: resolution, Resolution: resolution,
Timestamp: time.Now(), Timestamp: time.Now(),
Status: StatusPending, Status: StatusPending,
@ -215,11 +168,8 @@ func (fw *FileWatcher) processFile(filePath string) {
if err := fw.queue.Add(item); err != nil { if err := fw.queue.Add(item); err != nil {
log.Printf("Failed to add file to queue: %v", err) log.Printf("Failed to add file to queue: %v", err)
logging.MustGetFactory().Transfer.Error("failed to add file to queue", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
} else { } else {
log.Printf("Added file to queue: %s", filePath) log.Printf("Added file to queue: %s", filePath)
logging.MustGetFactory().Transfer.Debug("file queued", logging.KeyFile, filePath)
} }
} }

View File

@ -1,223 +0,0 @@
package transfer
import (
"context"
"os"
"path/filepath"
"sort"
"testing"
"time"
)
// settlingDelay is short so tests exercise the real timer path without
// waiting the production five seconds.
const testSettlingDelay = 40 * time.Millisecond
// newTestQueue builds a queue that accepts items but never transfers them:
// ProcessQueue is not started, so Add is a pure recording of intent.
func newTestQueue(t *testing.T) *TransferQueue {
t.Helper()
return NewTransferQueue(QueueConfig{
WorkerCount: 1,
PersistencePath: filepath.Join(t.TempDir(), "queue.json"),
MaxQueueSize: 100,
BatchSize: 10,
}, nil, nil)
}
// queuedDestinations returns the destination paths currently queued, sorted.
func queuedDestinations(q *TransferQueue) []string {
q.mu.RLock()
defer q.mu.RUnlock()
out := make([]string, 0, q.items.Len())
for _, item := range *q.items {
out = append(out, filepath.ToSlash(item.DestinationPath))
}
sort.Strings(out)
return out
}
// waitForQueue polls until the queue holds want items or the deadline passes.
// Polling beats a fixed sleep: it keeps the test fast when the watcher is
// prompt and tolerant when a loaded CI machine is not.
func waitForQueue(t *testing.T, q *TransferQueue, want int) []string {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
var got []string
for time.Now().Before(deadline) {
got = queuedDestinations(q)
if len(got) >= want {
return got
}
time.Sleep(10 * time.Millisecond)
}
return got
}
func startWatcher(t *testing.T, dir, eventName string, q *TransferQueue) {
t.Helper()
fw, err := NewFileWatcher(dir, eventName, q, testSettlingDelay)
if err != nil {
t.Fatalf("NewFileWatcher failed: %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
started := make(chan struct{})
go func() {
close(started)
_ = fw.Start(ctx)
}()
<-started
// Give Start a moment to register its initial watches before the test
// begins creating files under them.
time.Sleep(100 * time.Millisecond)
}
// TestWatcherPicksUpDirectoriesCreatedAfterStart covers the normal recording
// sequence: the transfer service starts watching the event directory, and only
// then does the downloader create the per-resolution directories inside it.
// fsnotify is not recursive, so those directories have to be picked up from
// the create event or nothing inside them is ever transferred.
func TestWatcherPicksUpDirectoriesCreatedAfterStart(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "2026-regionals", queue)
resolutionDir := filepath.Join(eventDir, "1080p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("failed to create resolution dir: %v", err)
}
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("segment written to a new resolution directory was never queued, queue = %v", got)
}
if want := "2026-regionals/1080p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
// TestWatcherSweepsFilesRacingDirectoryCreation covers the tighter race where
// segments are written immediately after the mkdir, before the new watch can
// be registered. Those writes produce no event the watcher would receive, so
// the directory has to be swept when it is first seen.
func TestWatcherSweepsFilesRacingDirectoryCreation(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "event", queue)
// Stage the tree elsewhere and move it into place, so the files are
// guaranteed to already exist the instant the directory appears.
staging := filepath.Join(t.TempDir(), "1080p")
if err := os.MkdirAll(staging, 0755); err != nil {
t.Fatalf("failed to stage: %v", err)
}
for _, name := range []string{"seg_1001.ts", "seg_1002.ts"} {
if err := os.WriteFile(filepath.Join(staging, name), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to stage segment: %v", err)
}
}
if err := os.Rename(staging, filepath.Join(eventDir, "1080p")); err != nil {
t.Fatalf("failed to move staged directory into place: %v", err)
}
got := waitForQueue(t, queue, 2)
want := []string{"event/1080p/seg_1001.ts", "event/1080p/seg_1002.ts"}
if len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
t.Errorf("queued %v, want %v", got, want)
}
}
// TestWatcherDestinationIncludesEventName pins the layout agreement between
// live transfers and the catch-up path in QueueExistingFiles: both must write
// to <nas>/<event>/<resolution>/<file>.
func TestWatcherDestinationIncludesEventName(t *testing.T) {
eventDir := t.TempDir()
resolutionDir := filepath.Join(eventDir, "720p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
queue := newTestQueue(t)
startWatcher(t, eventDir, "my-event", queue)
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("expected one queued item, got %v", got)
}
if want := "my-event/720p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
// TestWatcherWithoutEventNameUsesRoot mirrors a download run with no -event:
// segments land under the local output root, so the NAS destination must not
// invent a directory level for them.
func TestWatcherWithoutEventNameUsesRoot(t *testing.T) {
eventDir := t.TempDir()
resolutionDir := filepath.Join(eventDir, "1080p")
if err := os.MkdirAll(resolutionDir, 0755); err != nil {
t.Fatalf("setup failed: %v", err)
}
queue := newTestQueue(t)
startWatcher(t, eventDir, "", queue)
if err := os.WriteFile(filepath.Join(resolutionDir, "seg_1001.ts"), []byte("payload"), 0644); err != nil {
t.Fatalf("failed to write segment: %v", err)
}
got := waitForQueue(t, queue, 1)
if len(got) != 1 {
t.Fatalf("expected one queued item, got %v", got)
}
if want := "1080p/seg_1001.ts"; got[0] != want {
t.Errorf("destination = %q, want %q", got[0], want)
}
}
func TestWatcherIgnoresNonSegmentFiles(t *testing.T) {
eventDir := t.TempDir()
queue := newTestQueue(t)
startWatcher(t, eventDir, "event", queue)
for _, name := range []string{"notes.txt", "manifest.json", "segment.ts.part"} {
if err := os.WriteFile(filepath.Join(eventDir, name), []byte("x"), 0644); err != nil {
t.Fatalf("failed to write %s: %v", name, err)
}
}
time.Sleep(testSettlingDelay + 300*time.Millisecond)
if got := queuedDestinations(queue); len(got) != 0 {
t.Errorf("non-segment files should not be queued, got %v", got)
}
}
func TestExtractResolution(t *testing.T) {
fw := &FileWatcher{}
cases := map[string]string{
filepath.Join("data", "event", "1080p", "seg.ts"): "1080p",
filepath.Join("data", "event", "720p", "seg.ts"): "720p",
filepath.Join("data", "event", "seg.ts"): "",
}
for path, want := range cases {
if got := fw.extractResolution(path); got != want {
t.Errorf("extractResolution(%q) = %q, want %q", path, got, want)
}
}
}

View File

@ -1,85 +0,0 @@
package utils
import (
"fmt"
"os"
"path/filepath"
"strings"
)
func SafeJoin(base string, elements ...string) string {
path := filepath.Join(append([]string{base}, elements...)...)
return filepath.Clean(path)
}
func EnsureDir(path string) error {
if err := os.MkdirAll(path, 0755); err != nil {
return fmt.Errorf("failed to create directory %s: %w", path, err)
}
return nil
}
func PathExists(path string) bool {
_, err := os.Stat(path)
return !os.IsNotExist(err)
}
// IsValidPath reports whether path is usable on Windows, the stricter of the
// two platforms this project targets, which reserves <>:"|?* in names.
//
// A leading drive letter is checked separately: its colon is legal there and
// nowhere else in the path. Without that carve-out the function rejects every
// absolute Windows path, including the drive-qualified ones it exists to
// validate.
func IsValidPath(path string) bool {
if path == "" {
return false
}
rest := path
if len(rest) >= 2 && rest[1] == ':' && isDriveLetter(rest[0]) {
rest = rest[2:]
}
return !strings.ContainsAny(rest, `<>:"|?*`)
}
func isDriveLetter(c byte) bool {
return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
}
func NormalizePath(path string) string {
return filepath.Clean(strings.ReplaceAll(path, "\\", string(filepath.Separator)))
}
func GetRelativePath(basePath, targetPath string) (string, error) {
// filepath.Rel("", "") succeeds and returns ".", which is a meaningless
// answer to a meaningless question; callers passing an empty path have a
// bug worth surfacing rather than papering over.
if basePath == "" || targetPath == "" {
return "", fmt.Errorf("both base and target paths are required (base %q, target %q)", basePath, targetPath)
}
rel, err := filepath.Rel(basePath, targetPath)
if err != nil {
return "", fmt.Errorf("failed to get relative path: %w", err)
}
return rel, nil
}
func ValidateWritablePath(path string) error {
dir := filepath.Dir(path)
if err := EnsureDir(dir); err != nil {
return err
}
testFile := filepath.Join(dir, ".write_test")
file, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("path %s is not writable: %w", dir, err)
}
file.Close()
os.Remove(testFile)
return nil
}

View File

@ -1,235 +0,0 @@
package utils
import (
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
func TestSafeJoin(t *testing.T) {
tests := []struct {
name string
base string
elements []string
want string
}{
{
name: "basic join",
base: "data",
elements: []string{"events", "test-event"},
want: filepath.Join("data", "events", "test-event"),
},
{
name: "empty elements",
base: "data",
elements: []string{},
want: "data",
},
{
name: "with path separators",
base: "data/events",
elements: []string{"test-event", "1080p"},
want: filepath.Join("data", "events", "test-event", "1080p"),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := SafeJoin(tt.base, tt.elements...)
if got != tt.want {
t.Errorf("SafeJoin() = %v, want %v", got, tt.want)
}
})
}
}
func TestEnsureDir(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
testPath := filepath.Join(tempDir, "test", "nested", "directory")
// Test creating nested directories
err = EnsureDir(testPath)
if err != nil {
t.Errorf("EnsureDir() failed: %v", err)
}
// Verify directory was created
if _, err := os.Stat(testPath); os.IsNotExist(err) {
t.Errorf("Directory was not created: %s", testPath)
}
// Test with existing directory (should not fail)
err = EnsureDir(testPath)
if err != nil {
t.Errorf("EnsureDir() failed on existing directory: %v", err)
}
}
func TestPathExists(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Test existing path
if !PathExists(tempDir) {
t.Errorf("PathExists() should return true for existing path: %s", tempDir)
}
// Test non-existing path
nonExistentPath := filepath.Join(tempDir, "does-not-exist")
if PathExists(nonExistentPath) {
t.Errorf("PathExists() should return false for non-existent path: %s", nonExistentPath)
}
// Test with file
testFile := filepath.Join(tempDir, "test.txt")
f, err := os.Create(testFile)
if err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
f.Close()
if !PathExists(testFile) {
t.Errorf("PathExists() should return true for existing file: %s", testFile)
}
}
func TestIsValidPath(t *testing.T) {
tests := []struct {
name string
path string
want bool
}{
{"empty path", "", false},
{"valid path", "data/events/test", true},
{"path with colon", "data:events", false},
{"path with pipe", "data|events", false},
{"path with question mark", "data?events", false},
{"path with asterisk", "data*events", false},
{"path with quotes", "data\"events", false},
{"path with angle brackets", "data<events>", false},
{"normal windows path", "C:\\data\\events", true}, // Windows path separators are actually OK
{"unix path", "/home/user/data", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := IsValidPath(tt.path)
if got != tt.want {
t.Errorf("IsValidPath(%q) = %v, want %v", tt.path, got, tt.want)
}
})
}
}
func TestNormalizePath(t *testing.T) {
tests := []struct {
name string
path string
want string
}{
{
name: "windows backslashes",
path: "data\\events\\test",
want: filepath.Join("data", "events", "test"),
},
{
name: "unix forward slashes",
path: "data/events/test",
want: filepath.Join("data", "events", "test"),
},
{
name: "mixed slashes",
path: "data\\events/test\\file",
want: filepath.Join("data", "events", "test", "file"),
},
{
name: "redundant separators",
path: "data//events\\\\test",
want: filepath.Join("data", "events", "test"),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := NormalizePath(tt.path)
if got != tt.want {
t.Errorf("NormalizePath(%q) = %q, want %q", tt.path, got, tt.want)
}
})
}
}
func TestGetRelativePath(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
basePath := tempDir
targetPath := filepath.Join(tempDir, "subdir", "file.txt")
rel, err := GetRelativePath(basePath, targetPath)
if err != nil {
t.Errorf("GetRelativePath() failed: %v", err)
}
expected := filepath.Join("subdir", "file.txt")
if rel != expected {
t.Errorf("GetRelativePath() = %q, want %q", rel, expected)
}
// Test with invalid paths
_, err = GetRelativePath("", "")
if err == nil {
t.Error("GetRelativePath() should fail with empty paths")
}
}
func TestValidateWritablePath(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Test writable path
writablePath := filepath.Join(tempDir, "test", "file.txt")
err = ValidateWritablePath(writablePath)
if err != nil {
t.Errorf("ValidateWritablePath() failed for writable path: %v", err)
}
// Verify directory was created
dir := filepath.Dir(writablePath)
if _, err := os.Stat(dir); os.IsNotExist(err) {
t.Errorf("Directory should have been created: %s", dir)
}
// Test with read-only directory (if supported by OS)
if runtime.GOOS != "windows" { // Skip on Windows as it's more complex
readOnlyDir := filepath.Join(tempDir, "readonly")
os.MkdirAll(readOnlyDir, 0755)
os.Chmod(readOnlyDir, 0444) // Read-only
defer os.Chmod(readOnlyDir, 0755) // Restore permissions for cleanup
readOnlyPath := filepath.Join(readOnlyDir, "file.txt")
err = ValidateWritablePath(readOnlyPath)
if err == nil {
t.Error("ValidateWritablePath() should fail for read-only directory")
}
if !strings.Contains(err.Error(), "not writable") {
t.Errorf("Expected 'not writable' error, got: %v", err)
}
}
}

View File

@ -1,146 +0,0 @@
package main
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
func main() {
fmt.Println("🧪 StreamRecorder Test Suite")
fmt.Println("============================")
startTime := time.Now()
// Set test environment to avoid interfering with real data
originalEnv := setupTestEnvironment()
defer restoreEnvironment(originalEnv)
// Create temporary directory for test data
tempDir, err := os.MkdirTemp("", "streamrecorder_test_*")
if err != nil {
fmt.Printf("❌ Failed to create temp directory: %v\n", err)
os.Exit(1)
}
defer func() {
fmt.Printf("🧹 Cleaning up test directory: %s\n", tempDir)
os.RemoveAll(tempDir)
}()
// Set test-specific environment variables
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
os.Setenv("PROCESS_OUTPUT_DIR", filepath.Join(tempDir, "out"))
os.Setenv("ENABLE_NAS_TRANSFER", "false") // Disable NAS for tests
os.Setenv("PROCESSING_ENABLED", "false") // Disable processing that needs FFmpeg
fmt.Printf("📁 Using temporary directory: %s\n", tempDir)
fmt.Println()
// Run tests for each package
packages := []string{
"./pkg/config",
"./pkg/utils",
"./pkg/constants",
"./pkg/httpClient",
"./pkg/media",
"./pkg/processing",
}
var failedPackages []string
totalTests := 0
passedTests := 0
for _, pkg := range packages {
fmt.Printf("🔍 Testing package: %s\n", pkg)
cmd := exec.Command("go", "test", "-v", pkg)
cmd.Dir = "."
output, err := cmd.CombinedOutput()
outputStr := string(output)
// Count tests
testCount := strings.Count(outputStr, "=== RUN")
passCount := strings.Count(outputStr, "--- PASS:")
totalTests += testCount
passedTests += passCount
if err != nil {
fmt.Printf("❌ FAILED: %s (%d/%d tests passed)\n", pkg, passCount, testCount)
failedPackages = append(failedPackages, pkg)
// Show failure details
lines := strings.Split(outputStr, "\n")
for _, line := range lines {
if strings.Contains(line, "FAIL:") ||
strings.Contains(line, "Error:") ||
strings.Contains(line, "panic:") {
fmt.Printf(" %s\n", line)
}
}
} else {
fmt.Printf("✅ PASSED: %s (%d tests)\n", pkg, testCount)
}
fmt.Println()
}
// Print summary
duration := time.Since(startTime)
fmt.Println("📊 Test Summary")
fmt.Println("===============")
fmt.Printf("Total packages: %d\n", len(packages))
fmt.Printf("Passed packages: %d\n", len(packages)-len(failedPackages))
fmt.Printf("Failed packages: %d\n", len(failedPackages))
fmt.Printf("Total tests: %d\n", totalTests)
fmt.Printf("Passed tests: %d\n", passedTests)
fmt.Printf("Failed tests: %d\n", totalTests-passedTests)
fmt.Printf("Duration: %v\n", duration.Round(time.Millisecond))
if len(failedPackages) > 0 {
fmt.Println()
fmt.Println("❌ Failed packages:")
for _, pkg := range failedPackages {
fmt.Printf(" - %s\n", pkg)
}
os.Exit(1)
} else {
fmt.Println()
fmt.Println("🎉 All tests passed!")
}
}
func setupTestEnvironment() map[string]string {
// Save original environment variables that we'll modify
originalEnv := make(map[string]string)
envVars := []string{
"LOCAL_OUTPUT_DIR",
"PROCESS_OUTPUT_DIR",
"ENABLE_NAS_TRANSFER",
"PROCESSING_ENABLED",
"NAS_OUTPUT_PATH",
"FFMPEG_PATH",
"WORKER_COUNT",
}
for _, envVar := range envVars {
originalEnv[envVar] = os.Getenv(envVar)
}
return originalEnv
}
func restoreEnvironment(originalEnv map[string]string) {
fmt.Println("🔄 Restoring original environment...")
for envVar, originalValue := range originalEnv {
if originalValue == "" {
os.Unsetenv(envVar)
} else {
os.Setenv(envVar, originalValue)
}
}
}