Compare commits

..

14 Commits

Author SHA1 Message Date
309eeef59a Health/status http endpoint, logging
Some checks failed
Build Go Binaries / build (push) Has been cancelled
2026-08-06 01:24:29 -05:00
ce2c5f3181 Merge branch 'cleanup' 2026-08-06 00:22:18 -05:00
11b2ba8a8b Bug fixes 2026-08-06 00:20:54 -05:00
townandgown
7d67cec2b7 Claude Testing Suite 2025-08-11 00:06:34 -05:00
townandgown
20df800715 Post-Claude Cleanup 2025-08-10 23:41:45 -05:00
townandgown
3c49a3fa9f pre-cleanup 2025-08-10 22:56:26 -05:00
townandgown
718d69c12a Adding CI/CD 2025-08-10 22:48:53 -05:00
townandgown
0a349cc406 Eliminating cyclical imports 2025-08-10 15:16:04 -05:00
a10688279f Non-tested processing service 2025-08-10 02:35:05 -05:00
4d73ce25c2 Linux compilation 2025-08-09 18:22:35 -05:00
899ca31bb3 Working Version 1.0.1 2025-08-09 03:00:50 -05:00
townandgown
c9d6900d48 Adding processing skeleton pt. 2 2025-08-09 02:05:39 -05:00
townandgown
8223a3ac99 Adding processing skeleton 2025-08-08 15:44:52 -05:00
townandgown
99594597db Manifest architecture present. Individual items not populating 2025-07-27 00:27:21 -05:00
46 changed files with 5405 additions and 263 deletions

View File

@ -0,0 +1,37 @@
name: Build Go Binaries
on:
push:
branches:
- main
jobs:
build:
runs-on: docker
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: 1.23
- name: Cross-Compile
run: |
mkdir -p dist
for GOOS in linux windows darwin; do
for GOARCH in arm64 amd64; do
OUTPUT="dist/myapp_${GOOS}_${GOARCH}"
echo "Building ${OUTPUT}"
if [ "$GOOS" = "windows" ]; then
OUTPUT="${OUTPUT}.exe"
fi
GOOS=$GOOS GOARCH=$GOARCH go build -o $OUTPUT ./cmd/main
done
done
- name: Upload Artifact
uses: actions/upload-artifact@v4
with:
name: flo_download-binaries
path: dist

217
CLAUDE.md
View File

@ -4,31 +4,140 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview ## Project Overview
This is a Go-based M3U8 downloader that parses HLS (HTTP Live Streaming) playlists to extract video and audio stream metadata. The end goal of this project is to have a listening REST API take in m3u8 urls, parse them, and eventually send to a conversion service. This is a Go-based HLS (HTTP Live Streaming) recorder that monitors M3U8 playlists and downloads video segments in real-time with automatic NAS transfer capabilities. The program takes a master M3U8 playlist URL, parses all available stream variants (different qualities/bitrates), continuously monitors each variant's chunklist for new segments, downloads them locally, and optionally transfers them to network storage for long-term archival.
## Architecture ## Architecture
The project follows a clean separation of concerns: The project follows a modular architecture with clear separation of concerns:
- **main.go**: Entry point that demonstrates usage of the media package - **cmd/**: Entry points for different execution modes
- **media/**: Core package containing M3U8 parsing logic - **main/main.go**: Primary CLI entry point with URL input, event naming, and mode selection
- **types.go**: Contains the main parsing logic and data structures (`StreamSet`, `VideoURL`, `AudioURL`) - **downloader/download.go**: Core download orchestration logic with transfer service integration
- **utils.go**: Utility functions for parsing attributes and resolution calculations - **processor/process.go**: Alternative processing entry point
- **transfer/transfer.go**: Transfer-only mode entry point
- **pkg/**: Core packages containing the application logic
- **media/**: HLS streaming and download logic
- **stream.go**: Stream variant parsing and downloading orchestration (`GetAllVariants`, `VariantDownloader`)
- **playlist.go**: M3U8 playlist loading and parsing (`LoadMediaPlaylist`)
- **segment.go**: Individual segment downloading logic (`DownloadSegment`, `SegmentJob`)
- **manifest.go**: Manifest generation and segment tracking (`ManifestWriter`, `ManifestItem`)
- **transfer/**: NAS transfer system (complete implementation available)
- **service.go**: Transfer service orchestration
- **watcher.go**: File system monitoring for new downloads
- **queue.go**: Priority queue with worker pool management
- **nas.go**: NAS file transfer with retry logic
- **cleanup.go**: Local file cleanup after successful transfer
- **types.go**: Transfer system data structures
- **processing/**: Video processing and concatenation system
- **service.go**: Processing service orchestration with FFmpeg integration
- **segment.go**: Individual segment processing logic
- **types.go**: Processing system data structures
- **nas/**: NAS connection and file operations
- **config.go**: NAS configuration structure
- **nas.go**: NAS service with connection management and file operations
- **config/**: Centralized configuration management with validation
- **config.go**: Configuration loading, validation, and path resolution
- **utils/**: Utility functions for cross-platform compatibility
- **paths.go**: Path manipulation and validation utilities
- **constants/constants.go**: Configuration constants and singleton access
- **httpClient/error.go**: HTTP error handling utilities
The `GetStreamMetadata()` function is the main entry point that: ## Core Functionality
1. Fetches the M3U8 master playlist via HTTP
2. Parses the content line by line ### Download Workflow
3. Extracts video streams (`#EXT-X-STREAM-INF`) and audio streams (`#EXT-X-MEDIA`) 1. **Parse Master Playlist**: `GetAllVariants()` fetches and parses the master M3U8 to extract all stream variants with different qualities/bitrates
4. Returns a `StreamSet` containing all parsed metadata 2. **Concurrent Monitoring**: Each variant gets its own goroutine running `VariantDownloader()` that continuously polls for playlist updates
3. **Segment Detection**: When new segments appear in a variant's playlist, they are queued for download
4. **Parallel Downloads**: Segments are downloaded concurrently with configurable worker pools and retry logic
5. **Quality Organization**: Downloaded segments are organized by resolution (1080p, 720p, etc.) in separate directories
6. **Manifest Generation**: `ManifestWriter` tracks all downloaded segments with sequence numbers and resolutions
### NAS Transfer Workflow (Optional)
1. **File Watching**: `FileWatcher` monitors download directories for new `.ts` files
2. **Transfer Queuing**: New files are added to a priority queue after a settling delay
3. **Background Transfer**: Worker pool transfers files to NAS with retry logic and verification
4. **Local Cleanup**: Successfully transferred files are automatically cleaned up locally
5. **State Persistence**: Queue state is persisted to survive crashes and restarts
### Video Processing Workflow (Optional)
1. **Segment Collection**: Processing service reads downloaded segments from NAS storage
2. **Quality Selection**: Automatically selects the highest quality variant available
3. **FFmpeg Processing**: Uses FFmpeg to concatenate segments into a single MP4 file
4. **Output Management**: Processed videos are saved to the configured output directory
5. **Concurrent Processing**: Multiple events can be processed simultaneously with worker pools
## Key Data Structures
- `StreamVariant`: Represents a stream quality variant with URL, bandwidth, resolution, output directory, and manifest writer
- `SegmentJob`: Represents a segment download task with URI, sequence number, and variant info
- `ManifestWriter`: Tracks downloaded segments and generates JSON manifests
- `ManifestItem`: Individual segment record with sequence number and resolution
- `TransferItem`: Transfer queue item with source, destination, retry count, and status
- `TransferService`: Orchestrates file watching, queuing, transfer, and cleanup
- `ProcessingService`: Manages video processing operations with FFmpeg integration
- `ProcessConfig`: Configuration for processing operations including worker count and paths
- `NASService`: Handles NAS connection, authentication, and file operations
- `NASConfig`: Configuration structure for NAS connection parameters
## Configuration
Configuration is managed through a centralized system in `pkg/config/config.go` with environment variable support for deployment flexibility. The system provides validation, cross-platform path resolution, and sensible defaults:
### Core Settings
- `Core.WorkerCount`: Number of concurrent segment downloaders per variant (4) - ENV: `WORKER_COUNT`
- `Core.RefreshDelay`: How often to check for playlist updates (3 seconds) - ENV: `REFRESH_DELAY_SECONDS`
### Path Configuration
- `Paths.LocalOutput`: Base directory for local downloads (`data/`) - ENV: `LOCAL_OUTPUT_DIR`
- `Paths.ProcessOutput`: Directory for processed videos (`out/`) - ENV: `PROCESS_OUTPUT_DIR`
- `Paths.ManifestDir`: Directory for manifest JSON files (`data/`)
- `Paths.PersistenceFile`: Transfer queue state file location
### HTTP Settings
- `HTTPUserAgent`: User agent string for HTTP requests
- `REFERRER`: Referer header for HTTP requests (`https://www.flomarching.com`)
### NAS Transfer Settings
- `NAS.EnableTransfer`: Enable/disable automatic NAS transfer (true) - ENV: `ENABLE_NAS_TRANSFER`
- `NAS.OutputPath`: UNC path to NAS storage (`\\HomeLabNAS\dci\streams`) - ENV: `NAS_OUTPUT_PATH`
- `NAS.Username`/`NAS.Password`: NAS credentials for authentication - ENV: `NAS_USERNAME`/`NAS_PASSWORD`
- `Transfer.WorkerCount`: Concurrent transfer workers (2)
- `Transfer.RetryLimit`: Max retry attempts per file (3)
- `Transfer.Timeout`: Timeout per file transfer (30 seconds)
- `Transfer.FileSettlingDelay`: Wait before queuing new files (5 seconds)
- `Transfer.QueueSize`: Maximum queue size (100000)
- `Transfer.BatchSize`: Batch processing size (1000)
### Processing Settings
- `Processing.AutoProcess`: Enable automatic processing after download (true)
- `Processing.Enabled`: Enable processing functionality (true)
- `Processing.WorkerCount`: Concurrent processing workers (2)
- `Processing.FFmpegPath`: Path to FFmpeg executable (`ffmpeg`) - ENV: `FFMPEG_PATH`
### Cleanup Settings
- `Cleanup.AfterTransfer`: Delete local files after NAS transfer (true)
- `Cleanup.BatchSize`: Files processed per cleanup batch (1000)
- `Cleanup.RetainHours`: Hours to keep local files (0 = immediate cleanup)
### Configuration Access
```go
cfg := constants.MustGetConfig() // Get validated config singleton
eventPath := cfg.GetEventPath("my-event") // Get cross-platform paths
```
See `DEPLOYMENT.md` for detailed environment variable configuration and deployment examples.
## Common Development Commands ## Common Development Commands
```bash ```bash
# Build the project # Build the main application
go build -o m3u8-downloader go build -o stream-recorder ./cmd/main
# Run the project # Run with URL prompt
go run main.go go run ./cmd/main/main.go
# Run with command line arguments
go run ./cmd/main/main.go -url="https://example.com/playlist.m3u8" -event="my-event" -debug=true
# Run with module support # Run with module support
go mod tidy go mod tidy
@ -40,12 +149,80 @@ go test ./...
go fmt ./... go fmt ./...
``` ```
## Key Data Structures ## Command Line Options
- `StreamSet`: Root structure containing playlist URL and all streams - `-url`: M3U8 playlist URL (if not provided, prompts for input)
- `VideoURL`: Represents video stream with bandwidth, codecs, resolution, frame rate - `-event`: Event name for organizing downloads (defaults to current date)
- `AudioURL`: Represents audio stream with media type, group ID, name, and selection flags - `-debug`: Debug mode (only downloads 1080p variant for easier testing)
- `-transfer`: Transfer-only mode (transfer existing files without downloading)
- `-process`: Process-only mode (process existing files without downloading)
## Monitoring and Downloads
The application implements comprehensive real-time stream monitoring:
### Download Features
- **Continuous Polling**: Each variant playlist is checked every 3 seconds for new segments
- **Deduplication**: Uses segment URIs and sequence numbers to avoid re-downloading
- **Graceful Shutdown**: Responds to SIGINT/SIGTERM signals for clean exit
- **Error Resilience**: Retries failed downloads and handles HTTP 403 errors specially
- **Quality Detection**: Automatically determines resolution from bandwidth or explicit resolution data
- **Context Cancellation**: Proper timeout and cancellation handling for clean shutdowns
### Transfer Features (when enabled)
- **Real-time Transfer**: Files are transferred to NAS as soon as they're downloaded
- **Queue Persistence**: Transfer queue survives application restarts
- **Retry Logic**: Failed transfers are retried with exponential backoff
- **Verification**: File sizes are verified after transfer
- **Automatic Cleanup**: Local files are removed after successful NAS transfer
- **Statistics Reporting**: Transfer progress and statistics are logged regularly
### Manifest Generation
- **Segment Tracking**: All downloaded segments are tracked with sequence numbers
- **Resolution Mapping**: Segments are associated with their quality variants
- **JSON Output**: Manifest files are generated as sorted JSON arrays for easy processing
## Error Handling ## Error Handling
The current implementation uses `panic()` for error handling. When extending functionality, consider implementing proper error handling with returned error values following Go conventions. The implementation uses proper Go error handling patterns:
- **Custom HTTP Errors**: Structured error types for HTTP failures
- **Context-Aware Cancellation**: Proper handling of shutdown scenarios
- **Retry Logic**: Exponential backoff for transient failures
- **Logging**: Clear status indicators (✓ for success, ✗ for failure)
- **Graceful Degradation**: Transfer service failures don't stop downloads
## Dependencies
- `github.com/grafov/m3u8`: M3U8 playlist parsing
- `github.com/fsnotify/fsnotify`: File system event monitoring for NAS transfers
## Data Organization
Downloaded files are organized as:
```
./data/
├── {event-name}.json # Manifest file
├── {event-name}/ # Event-specific directory
│ ├── 1080p/ # High quality segments
│ ├── 720p/ # Medium quality segments
│ └── 480p/ # Lower quality segments
├── transfer_queue.json # Transfer queue state
├── refresh_token.txt # Authentication tokens
└── tokens.txt # Session tokens
```
NAS files mirror the local structure:
```
\\HomeLabNAS\dci\streams\
└── {event-name}/
├── 1080p/
├── 720p/
└── 480p/
```
Processed files are output to:
```
./out/
└── {event-name}/
└── concatenated_segments.mp4 # Final processed video
```

192
DEPLOYMENT.md Normal file
View File

@ -0,0 +1,192 @@
# Deployment Guide
This document outlines how to deploy and configure the StreamRecorder application in different environments.
## Environment Variables
The application supports configuration through environment variables for flexible deployment:
### Core Settings
- `WORKER_COUNT`: Number of concurrent segment downloaders per variant (default: 4)
- `REFRESH_DELAY_SECONDS`: How often to check for playlist updates in seconds (default: 3)
### NAS Transfer Settings
- `NAS_OUTPUT_PATH`: UNC path to NAS storage (default: "\\\\HomeLabNAS\\dci\\streams")
- `NAS_USERNAME`: NAS authentication username
- `NAS_PASSWORD`: NAS authentication password
- `ENABLE_NAS_TRANSFER`: Enable/disable automatic NAS transfer (default: true)
### Path Configuration
- `LOCAL_OUTPUT_DIR`: Base directory for local downloads (default: "data")
- `PROCESS_OUTPUT_DIR`: Output directory for processed videos (default: "out")
### Processing Settings
- `FFMPEG_PATH`: Path to FFmpeg executable (default: "ffmpeg")
## Docker Deployment
### Dockerfile Example
```dockerfile
FROM golang:1.23-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o stream-recorder ./cmd/main
FROM alpine:latest
RUN apk --no-cache add ca-certificates ffmpeg
WORKDIR /root/
COPY --from=builder /app/stream-recorder .
CMD ["./stream-recorder"]
```
### Docker Compose Example
```yaml
version: '3.8'
services:
stream-recorder:
build: .
environment:
- NAS_OUTPUT_PATH=/mnt/nas/streams
- NAS_USERNAME=${NAS_USERNAME}
- NAS_PASSWORD=${NAS_PASSWORD}
- LOCAL_OUTPUT_DIR=/app/data
- PROCESS_OUTPUT_DIR=/app/out
- FFMPEG_PATH=ffmpeg
volumes:
- ./data:/app/data
- ./out:/app/out
- nas_mount:/mnt/nas
networks:
- stream_network
volumes:
nas_mount:
driver: local
driver_opts:
type: cifs
device: "//HomeLabNAS/dci"
o: username=${NAS_USERNAME},password=${NAS_PASSWORD},iocharset=utf8
networks:
stream_network:
driver: bridge
```
## Kubernetes Deployment
### ConfigMap Example
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: stream-recorder-config
data:
WORKER_COUNT: "4"
REFRESH_DELAY_SECONDS: "3"
ENABLE_NAS_TRANSFER: "true"
LOCAL_OUTPUT_DIR: "/app/data"
PROCESS_OUTPUT_DIR: "/app/out"
FFMPEG_PATH: "ffmpeg"
```
### Secret Example
```yaml
apiVersion: v1
kind: Secret
metadata:
name: stream-recorder-secrets
type: Opaque
data:
NAS_USERNAME: <base64-encoded-username>
NAS_PASSWORD: <base64-encoded-password>
```
### Deployment Example
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: stream-recorder
spec:
replicas: 1
selector:
matchLabels:
app: stream-recorder
template:
metadata:
labels:
app: stream-recorder
spec:
containers:
- name: stream-recorder
image: stream-recorder:latest
envFrom:
- configMapRef:
name: stream-recorder-config
- secretRef:
name: stream-recorder-secrets
volumeMounts:
- name: data-storage
mountPath: /app/data
- name: output-storage
mountPath: /app/out
volumes:
- name: data-storage
persistentVolumeClaim:
claimName: stream-data-pvc
- name: output-storage
persistentVolumeClaim:
claimName: stream-output-pvc
```
## Production Considerations
### Security
- Never commit credentials to version control
- Use environment variables or secret management systems for sensitive data
- Consider using service accounts or IAM roles for cloud deployments
- Rotate credentials regularly
### Monitoring
- Implement health checks for the application
- Monitor disk space for download directories
- Set up alerts for failed transfers or processing
- Log to centralized logging systems
### Scaling
- Use horizontal scaling for multiple concurrent streams
- Consider using message queues for segment processing
- Implement distributed storage for high availability
- Use load balancers for multiple instances
### Backup and Recovery
- Regular backups of configuration and state files
- Test recovery procedures
- Document rollback processes
- Maintain disaster recovery plans
## Configuration Validation
The application validates configuration at startup and will fail fast if:
- Required directories cannot be created
- NAS paths are invalid when transfer is enabled
- FFmpeg is not found when processing is enabled
- Critical environment variables are malformed
## Troubleshooting
### Common Issues
1. **Path Permission Errors**: Ensure the application has write access to configured directories
2. **NAS Connection Failures**: Verify network connectivity and credentials
3. **FFmpeg Not Found**: Install FFmpeg or set correct FFMPEG_PATH
4. **Environment Variable Format**: Check for typos and correct boolean values
### Debug Mode
Run with `-debug=true` to enable debug logging and download only 1080p variants for testing.

128
Makefile Normal file
View File

@ -0,0 +1,128 @@
# StreamRecorder Makefile
.PHONY: test test-verbose test-coverage build clean help
# Default target
all: build
# Build the application
build:
@echo "🔨 Building StreamRecorder..."
go build -o stream-recorder.exe ./cmd/main
# Build for different platforms
build-windows:
@echo "🔨 Building for Windows..."
GOOS=windows GOARCH=amd64 go build -o stream-recorder-windows.exe ./cmd/main
build-linux:
@echo "🔨 Building for Linux..."
GOOS=linux GOARCH=amd64 go build -o stream-recorder-linux ./cmd/main
build-all: build-windows build-linux
# Run unit tests
test:
@echo "🧪 Running unit tests..."
go run test_runner.go
# Run tests with verbose output
test-verbose:
@echo "🧪 Running unit tests (verbose)..."
go test -v ./pkg/...
# Run tests with coverage
test-coverage:
@echo "🧪 Running tests with coverage..."
go test -coverprofile=coverage.out ./pkg/...
go tool cover -html=coverage.out -o coverage.html
@echo "📊 Coverage report generated: coverage.html"
# Run tests for a specific package
test-pkg:
@if [ -z "$(PKG)" ]; then \
echo "❌ Please specify package: make test-pkg PKG=./pkg/config"; \
exit 1; \
fi
@echo "🧪 Testing package: $(PKG)"
go test -v $(PKG)
# Run benchmarks
benchmark:
@echo "🏃 Running benchmarks..."
go test -bench=. -benchmem ./pkg/...
# Clean build artifacts
clean:
@echo "🧹 Cleaning build artifacts..."
rm -f stream-recorder.exe stream-recorder-windows.exe stream-recorder-linux
rm -f coverage.out coverage.html
rm -rf data/ out/ *.json
# Format code
fmt:
@echo "🎨 Formatting code..."
go fmt ./...
# Lint code (requires golangci-lint)
lint:
@echo "🔍 Linting code..."
golangci-lint run
# Tidy dependencies
tidy:
@echo "📦 Tidying dependencies..."
go mod tidy
# Run security check (requires gosec)
security:
@echo "🔒 Running security check..."
gosec ./...
# Install development tools
install-tools:
@echo "🛠️ Installing development tools..."
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
go install github.com/securecodewarrior/gosec/v2/cmd/gosec@latest
# Quick development cycle: format, tidy, build, test
dev: fmt tidy build test
# CI pipeline: format check, lint, security, test, build
ci: fmt tidy lint security test build
# Help
help:
@echo "StreamRecorder Build Commands"
@echo "============================="
@echo ""
@echo "Build Commands:"
@echo " build - Build the main application"
@echo " build-windows - Build for Windows (x64)"
@echo " build-linux - Build for Linux (x64)"
@echo " build-all - Build for all platforms"
@echo ""
@echo "Test Commands:"
@echo " test - Run unit tests with custom runner"
@echo " test-verbose - Run tests with verbose output"
@echo " test-coverage - Run tests with coverage report"
@echo " test-pkg PKG=<pkg> - Test specific package"
@echo " benchmark - Run benchmarks"
@echo ""
@echo "Quality Commands:"
@echo " fmt - Format code"
@echo " lint - Lint code (requires golangci-lint)"
@echo " security - Security analysis (requires gosec)"
@echo " tidy - Tidy dependencies"
@echo ""
@echo "Development Commands:"
@echo " dev - Quick dev cycle (fmt, tidy, build, test)"
@echo " ci - Full CI pipeline"
@echo " clean - Clean build artifacts"
@echo " install-tools - Install development tools"
@echo ""
@echo "Examples:"
@echo " make test"
@echo " make test-pkg PKG=./pkg/config"
@echo " make build-all"
@echo " make dev"

180
README.md Normal file
View File

@ -0,0 +1,180 @@
# m3u8-downloader
A Go command-line tool that records live HLS (HTTP Live Streaming) broadcasts. Point it at a master `.m3u8` playlist URL and it will:
1. **Download** every quality variant (1080p, 720p, ...) in real time as new segments appear.
2. **Transfer** finished segments to NAS storage for archival, with retry logic and local cleanup.
3. **Process** the archived segments into a single playable `.mp4` per event via FFmpeg.
**Download and transfer run together, in one invocation.** When NAS transfer is enabled (the default), starting a download also starts the transfer service in the same process: as soon as a segment is written to disk, a file watcher picks it up, transfers it to NAS, and deletes the local copy — all while the download keeps running. This keeps local disk usage bounded during a long recording instead of accumulating the whole event locally.
**Processing is a separate, later invocation** (see [Usage](#usage) below), because it reads finished segments *from NAS*, after transfer has fully drained — it can't run concurrently with the other two in any useful way.
There's also a standalone `-transfer` mode, but it's a catch-up/recovery tool, not the normal path: use it when a download session already ended with some files not yet confirmed on NAS (killed before the queue drained, or transfer was disabled during the recording and you want to sync afterward).
The project also ships structured JSON logging per subsystem and an HTTP status endpoint with a small web dashboard, so you can tell what's happening (and especially what's failing) without staring at a terminal.
> Project docs also refer to this as "StreamRecorder" (see `DEPLOYMENT.md`/`TESTING.md`) — same project, the Go module is named `m3u8-downloader`.
## Architecture
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
.m3u8 URL ──▶ │ DOWNLOAD │──────▶ │ TRANSFER │──────▶ │ PROCESSING │──▶ event.mp4
│ (pkg/media) │ .ts │(pkg/transfer,│ NAS │(pkg/processing)
└──────────────┘ files │ pkg/nas) │ files └──────────────┘
│ └──────────────┘ │
└──────────────┬─────────────────────────────────┘
▼
pkg/logging (JSON files) + pkg/health (HTTP /status, dashboard)
```
### Download — `pkg/media`
- `stream.go`: `GetAllVariants()` fetches and parses the master playlist, extracting every quality variant. `VariantDownloader()` runs one per-variant polling loop (every `Core.RefreshDelay`, default 3s) that watches the variant's chunklist for new segments and dispatches downloads through a worker-limited semaphore (`Core.WorkerCount` per variant, default 4).
- `playlist.go`: `LoadMediaPlaylist()` fetches and parses an individual variant's chunklist.
- `segment.go`: `DownloadSegment()` downloads one `.ts` file, with a couple of retries (including a dedicated retry for transient HTTP 403s some CDNs return).
- `manifest.go`: `ManifestWriter` tracks every segment downloaded per event and writes a JSON manifest (sequence number + resolution) at the end of a run.
Segments land on disk under `Paths.LocalOutput/<event>/<resolution>/`.
### Transfer — `pkg/transfer` + `pkg/nas`
- `service.go`: `TransferService` wires together the watcher, queue, and cleanup service, and owns the connection to NAS for the run.
- `watcher.go`: `FileWatcher` (via `fsnotify`) watches the local event directory for new `.ts` files and queues them after a short settling delay (`Transfer.FileSettlingDelay`, default 5s), so a file isn't queued while still being written.
- `queue.go`: `TransferQueue` is a priority-queued worker pool (`Transfer.WorkerCount`, default 2) that transfers queued files to NAS with retry + exponential backoff, and persists its state to `transfer_queue.json` so it survives a restart.
- `nas.go` / `pkg/nas`: `NASService` establishes the NAS connection (on Windows, via `net use` for UNC paths) and does the actual file copy + size verification.
- `cleanup.go`: `CleanupService` deletes local copies after a successful, verified transfer (`Cleanup.AfterTransfer`, default true).
During a normal download run, this whole stage starts automatically in the same process as the downloader (see the note above) — you don't invoke transfer separately for a live recording. The standalone `-transfer` mode exists only to catch up files that a prior download session ended without fully transferring.
### Processing — `pkg/processing`
- `ProcessingService` reads segments **from NAS** (not the local disk) for a given event, picks the highest-resolution variant available for each sequence number, writes an FFmpeg concat list, and shells out to FFmpeg to produce one `.mp4`.
- This is why processing is normally the *last* step, run after transfer has fully drained — it only sees what's actually on NAS.
- Requires a working `ffmpeg` on `PATH` (or `Processing.FFmpegPath` / `FFMPEG_PATH`), and connects to NAS on startup regardless of whether `Processing.Enabled` is set.
### Configuration — `pkg/config` + `pkg/constants`
`pkg/config/config.go` defines defaults and environment-variable overrides for every setting below. `pkg/constants.MustGetConfig()` lazily loads and caches this config as a process-wide singleton — most of the codebase reads config this way rather than passing it around explicitly.
### Logging — `pkg/logging`
A small factory (modeled after a structured-logging pattern used in another project) that hands out one `log/slog` JSON logger per pipeline stage: `App`, `Download`, `Transfer`, `Processing`. Each writes to its own file under `Logging.Dir` (default `data/logs/{app,download,transfer,processing}.log`). This is **additive** — every existing human-readable console log line is untouched, so you can still watch progress live in the terminal; the structured logs are there for after-the-fact debugging.
### Health / status — `pkg/health`
An in-memory registry that every subsystem reports into (active variants, segment success/failure counts, NAS connectivity, transfer queue stats, processing pipeline state), served over HTTP:
- `GET /status` — full JSON snapshot, `status: "healthy"` or `"degraded"` (degraded the moment anything has a recent error, NAS is disconnected while in use, or processing has failed).
- `GET /healthz` — plain liveness check.
- `GET /` — a small auto-refreshing dashboard (polls `/status` every 3s) showing the same information visually. No external dependencies (fonts/JS/CSS), so it works on a venue LAN with no internet access.
**Important scope note:** the health server only reflects whatever is live in the *current* process. During a normal download run, `/status` shows real-time download + transfer state together (they're the same process) with processing reported as `idle`. Running `-process` afterward is a separate process with its own fresh health server showing real-time processing state (and idle download/transfer). There's no single dashboard spanning download/transfer and processing at the same time, since processing only ever runs after the fact.
By default the health server binds all network interfaces (`:8090`) so it can be checked from another device on the same LAN (e.g. a phone) during an event. **There is no authentication or TLS** — this is fine on a trusted local network only, never expose it to the open internet.
## Configuration
All settings have sane defaults; override via environment variables as needed.
| Env var | Default | Meaning |
|---|---|---|
| `WORKER_COUNT` | `4` | Concurrent segment downloads per variant |
| `REFRESH_DELAY_SECONDS` | `3` | How often each variant's chunklist is polled |
| `ENABLE_NAS_TRANSFER` | `true` | Whether to run the transfer service alongside download |
| `NAS_OUTPUT_PATH` | `\\HomeLabNAS\dci\streams` | UNC/NAS root path segments are copied to |
| `NAS_USERNAME` / `NAS_PASSWORD` | *(see note below)* | NAS auth credentials |
| `LOCAL_OUTPUT_DIR` | `data` | Local base directory for downloaded segments |
| `PROCESS_OUTPUT_DIR` | `out` | Output directory for the final concatenated `.mp4` |
| `FFMPEG_PATH` | `ffmpeg` | Path to the FFmpeg binary used by the processing stage |
| `LOG_DIR` | `data/logs` | Directory for the per-subsystem JSON log files |
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` (also forced to `debug` by the `-debug` flag) |
| `HEALTH_ENABLED` | `true` | Whether to start the health/status HTTP server |
| `HEALTH_ADDR` | `:8090` | Bind address for the health server + dashboard |
> **Security note:** the default NAS credentials currently ship as hardcoded defaults in `pkg/config/config.go` for this home-lab setup. If you fork or redeploy this, set `NAS_USERNAME`/`NAS_PASSWORD` explicitly and rotate the shipped default — see `TODO.md` for the full flagged list of housekeeping items like this one.
## Building
```bash
go build -o m3u8-downloader ./cmd/main
# or, via the Makefile (produces stream-recorder(.exe)):
make build
```
## Usage
All modes are invoked through the single `cmd/main` binary:
```
m3u8-downloader [-url <m3u8-url>] [-event <name>] [-debug] [-transfer] [-process]
```
| Flag | Applies to | Meaning |
|---|---|---|
| `-url` | download mode | Master `.m3u8` playlist URL. If omitted (and neither `-transfer` nor `-process` is set), you'll be prompted for it interactively. |
| `-event` | all modes | Event name — used to namespace local output, NAS destination, and the manifest file. **Always set this explicitly**; if left blank in download mode, segments land directly under the local output root instead of a per-event subfolder. In `-transfer`/`-process` mode, omitting it shows an interactive picker over existing event directories. |
| `-debug` | download mode | Restricts download to the 1080p variant only (for faster iteration) and forces log level to `debug`. |
| `-transfer` | transfer-only mode | Scans an already-downloaded event's local directory and transfers/re-transfers anything not yet confirmed on NAS, then runs the normal transfer service until shutdown. Requires `ENABLE_NAS_TRANSFER=true`. |
| `-process` | process-only mode | Reads an event's segments from NAS and concatenates the highest-quality ones into a single `.mp4` via FFmpeg. Always attempts a NAS connection on startup, even if processing is otherwise disabled in config. |
### Which command for which situation
**Recording a live event** (the normal case — downloads and transfers to NAS concurrently, in one process, so local disk usage stays bounded rather than accumulating the whole event):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "2026-regionals"
```
Stop it cleanly with Ctrl+C (SIGINT/SIGTERM); it finishes in-flight downloads, saves transfer queue state, and writes the manifest before exiting.
**Quick test/dry run** (single variant, verbose logs, to confirm a playlist URL actually works before committing to a full multi-quality recording):
```bash
m3u8-downloader -url "https://example.com/live/master.m3u8" -event "test-run" -debug
```
**Catching up NAS transfer after the fact** (recovery only — the recording process above already transfers live; use this if a session ended before every file made it to NAS, or transfer was disabled during the recording and you want to sync afterward):
```bash
m3u8-downloader -event "2026-regionals" -transfer
```
**Producing the final video** (after transfer has fully drained — check `/status` or the NAS directory to confirm nothing's still pending):
```bash
m3u8-downloader -event "2026-regionals" -process
```
**Checking on a running instance** (from any device on the same network as the recorder):
```
http://<recorder-host>:8090/ # dashboard
http://<recorder-host>:8090/status # raw JSON
```
## Data layout
```
data/
├── {event}.json # download manifest (sequence numbers + resolutions)
├── {event}/
│ ├── 1080p/ # downloaded segments, before NAS transfer + cleanup
│ ├── 720p/
│ └── ...
├── logs/
│ ├── app.log # per-subsystem structured JSON logs
│ ├── download.log
│ ├── transfer.log
│ └── processing.log
└── transfer_queue.json # persisted transfer queue state
\\<NAS>\...\{event}\ # mirrors the local structure once transferred
├── 1080p/
└── ...
out/
└── {event}/
└── {event}.mp4 # final FFmpeg output (process-only mode)
```
## More documentation
- `DEPLOYMENT.md` — Docker/Kubernetes deployment examples and production considerations.
- `TESTING.md` — test suite layout and how to run it (`make test`, `make test-coverage`, etc.).
- `TODO.md` — known issues, pre-existing bugs, and roadmap items found during review, not yet addressed.

253
TESTING.md Normal file
View File

@ -0,0 +1,253 @@
# Testing Guide
This document describes the test suite for the StreamRecorder application.
## Overview
The test suite provides comprehensive coverage of core application components without requiring external dependencies like video files, NAS connectivity, or FFmpeg. All tests are self-contained and clean up after themselves.
## Test Structure
### Unit Tests by Package
#### `pkg/config`
- **File**: `config_test.go`
- **Coverage**: Configuration loading, environment variable override, path validation, validation errors
- **Key Tests**:
- Default config loading
- Environment variable overrides
- Path resolution and creation
- Validation error scenarios
#### `pkg/utils`
- **File**: `paths_test.go`
- **Coverage**: Cross-platform path utilities, directory operations, validation
- **Key Tests**:
- Safe path joining
- Directory creation
- Path existence checking
- Path validation
- Write permission testing
#### `pkg/constants`
- **File**: `constants_test.go`
- **Coverage**: Constants values, configuration singleton, integration
- **Key Tests**:
- Constant value verification
- Singleton pattern testing
- Config integration
- Concurrent access safety
#### `pkg/httpClient`
- **File**: `error_test.go`
- **Coverage**: HTTP error handling, status code management
- **Key Tests**:
- HTTP error creation and formatting
- Error comparison and detection
- Status code extraction
- Error wrapping support
#### `pkg/media`
- **File**: `manifest_test.go`
- **Coverage**: Manifest generation, segment tracking, JSON serialization
- **Key Tests**:
- Manifest writer initialization
- Segment addition and updates
- Quality resolution logic
- JSON file generation
- Sorting and deduplication
#### `pkg/processing`
- **File**: `service_test.go`
- **Coverage**: Processing service logic, path resolution, FFmpeg handling
- **Key Tests**:
- Service initialization
- Event directory scanning
- Resolution detection
- Segment aggregation
- File concatenation list generation
- FFmpeg path resolution
## Running Tests
### Quick Test Run
```bash
make test
```
### Verbose Output
```bash
make test-verbose
```
### Coverage Report
```bash
make test-coverage
```
Generates `coverage.html` with detailed coverage report.
### Test Specific Package
```bash
make test-pkg PKG=./pkg/config
```
### Manual Test Execution
```bash
# Run custom test runner
go run test_runner.go
# Run standard go test
go test ./pkg/...
# Run with coverage
go test -coverprofile=coverage.out ./pkg/...
go tool cover -html=coverage.out
```
## Test Features
### ✅ Self-Contained
- No external file dependencies
- No network connections required
- No NAS or FFmpeg installation needed
### ✅ Automatic Cleanup
- All temporary files/directories removed after tests
- Original environment variables restored
- No side effects on host system
### ✅ Isolated Environment
- Tests use temporary directories
- Environment variables safely overridden
- Configuration isolated from production settings
### ✅ Cross-Platform
- Path handling tested on Windows/Unix
- Platform-specific behavior validated
- Cross-platform compatibility verified
### ✅ Comprehensive Coverage
- Configuration management
- Path utilities and validation
- Error handling patterns
- Data structures and serialization
- Business logic without external dependencies
## Test Environment
The test suite automatically:
1. **Creates Temporary Workspace**: Each test run uses a fresh temporary directory
2. **Sets Test Environment**: Overrides environment variables to use test settings
3. **Disables External Dependencies**: Sets flags to disable NAS transfer and processing
4. **Cleans Up Completely**: Removes all test artifacts and restores environment
### Environment Variables Set During Tests
- `LOCAL_OUTPUT_DIR`: Points to temp directory
- `PROCESS_OUTPUT_DIR`: Points to temp directory
- `ENABLE_NAS_TRANSFER`: Set to `false`
- `PROCESSING_ENABLED`: Set to `false`
## Extending Tests
### Adding New Test Cases
1. **Create test file**: `pkg/yourpackage/yourfile_test.go`
2. **Follow naming convention**: `TestFunctionName`
3. **Use temp directories**: Always clean up created files
4. **Mock external dependencies**: Avoid real file operations where possible
### Test Template
```go
package yourpackage
import (
"os"
"testing"
)
func TestYourFunction(t *testing.T) {
// Setup
tempDir, err := os.MkdirTemp("", "test_*")
if err != nil {
t.Fatalf("Setup failed: %v", err)
}
defer os.RemoveAll(tempDir)
// Test logic
result := YourFunction()
// Assertions
if result != expected {
t.Errorf("Expected %v, got %v", expected, result)
}
}
```
### Best Practices
- **Always clean up**: Use `defer os.RemoveAll()` for temp directories
- **Test error cases**: Don't just test happy paths
- **Use table-driven tests**: For multiple similar test cases
- **Mock external dependencies**: Use echo/dummy commands instead of real tools
- **Validate cleanup**: Ensure tests don't leave artifacts
## CI/CD Integration
The test suite is designed for automated environments:
```bash
# Complete CI pipeline
make ci
# Just run tests in CI
make test
```
The custom test runner provides:
- ✅ Colored output for easy reading
- ✅ Test count and timing statistics
- ✅ Failure details and summaries
- ✅ Automatic environment management
- ✅ Exit codes for CI integration
## Troubleshooting
### Common Issues
**Tests fail with permission errors**
- Ensure write permissions in temp directory
- Check antivirus software isn't blocking file operations
**Config tests fail**
- Verify no conflicting environment variables are set
- Check that temp directories can be created
**Path tests fail on Windows**
- Confirm path separator handling is correct
- Verify Windows path validation logic
### Debug Mode
```bash
# Run with verbose output to see detailed failures
go test -v ./pkg/...
# Run specific failing test
go test -v -run TestSpecificFunction ./pkg/config
```
## Coverage Goals
Current test coverage targets:
- **Configuration**: 95%+ (critical for startup validation)
- **Path utilities**: 90%+ (cross-platform compatibility critical)
- **Constants**: 85%+ (verify all values and singleton behavior)
- **HTTP client**: 90%+ (error handling is critical)
- **Media handling**: 85%+ (core business logic)
- **Processing**: 70%+ (limited by external FFmpeg dependency)
Generate coverage report to verify:
```bash
make test-coverage
open coverage.html
```

111
TODO.md Normal file
View File

@ -0,0 +1,111 @@
# Return-from-a-year-away Audit — 2026-08-05
Scope reviewed: `main` (checked out), `manifest`, `cleanup` (all local + `origin` equivalents).
Remote: `https://git.aaronic.cc/kacarmichael/StreamRecorder.git`
## Branch situation (read this first)
The three branches are **not parallel efforts** — they're one linear line of development that `main` never caught up to:
```
main (0f4a39f, 2025-07-26) ──┐ stale, single-file-era downloader only
│
manifest (718d69c, 2025-08-10) ── +restructured cmd/pkg layout, config-less
│
cleanup (7d67cec, 2025-08-11) ── +pkg/config, tests, CI, Makefile, deployment docs
```
`manifest` is literally an ancestor of `cleanup` (same commit `718d69c`). `cleanup` is the most complete state that ever existed and was the last thing touched (2025-08-11) before the year gap. **None of this reached `main`.** If you pick this project back up, decide this before anything else:
- [ ] **Merge `cleanup` → `main`** (recommended — it's a strict superset: config validation, `pkg/nas`, `pkg/processing`, unit tests, CI workflow, `DEPLOYMENT.md`/`TESTING.md`). Then delete `manifest` (fully subsumed) and re-evaluate `cleanup` for deletion once merged.
- [ ] Or, if `cleanup`'s direction (FFmpeg concatenation, NAS-mirroring processing pipeline) is no longer where you want to go, explicitly archive it and restart `main` from a clean design — but do this as a decision, not by default.
---
## 🔴 Critical — fix/rotate before running this again
1. **Leaked NAS credentials, committed in plaintext.** `pkg/config/config.go` (branch `cleanup`) hardcodes a real default:
```go
Username: "NASAdmin",
Password: "s3tkY6tzA&KN6M",
```
This is in git history on a remote (`git.aaronic.cc`) regardless of whether you later change the default. **Rotate this NAS account's password now** — changing the code doesn't un-leak a password that's sat in a repo for a year. Then remove the credential entirely (fail closed: require `NAS_USERNAME`/`NAS_PASSWORD` env vars with no default, error out if unset when transfer is enabled) rather than shipping a working default.
2. **Guaranteed crash: double mutex unlock in `pkg/transfer/cleanup.go`.** In `ExecuteCleanup` (present on **`main` too**, not just `cleanup`):
```go
cs.mu.Lock()
...
cs.mu.Unlock() // line 72 — releases the lock
...
batch := make([]string, batchSize)
copy(batch, cs.pendingFiles[:batchSize])
cs.pendingFiles = cs.pendingFiles[batchSize:] // mutated WITHOUT the lock held
cs.mu.Unlock() // line 79 — unlocking an already-unlocked mutex
```
An unlock of an unlocked `sync.Mutex` is a **fatal, unrecoverable runtime error** (`fatal error: sync: unlock of unlocked mutex`) — it kills the whole process, not just a goroutine. Since `Cleanup.AfterTransfer` defaults to `true` and `RetainHours: 0` (immediate eligibility), this fires the first time any file finishes transferring and cleanup runs. This is very likely why any end-to-end run with NAS transfer enabled never survived long. Fix: hold the lock for the whole slice mutation, or copy-then-release-then-mutate-under-a-second-short-lock, but don't unlock twice.
3. **Nil-pointer crash in `pkg/media/stream.go` (`VariantDownloader`, `main` branch):**
```go
playlist, err := LoadMediaPlaylist(variant.URL)
seq := playlist.SeqNo // <-- dereferenced before the err check
if err != nil {
...
goto waitTick
}
```
`LoadMediaPlaylist` returns `nil, err` on any HTTP/parse failure, so `playlist` is a typed-nil pointer here — `playlist.SeqNo` panics. A panic in a goroutine takes down the entire process (Go doesn't isolate goroutine panics). Practically: any single transient network hiccup on any variant's playlist fetch kills the whole download session, mid-recording. Move the `err != nil` check above the `SeqNo` read.
---
## 🟠 High-priority bugs / correctness
4. **`PriorityQueue.Less` in `pkg/transfer/queue.go` sorts newest-first** (`pq[i].Timestamp.After(pq[j].Timestamp)`), making the transfer queue LIFO. For a live-recording pipeline that keeps producing new segments, this means the *oldest* queued segments can be starved indefinitely as long as new ones keep arriving — the opposite of what you'd want (drain oldest first so nothing waits forever). Worth confirming this was intentional; if not, flip the comparison.
5. **Retry backoff runs before the *first* attempt, not just retries** — `pkg/transfer/queue.go`, `processItem`:
```go
for attempt := 1; attempt <= maxRetries; attempt++ {
if attempt > 0 { // always true, since attempt starts at 1
... backoff ...
```
Every single transfer sleeps `1s` before even trying once. Should be `attempt > 1`.
6. **`pkg/constants/constants.go` (main) hardcodes a live FloSports auth token** directly in source (`MasterURL` with `token=...`, `hmac2=...`). It's expired now (epoch ~2025-07), so no live exposure, but it's the same pattern as #1 — secrets/tokens don't belong in source. Worth a rule going forward: URLs/tokens come from CLI args or env, never constants.
7. **`http.Client{}` created with no timeout** in `pkg/media/stream.go`, `playlist.go`, and `segment.go`. A stalled TCP connection (dead CDN edge, network partition) can hang a fetch forever — nothing in the surrounding code will time it out except the 10s context wrapped specifically around segment downloads in `VariantDownloader`. Playlist-fetch calls (`GetAllVariants`, `LoadMediaPlaylist`) have no such wrapper. Give the shared client a sane `Timeout` (e.g. 15–30s) as a backstop.
---
## 🟡 Cleanup / dead weight
8. **`cmd/proc/main.go` is a one-line stub** (`package proc`) on `main` — dead entry point, no `func main()`. Either finish it or delete it; right now `go build ./...` would fail on this if it were ever wired as a real `main` package (it currently silently isn't, which is its own trap for future-you).
9. **Built binaries committed to git** (`bin/flo_download`, `bin/flo_download.exe`, ~18MB, on `cleanup`/`manifest`). The `.gitea/workflows/build.yml` already cross-compiles and uploads artifacts — there's no reason binaries should also live in the repo/history. Remove and add `bin/` to `.gitignore`.
10. **`scratch.txt` is tracked despite being in `.gitignore`.** Adding a path to `.gitignore` after it's already tracked does nothing — it's still in the repo (and it's just a raw debug log dump). `git rm --cached scratch.txt`.
11. `.idea/*` (JetBrains project files) are committed. Low stakes solo-repo, but standard practice is to gitignore the whole `.idea/` directory rather than cherry-pick.
---
## 🔵 Program / architecture — bigger picture
12. **The project's own `CLAUDE.md` states the goal is "a listening REST API [that] take[s] in m3u8 urls, parse[s] them, and eventually send[s] to a conversion service."** None of the three branches implement any HTTP server — all three are CLI-only, one-shot-per-invocation tools. This is the largest gap between stated intent and actual code. Worth deciding: is the REST-API framing still the goal, or has the project's actual direction (CLI recorder + NAS mirror + FFmpeg concat, per `cleanup`'s `CLAUDE.md`) superseded it? Right now the docs and the code disagree with each other.
13. **`pkg/nas/nas.go`'s "connection" is a Windows-only shell-out to `net use`** (`EstablishConnection`), while CI (`build.yml`) cross-compiles for `linux`/`darwin`/`windows`. On non-Windows the UNC-path logic (`\\server\share`) and `net use` simply don't apply — there's no SMB client abstraction, so the Linux/macOS builds can produce a binary that can't actually do NAS transfer. If cross-platform deployment is real (not just "it happens to compile"), this needs a real cross-platform SMB/CIFS story (e.g., assume the NAS is pre-mounted by the OS and just treat it as a local path, which is actually simpler and already half-true for the Linux CIFS example in `DEPLOYMENT.md`).
14. **Zero tests on `main`.** `cleanup` has real coverage (`pkg/config`, `pkg/utils`, `pkg/constants`, `pkg/httpClient`, `pkg/media/manifest`, `pkg/processing`) plus a `Makefile` (`test`, `test-coverage`, `lint`, `security` via `gosec`) and a `TESTING.md`. All of this is stuck on a branch that was never merged — it's not a "write tests" TODO so much as a "land the tests you already wrote" TODO.
15. **CI never actually runs tests, lint, or `gosec`** despite the `Makefile` defining a full `ci` target (`fmt tidy lint security test build`). `.gitea/workflows/build.yml` only cross-compiles. Wire `make ci` (or its steps) into the pipeline so a broken build/test doesn't silently ship a binary artifact.
16. **Unbounded `seen` map** in `VariantDownloader` (`pkg/media/stream.go`) grows for the lifetime of a variant's polling loop with no eviction — fine for a few-hour event, worth a note if this is ever meant to run continuously/24-7.
---
## 🟢 Deployment
17. `DEPLOYMENT.md` (on `cleanup`) documents Docker/Compose/Kubernetes deployment in detail, but there's no actual `Dockerfile` committed anywhere in the repo — it's a doc-only example. If containerized deployment is the real target, commit the Dockerfile and at least smoke-test the build.
18. No secrets-management story beyond "use env vars" — fine for a home lab, but the doc's own "Security" section says "use secret management systems," which isn't backed by anything concrete (no `.env.example`, no sealed-secrets/vault reference). Given finding #1, this is worth actually doing, not just documenting.
19. Health checks, metrics/log shipping are listed as "considerations" in `DEPLOYMENT.md` but nothing is implemented — the app is a fire-and-forget CLI process with `log.Printf` to stdout only. If this runs unattended (e.g., in a container watching for the next event), you have no way to know it's alive short of watching logs.
---
## Suggested order of attack
1. Rotate the NAS credential (#1) — this is time-sensitive regardless of anything else.
2. Fix the two crash bugs (#2, #3) — nothing else is worth doing until a full recording session can survive.
3. Decide the branch-merge question and land `cleanup` → `main` (or consciously don't).
4. Wire CI to actually test/lint (#15) so regressions get caught going forward.
5. Then tackle the architecture question (#12: REST API vs. CLI-recorder-as-is) before investing more in either direction.

BIN
bin/flo_download Normal file

Binary file not shown.

BIN
bin/flo_download.exe Normal file

Binary file not shown.

123
cmd/downloader/download.go Normal file
View File

@ -0,0 +1,123 @@
package downloader
import (
"context"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/media"
"m3u8-downloader/pkg/transfer"
"m3u8-downloader/pkg/utils"
"os"
"os/signal"
"sync"
"syscall"
"time"
)
func Download(masterURL string, eventName string, debug bool) {
appLog := logging.MustGetFactory().App
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Goroutine to listen for shutdown signals
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-sigChan
log.Println("Shutting down...")
appLog.Info("shutdown signal received")
cancel()
}()
cfg := constants.MustGetConfig()
if cfg.Health.Enabled {
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
var wg sync.WaitGroup
var transferService *transfer.TransferService
if cfg.NAS.EnableTransfer {
ts, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
if err != nil {
log.Printf("Failed to create transfer service: %v", err)
log.Println("Continuing without transfer service...")
appLog.Error("failed to create transfer service, continuing without it", logging.KeyError, err.Error())
} else {
transferService = ts
wg.Add(1)
go func() {
defer wg.Done()
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err)
appLog.Error("transfer service error", logging.KeyError, err.Error())
}
}()
log.Println("Transfer service started.")
appLog.Info("transfer service started")
}
}
manifestWriter := media.NewManifestWriter(eventName)
eventPath := cfg.GetEventPath(eventName)
if err := utils.EnsureDir(eventPath); err != nil {
appLog.Error("failed to create event directory", logging.KeyError, err.Error())
log.Fatalf("Failed to create event directory: %v", err)
}
variants, err := media.GetAllVariants(masterURL, eventPath, manifestWriter)
if err != nil {
appLog.Error("failed to get variants", logging.KeyError, err.Error())
log.Fatalf("Failed to get variants: %v", err)
}
log.Printf("Found %d variants", len(variants))
sem := make(chan struct{}, constants.WorkerCount*len(variants))
manifest := media.NewManifestWriter(eventName)
for _, variant := range variants {
// Debug mode only tracks one variant for easier debugging
if debug {
if variant.Resolution != "1080p" {
continue
}
}
wg.Add(1)
go func(v *media.StreamVariant) {
defer wg.Done()
media.VariantDownloader(ctx, v, sem, manifest)
}(variant)
}
wg.Wait()
log.Println("All variant downloaders finished.")
appLog.Info("all variant downloaders finished")
if transferService != nil {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
transferService.Shutdown(shutdownCtx)
}
log.Println("All Services shut down.")
appLog.Info("all services shut down")
manifestWriter.WriteManifest()
log.Println("Manifest written.")
appLog.Info("manifest written", logging.KeyEvent, eventName)
}

View File

@ -1,75 +0,0 @@
package main
import (
"context"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/media"
"m3u8-downloader/pkg/transfer"
"os"
"os/signal"
"sync"
"syscall"
"time"
)
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Goroutine to listen for shutdown signals
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-sigChan
log.Println("Shutting down...")
cancel()
}()
var wg sync.WaitGroup
var transferService *transfer.TransferService
if constants.EnableNASTransfer {
ts, err := transfer.NewTrasferService(constants.NASPath)
if err != nil {
log.Printf("Failed to create transfer service: %v", err)
log.Println("Continuing without transfer service...")
} else {
transferService = ts
wg.Add(1)
go func() {
defer wg.Done()
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err)
}
}()
log.Println("Transfer service started.")
}
}
variants, err := media.GetAllVariants(constants.MasterURL)
if err != nil {
log.Fatalf("Failed to get variants: %v", err)
}
log.Printf("Found %d variants", len(variants))
sem := make(chan struct{}, constants.WorkerCount*len(variants))
for _, variant := range variants {
wg.Add(1)
go func(v *media.StreamVariant) {
defer wg.Done()
media.VariantDownloader(ctx, v, sem)
}(variant)
}
wg.Wait()
log.Println("All variant downloaders finished.")
if transferService != nil {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
transferService.Shutdown(shutdownCtx)
}
log.Println("All Services shut down.")
}

53
cmd/main/main.go Normal file
View File

@ -0,0 +1,53 @@
package main
import (
"bufio"
"flag"
"fmt"
"m3u8-downloader/cmd/downloader"
"m3u8-downloader/cmd/processor"
"m3u8-downloader/cmd/transfer"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"os"
"strings"
)
func main() {
url := flag.String("url", "", "M3U8 playlist URL")
eventName := flag.String("event", "", "Event name")
debug := flag.Bool("debug", false, "Enable debug mode")
transferOnly := flag.Bool("transfer", false, "Transfer-only mode: transfer existing files without downloading")
processOnly := flag.Bool("process", false, "Process-only mode: process existing files without downloading")
flag.Parse()
// Must happen before the first logging.MustGetFactory() call anywhere in
// the program, since the logging factory reads its level from this
// config singleton once and caches it.
if *debug {
constants.MustGetConfig().Logging.Level = "debug"
}
logging.MustGetFactory().App.Info("m3u8-downloader starting", "debug", *debug)
if *transferOnly {
transfer.RunTransferOnly(*eventName)
return
}
if *processOnly {
processor.Process(*eventName)
return
}
if *url == "" {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter M3U8 playlist URL: ")
inputUrl, _ := reader.ReadString('\n')
inputUrl = strings.TrimSpace(inputUrl)
downloader.Download(inputUrl, *eventName, *debug)
return
}
downloader.Download(*url, *eventName, *debug)
}

View File

@ -1 +0,0 @@
package proc

42
cmd/processor/process.go Normal file
View File

@ -0,0 +1,42 @@
package processor
import (
"context"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/processing"
"time"
)
func Process(eventName string) {
appLog := logging.MustGetFactory().App
log.Printf("Starting processing for event: %s", eventName)
appLog.Info("starting processing", logging.KeyEvent, eventName)
cfg := constants.MustGetConfig()
if cfg.Health.Enabled {
healthSrv, err := health.Start(context.Background(), &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
ps, err := processing.NewProcessingService(eventName, cfg)
if err != nil {
appLog.Error("failed to create processing service", logging.KeyError, err.Error())
log.Fatalf("Failed to create processing service: %v", err)
}
if err := ps.Start(context.Background()); err != nil {
appLog.Error("processing service failed", logging.KeyError, err.Error())
log.Fatalf("Failed to run processing service: %v", err)
}
}

139
cmd/transfer/transfer.go Normal file
View File

@ -0,0 +1,139 @@
package transfer
import (
"bufio"
"context"
"fmt"
"log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/transfer"
"m3u8-downloader/pkg/utils"
"os"
"os/signal"
"strconv"
"strings"
"syscall"
"time"
)
func getEventDirs(cfg *config.Config) ([]string, error) {
dirs, err := os.ReadDir(cfg.Paths.LocalOutput)
if err != nil {
return nil, fmt.Errorf("failed to read directory: %w", err)
}
var eventDirs []string
for _, dir := range dirs {
if dir.IsDir() {
eventDirs = append(eventDirs, dir.Name())
}
}
return eventDirs, nil
}
func RunTransferOnly(eventName string) {
appLog := logging.MustGetFactory().App
cfg := constants.MustGetConfig()
// Check if NAS transfer is enabled
if !cfg.NAS.EnableTransfer {
appLog.Error("NAS transfer disabled, cannot run transfer-only mode")
log.Fatal("NAS transfer is disabled in configuration. Please enable it to use transfer-only mode.")
}
if eventName == "" {
events, err := getEventDirs(cfg)
if err != nil {
log.Fatalf("Failed to get event directories: %v", err)
}
if len(events) == 0 {
log.Fatal("No events found")
}
if len(events) > 1 {
fmt.Println("Multiple events found, please select one:")
for i, event := range events {
fmt.Printf("%d. %s\n", i+1, event)
}
reader := bufio.NewReader(os.Stdin)
input, _ := reader.ReadString('\n')
input = strings.TrimSpace(input)
index, err := strconv.Atoi(input)
if err != nil {
log.Fatalf("Failed to parse input: %v", err)
}
if index < 1 || index > len(events) {
log.Fatal("Invalid input")
}
eventName = events[index-1]
} else {
eventName = events[0]
}
}
log.Printf("Starting transfer-only mode for event: %s", eventName)
// Setup context and signal handling
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-sigChan
log.Println("Shutting down transfer service...")
appLog.Info("shutdown signal received")
cancel()
}()
if cfg.Health.Enabled {
healthSrv, err := health.Start(ctx, &cfg.Health, health.MustGetRegistry())
if err != nil {
appLog.Error("failed to start health server", logging.KeyError, err.Error())
} else {
appLog.Info("health server listening", "addr", cfg.Health.Addr)
defer func() {
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shutdownCancel()
_ = healthSrv.Shutdown(shutdownCtx)
}()
}
}
// Verify local event directory exists
localEventPath := cfg.GetEventPath(eventName)
if !utils.PathExists(localEventPath) {
appLog.Error("local event directory does not exist", logging.KeyEvent, eventName)
log.Fatalf("Local event directory does not exist: %s", localEventPath)
}
// Create transfer service
transferService, err := transfer.NewTrasferService(cfg.NAS.OutputPath, eventName)
if err != nil {
appLog.Error("failed to create transfer service", logging.KeyError, err.Error())
log.Fatalf("Failed to create transfer service: %v", err)
}
// Find and queue existing files
if err := transferService.QueueExistingFiles(localEventPath); err != nil {
appLog.Error("failed to queue existing files", logging.KeyError, err.Error())
log.Fatalf("Failed to queue existing files: %v", err)
}
// Start transfer service
log.Println("Starting transfer service...")
appLog.Info("starting transfer service", logging.KeyEvent, eventName)
if err := transferService.Start(ctx); err != nil && err != context.Canceled {
log.Printf("Transfer service error: %v", err)
appLog.Error("transfer service error", logging.KeyError, err.Error())
}
// Graceful shutdown
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer shutdownCancel()
transferService.Shutdown(shutdownCtx)
log.Println("Transfer-only mode completed.")
appLog.Info("transfer-only mode completed", logging.KeyEvent, eventName)
}

6
go.sum Normal file
View File

@ -0,0 +1,6 @@
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/grafov/m3u8 v0.12.1 h1:DuP1uA1kvRRmGNAZ0m+ObLv1dvrfNO0TPx0c/enNk0s=
github.com/grafov/m3u8 v0.12.1/go.mod h1:nqzOkfBiZJENr52zTVd/Dcl03yzphIMbJqkXGu+u080=
golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

278
pkg/config/config.go Normal file
View File

@ -0,0 +1,278 @@
package config
import (
"fmt"
"os"
"path/filepath"
"strconv"
"time"
)
type Config struct {
Core CoreConfig
HTTP HTTPConfig
NAS NASConfig
Processing ProcessingConfig
Transfer TransferConfig
Cleanup CleanupConfig
Paths PathsConfig
Logging LoggingConfig
Health HealthConfig
}
type CoreConfig struct {
WorkerCount int
RefreshDelay time.Duration
}
type HTTPConfig struct {
UserAgent string
Referer string
}
type NASConfig struct {
EnableTransfer bool
OutputPath string
Username string
Password string
Timeout time.Duration
RetryLimit int
}
type ProcessingConfig struct {
Enabled bool
AutoProcess bool
WorkerCount int
FFmpegPath string
}
type TransferConfig struct {
WorkerCount int
RetryLimit int
Timeout time.Duration
FileSettlingDelay time.Duration
QueueSize int
BatchSize int
}
type CleanupConfig struct {
AfterTransfer bool
BatchSize int
RetainHours int
}
type PathsConfig struct {
BaseDir string
LocalOutput string
ProcessOutput string
ManifestDir string
PersistenceFile string
}
type LoggingConfig struct {
Dir string
Level string
}
type HealthConfig struct {
Enabled bool
Addr string
}
var defaultConfig = Config{
Core: CoreConfig{
WorkerCount: 4,
RefreshDelay: 3 * time.Second,
},
HTTP: HTTPConfig{
UserAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
Referer: "https://www.flomarching.com",
},
NAS: NASConfig{
EnableTransfer: true,
OutputPath: "\\\\HomeLabNAS\\dci\\streams",
Username: "NASAdmin",
Password: "s3tkY6tzA&KN6M",
Timeout: 30 * time.Second,
RetryLimit: 3,
},
Processing: ProcessingConfig{
Enabled: true,
AutoProcess: true,
WorkerCount: 2,
FFmpegPath: "ffmpeg",
},
Transfer: TransferConfig{
WorkerCount: 2,
RetryLimit: 3,
Timeout: 30 * time.Second,
FileSettlingDelay: 5 * time.Second,
QueueSize: 100000,
BatchSize: 1000,
},
Cleanup: CleanupConfig{
AfterTransfer: true,
BatchSize: 1000,
RetainHours: 0,
},
Paths: PathsConfig{
BaseDir: "data",
LocalOutput: "data",
ProcessOutput: "out",
ManifestDir: "data",
PersistenceFile: "transfer_queue.json",
},
Logging: LoggingConfig{
Dir: "data/logs",
Level: "info",
},
Health: HealthConfig{
Enabled: true,
Addr: ":8090",
},
}
func Load() (*Config, error) {
cfg := defaultConfig
if err := cfg.loadFromEnvironment(); err != nil {
return nil, fmt.Errorf("failed to load environment config: %w", err)
}
if err := cfg.resolveAndValidatePaths(); err != nil {
return nil, fmt.Errorf("path validation failed: %w", err)
}
return &cfg, nil
}
func (c *Config) loadFromEnvironment() error {
if val := os.Getenv("WORKER_COUNT"); val != "" {
if parsed, err := strconv.Atoi(val); err == nil {
c.Core.WorkerCount = parsed
}
}
if val := os.Getenv("REFRESH_DELAY_SECONDS"); val != "" {
if parsed, err := strconv.Atoi(val); err == nil {
c.Core.RefreshDelay = time.Duration(parsed) * time.Second
}
}
if val := os.Getenv("NAS_OUTPUT_PATH"); val != "" {
c.NAS.OutputPath = val
}
if val := os.Getenv("NAS_USERNAME"); val != "" {
c.NAS.Username = val
}
if val := os.Getenv("NAS_PASSWORD"); val != "" {
c.NAS.Password = val
}
if val := os.Getenv("ENABLE_NAS_TRANSFER"); val != "" {
c.NAS.EnableTransfer = val == "true"
}
if val := os.Getenv("LOCAL_OUTPUT_DIR"); val != "" {
c.Paths.LocalOutput = val
}
if val := os.Getenv("PROCESS_OUTPUT_DIR"); val != "" {
c.Paths.ProcessOutput = val
}
if val := os.Getenv("FFMPEG_PATH"); val != "" {
c.Processing.FFmpegPath = val
}
if val := os.Getenv("LOG_DIR"); val != "" {
c.Logging.Dir = val
}
if val := os.Getenv("LOG_LEVEL"); val != "" {
c.Logging.Level = val
}
if val := os.Getenv("HEALTH_ENABLED"); val != "" {
c.Health.Enabled = val == "true"
}
if val := os.Getenv("HEALTH_ADDR"); val != "" {
c.Health.Addr = val
}
return nil
}
func (c *Config) resolveAndValidatePaths() error {
cwd, err := os.Getwd()
if err != nil {
return fmt.Errorf("failed to get working directory: %w", err)
}
// Only join with cwd if path is not already absolute
if !filepath.IsAbs(c.Paths.BaseDir) {
c.Paths.BaseDir = filepath.Join(cwd, c.Paths.BaseDir)
}
if !filepath.IsAbs(c.Paths.LocalOutput) {
c.Paths.LocalOutput = filepath.Join(cwd, c.Paths.LocalOutput)
}
if !filepath.IsAbs(c.Paths.ProcessOutput) {
c.Paths.ProcessOutput = filepath.Join(cwd, c.Paths.ProcessOutput)
}
if !filepath.IsAbs(c.Paths.ManifestDir) {
c.Paths.ManifestDir = filepath.Join(cwd, c.Paths.ManifestDir)
}
if !filepath.IsAbs(c.Paths.PersistenceFile) {
c.Paths.PersistenceFile = filepath.Join(c.Paths.BaseDir, c.Paths.PersistenceFile)
}
if !filepath.IsAbs(c.Logging.Dir) {
c.Logging.Dir = filepath.Join(cwd, c.Logging.Dir)
}
requiredDirs := []string{
c.Paths.BaseDir,
c.Paths.LocalOutput,
c.Paths.ProcessOutput,
c.Paths.ManifestDir,
c.Logging.Dir,
}
for _, dir := range requiredDirs {
if err := os.MkdirAll(dir, 0755); err != nil {
return fmt.Errorf("failed to create directory %s: %w", dir, err)
}
}
if c.NAS.EnableTransfer && c.NAS.OutputPath == "" {
return fmt.Errorf("NAS output path is required when transfer is enabled")
}
if c.Processing.Enabled && c.Processing.FFmpegPath == "" {
return fmt.Errorf("FFmpeg path is required when processing is enabled")
}
return nil
}
func (c *Config) GetEventPath(eventName string) string {
return filepath.Join(c.Paths.LocalOutput, eventName)
}
func (c *Config) GetManifestPath(eventName string) string {
return filepath.Join(c.Paths.ManifestDir, eventName+".json")
}
func (c *Config) GetNASEventPath(eventName string) string {
return filepath.Join(c.NAS.OutputPath, eventName)
}
func (c *Config) GetProcessOutputPath(eventName string) string {
return filepath.Join(c.Paths.ProcessOutput, eventName)
}
func (c *Config) GetQualityPath(eventName, quality string) string {
return filepath.Join(c.GetEventPath(eventName), quality)
}

181
pkg/config/config_test.go Normal file
View File

@ -0,0 +1,181 @@
package config
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestConfig_Load(t *testing.T) {
// Save original env vars
originalVars := map[string]string{
"WORKER_COUNT": os.Getenv("WORKER_COUNT"),
"NAS_USERNAME": os.Getenv("NAS_USERNAME"),
"LOCAL_OUTPUT_DIR": os.Getenv("LOCAL_OUTPUT_DIR"),
"ENABLE_NAS_TRANSFER": os.Getenv("ENABLE_NAS_TRANSFER"),
}
defer func() {
// Restore original env vars
for key, value := range originalVars {
if value == "" {
os.Unsetenv(key)
} else {
os.Setenv(key, value)
}
}
}()
// Test default config load
cfg, err := Load()
if err != nil {
t.Fatalf("Load() failed: %v", err)
}
// Verify defaults
if cfg.Core.WorkerCount != 4 {
t.Errorf("Expected WorkerCount=4, got %d", cfg.Core.WorkerCount)
}
if cfg.Core.RefreshDelay != 3*time.Second {
t.Errorf("Expected RefreshDelay=3s, got %v", cfg.Core.RefreshDelay)
}
if !cfg.NAS.EnableTransfer {
t.Errorf("Expected NAS.EnableTransfer=true, got false")
}
// Test environment variable override
os.Setenv("WORKER_COUNT", "8")
os.Setenv("NAS_USERNAME", "testuser")
os.Setenv("ENABLE_NAS_TRANSFER", "false")
os.Setenv("LOCAL_OUTPUT_DIR", "custom_data")
cfg2, err := Load()
if err != nil {
t.Fatalf("Load() with env vars failed: %v", err)
}
if cfg2.Core.WorkerCount != 8 {
t.Errorf("Expected WorkerCount=8 from env, got %d", cfg2.Core.WorkerCount)
}
if cfg2.NAS.Username != "testuser" {
t.Errorf("Expected NAS.Username='testuser' from env, got %s", cfg2.NAS.Username)
}
if cfg2.NAS.EnableTransfer {
t.Errorf("Expected NAS.EnableTransfer=false from env, got true")
}
if !strings.Contains(cfg2.Paths.LocalOutput, "custom_data") {
t.Errorf("Expected LocalOutput to contain 'custom_data', got %s", cfg2.Paths.LocalOutput)
}
}
func TestConfig_PathMethods(t *testing.T) {
cfg, err := Load()
if err != nil {
t.Fatalf("Load() failed: %v", err)
}
testEvent := "test-event"
testQuality := "1080p"
// Test GetEventPath
eventPath := cfg.GetEventPath(testEvent)
if !strings.Contains(eventPath, testEvent) {
t.Errorf("GetEventPath should contain event name, got %s", eventPath)
}
// Test GetManifestPath
manifestPath := cfg.GetManifestPath(testEvent)
if !strings.Contains(manifestPath, testEvent) {
t.Errorf("GetManifestPath should contain event name, got %s", manifestPath)
}
if !strings.HasSuffix(manifestPath, ".json") {
t.Errorf("GetManifestPath should end with .json, got %s", manifestPath)
}
// Test GetNASEventPath
nasPath := cfg.GetNASEventPath(testEvent)
if !strings.Contains(nasPath, testEvent) {
t.Errorf("GetNASEventPath should contain event name, got %s", nasPath)
}
// Test GetProcessOutputPath
processPath := cfg.GetProcessOutputPath(testEvent)
if !strings.Contains(processPath, testEvent) {
t.Errorf("GetProcessOutputPath should contain event name, got %s", processPath)
}
// Test GetQualityPath
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
if !strings.Contains(qualityPath, testEvent) {
t.Errorf("GetQualityPath should contain event name, got %s", qualityPath)
}
if !strings.Contains(qualityPath, testQuality) {
t.Errorf("GetQualityPath should contain quality, got %s", qualityPath)
}
}
func TestConfig_PathValidation(t *testing.T) {
// Create a temporary directory for testing
tempDir, err := os.MkdirTemp("", "config_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Set environment variables to use temp directory
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
cfg, err := Load()
if err != nil {
t.Fatalf("Load() failed: %v", err)
}
// Verify directories were created
if _, err := os.Stat(cfg.Paths.LocalOutput); os.IsNotExist(err) {
t.Errorf("LocalOutput directory should have been created: %s", cfg.Paths.LocalOutput)
}
if _, err := os.Stat(cfg.Paths.ProcessOutput); os.IsNotExist(err) {
t.Errorf("ProcessOutput directory should have been created: %s", cfg.Paths.ProcessOutput)
}
}
func TestConfig_ValidationErrors(t *testing.T) {
// Save original env vars
originalNASPath := os.Getenv("NAS_OUTPUT_PATH")
originalFFmpegPath := os.Getenv("FFMPEG_PATH")
defer func() {
if originalNASPath == "" {
os.Unsetenv("NAS_OUTPUT_PATH")
} else {
os.Setenv("NAS_OUTPUT_PATH", originalNASPath)
}
if originalFFmpegPath == "" {
os.Unsetenv("FFMPEG_PATH")
} else {
os.Setenv("FFMPEG_PATH", originalFFmpegPath)
}
}()
// Note: Validation tests are limited because the default config
// has working defaults. We can test that Load() works with valid configs.
// Test that Load works with proper paths set
tempDir2, err := os.MkdirTemp("", "config_validation_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir2)
os.Setenv("NAS_OUTPUT_PATH", "\\\\test\\path")
os.Setenv("LOCAL_OUTPUT_DIR", tempDir2)
cfg, err := Load()
if err != nil {
t.Errorf("Load() should work with valid config: %v", err)
}
if cfg == nil {
t.Error("Config should not be nil")
}
}

View File

@ -1,29 +1,51 @@
package constants package constants
import "time" import (
"m3u8-downloader/pkg/config"
"sync"
)
var (
globalConfig *config.Config
configOnce sync.Once
configError error
)
func GetConfig() (*config.Config, error) {
configOnce.Do(func() {
globalConfig, configError = config.Load()
})
return globalConfig, configError
}
func MustGetConfig() *config.Config {
cfg, err := GetConfig()
if err != nil {
panic("Failed to load configuration: " + err.Error())
}
return cfg
}
const ( const (
MasterURL = "https://live-fastly.flosports.tv/streams/mr159021-260419/playlist.m3u8?token=st%3D1753571418%7Eexp%3D1753571448%7Eacl%3D%2Fstreams%2Fmr159021-260419%2Fplaylist.m3u8%7Edata%3Dssai%3A0%3BuserId%3A14025903%3BstreamId%3A260419%3BmediaPackageRegion%3Afalse%3BdvrMinutes%3A360%3BtokenId%3Abadd289a-ade5-48fe-852f-7dbd1d57aca8%3Bpv%3A86400%7Ehmac2%3D8de65c26b185084a6be77e788cb0ba41be5fcac3ab86159b06f7572ca925d77ba7bd182124af2a432953d4223548f198742d1a238e937d875976cd42fe549838&mid_origin=media_store&keyName=FLOSPORTS_TOKEN_KEY_2023-08-02&streamCode=mr159021-260419"
WorkerCount = 4 WorkerCount = 4
RefreshDelay = 3 * time.Second RefreshDelay = 3
HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36" HTTPUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
REFERRER = "https://www.flomarching.com" REFERRER = "https://www.flomarching.com"
OutputDirPath = "./data/flo_radio"
EnableNASTransfer = true DefaultNASOutputPath = "\\\\HomeLabNAS\\dci\\streams"
NASPath = "\\\\HomeLabNAS\\dci\\streams\\2025_Atlanta" DefaultNASUsername = "NASAdmin"
NASUsername = ""
NASPassword = ""
TransferWorkerCount = 2
TransferRetryLimit = 3
TransferTimeout = 30 * time.Second
FileSettlingDelay = 5 * time.Second
PersistencePath = "./data/transfer_queue.json"
TransferQueueSize = 1000
BatchSize = 10
CleanupAfterTransfer = true DefaultTransferWorkerCount = 2
CleanupBatchSize = 10 DefaultTransferRetryLimit = 3
RetainLocalHours = 0 DefaultTransferTimeout = 30
DefaultFileSettlingDelay = 5
DefaultTransferQueueSize = 100000
DefaultBatchSize = 1000
DefaultCleanupBatchSize = 1000
DefaultRetainLocalHours = 0
DefaultProcessWorkerCount = 2
DefaultFFmpegPath = "ffmpeg"
) )

View File

@ -0,0 +1,240 @@
package constants
import (
"os"
"strings"
"testing"
"time"
)
func TestGetConfig(t *testing.T) {
// Test successful config loading
cfg, err := GetConfig()
if err != nil {
t.Fatalf("GetConfig() failed: %v", err)
}
if cfg == nil {
t.Fatal("GetConfig() returned nil config")
}
// Test that subsequent calls return the same instance (singleton)
cfg2, err := GetConfig()
if err != nil {
t.Fatalf("Second GetConfig() call failed: %v", err)
}
// Both should be the same instance due to sync.Once
if cfg != cfg2 {
t.Error("GetConfig() should return the same instance (singleton)")
}
}
func TestMustGetConfig(t *testing.T) {
// This should not panic with valid environment
cfg := MustGetConfig()
if cfg == nil {
t.Fatal("MustGetConfig() returned nil")
}
// Verify it returns a properly initialized config
if cfg.Core.WorkerCount <= 0 {
t.Errorf("Expected positive WorkerCount, got %d", cfg.Core.WorkerCount)
}
if cfg.Core.RefreshDelay <= 0 {
t.Errorf("Expected positive RefreshDelay, got %v", cfg.Core.RefreshDelay)
}
}
func TestMustGetConfig_Panic(t *testing.T) {
// We can't easily test the panic scenario without breaking the singleton,
// but we can test that MustGetConfig works normally
defer func() {
if r := recover(); r != nil {
t.Errorf("MustGetConfig() panicked unexpectedly: %v", r)
}
}()
cfg := MustGetConfig()
if cfg == nil {
t.Fatal("MustGetConfig() returned nil without panicking")
}
}
func TestConfigSingleton(t *testing.T) {
// Reset the singleton for this test (this is a bit hacky but necessary for testing)
// We'll create multiple goroutines to test concurrent access
configs := make(chan interface{}, 10)
// Launch multiple goroutines to call GetConfig concurrently
for i := 0; i < 10; i++ {
go func() {
cfg, _ := GetConfig()
configs <- cfg
}()
}
// Collect all configs
var allConfigs []interface{}
for i := 0; i < 10; i++ {
allConfigs = append(allConfigs, <-configs)
}
// All should be the same instance
firstConfig := allConfigs[0]
for i, cfg := range allConfigs {
if cfg != firstConfig {
t.Errorf("Config %d is different from first config", i)
}
}
}
func TestConstants_Values(t *testing.T) {
// Test that constants have expected values
if WorkerCount != 4 {
t.Errorf("Expected WorkerCount=4, got %d", WorkerCount)
}
if RefreshDelay != 3 {
t.Errorf("Expected RefreshDelay=3, got %d", RefreshDelay)
}
// Test HTTP constants
if HTTPUserAgent == "" {
t.Error("HTTPUserAgent should not be empty")
}
if !strings.Contains(HTTPUserAgent, "Mozilla") {
t.Error("HTTPUserAgent should contain 'Mozilla'")
}
if REFERRER != "https://www.flomarching.com" {
t.Errorf("Expected REFERRER='https://www.flomarching.com', got '%s'", REFERRER)
}
// Test default NAS constants
if DefaultNASOutputPath != "\\\\HomeLabNAS\\dci\\streams" {
t.Errorf("Expected DefaultNASOutputPath='\\\\HomeLabNAS\\dci\\streams', got '%s'", DefaultNASOutputPath)
}
if DefaultNASUsername != "NASAdmin" {
t.Errorf("Expected DefaultNASUsername='NASAdmin', got '%s'", DefaultNASUsername)
}
// Test transfer constants
if DefaultTransferWorkerCount != 2 {
t.Errorf("Expected DefaultTransferWorkerCount=2, got %d", DefaultTransferWorkerCount)
}
if DefaultTransferRetryLimit != 3 {
t.Errorf("Expected DefaultTransferRetryLimit=3, got %d", DefaultTransferRetryLimit)
}
if DefaultTransferTimeout != 30 {
t.Errorf("Expected DefaultTransferTimeout=30, got %d", DefaultTransferTimeout)
}
if DefaultFileSettlingDelay != 5 {
t.Errorf("Expected DefaultFileSettlingDelay=5, got %d", DefaultFileSettlingDelay)
}
if DefaultTransferQueueSize != 100000 {
t.Errorf("Expected DefaultTransferQueueSize=100000, got %d", DefaultTransferQueueSize)
}
if DefaultBatchSize != 1000 {
t.Errorf("Expected DefaultBatchSize=1000, got %d", DefaultBatchSize)
}
// Test cleanup constants
if DefaultCleanupBatchSize != 1000 {
t.Errorf("Expected DefaultCleanupBatchSize=1000, got %d", DefaultCleanupBatchSize)
}
if DefaultRetainLocalHours != 0 {
t.Errorf("Expected DefaultRetainLocalHours=0, got %d", DefaultRetainLocalHours)
}
// Test processing constants
if DefaultProcessWorkerCount != 2 {
t.Errorf("Expected DefaultProcessWorkerCount=2, got %d", DefaultProcessWorkerCount)
}
if DefaultFFmpegPath != "ffmpeg" {
t.Errorf("Expected DefaultFFmpegPath='ffmpeg', got '%s'", DefaultFFmpegPath)
}
}
func TestConfig_Integration(t *testing.T) {
cfg := MustGetConfig()
// Test that config values match or override constants appropriately
if cfg.Core.WorkerCount != WorkerCount && os.Getenv("WORKER_COUNT") == "" {
t.Errorf("Config WorkerCount (%d) should match constant (%d) when no env override", cfg.Core.WorkerCount, WorkerCount)
}
if cfg.Core.RefreshDelay != time.Duration(RefreshDelay)*time.Second && os.Getenv("REFRESH_DELAY_SECONDS") == "" {
t.Errorf("Config RefreshDelay (%v) should match constant (%v) when no env override", cfg.Core.RefreshDelay, time.Duration(RefreshDelay)*time.Second)
}
// Test HTTP settings
if cfg.HTTP.UserAgent != HTTPUserAgent {
t.Errorf("Config UserAgent (%s) should match constant (%s)", cfg.HTTP.UserAgent, HTTPUserAgent)
}
if cfg.HTTP.Referer != REFERRER {
t.Errorf("Config Referer (%s) should match constant (%s)", cfg.HTTP.Referer, REFERRER)
}
}
func TestConfig_PathMethods(t *testing.T) {
cfg := MustGetConfig()
testEvent := "test-event-123"
testQuality := "1080p"
// Test GetEventPath
eventPath := cfg.GetEventPath(testEvent)
if !strings.Contains(eventPath, testEvent) {
t.Errorf("GetEventPath should contain event name '%s', got: %s", testEvent, eventPath)
}
// Test GetManifestPath
manifestPath := cfg.GetManifestPath(testEvent)
if !strings.Contains(manifestPath, testEvent) {
t.Errorf("GetManifestPath should contain event name '%s', got: %s", testEvent, manifestPath)
}
if !strings.HasSuffix(manifestPath, ".json") {
t.Errorf("GetManifestPath should end with '.json', got: %s", manifestPath)
}
// Test GetNASEventPath
nasPath := cfg.GetNASEventPath(testEvent)
if !strings.Contains(nasPath, testEvent) {
t.Errorf("GetNASEventPath should contain event name '%s', got: %s", testEvent, nasPath)
}
// Test GetProcessOutputPath
processPath := cfg.GetProcessOutputPath(testEvent)
if !strings.Contains(processPath, testEvent) {
t.Errorf("GetProcessOutputPath should contain event name '%s', got: %s", testEvent, processPath)
}
// Test GetQualityPath
qualityPath := cfg.GetQualityPath(testEvent, testQuality)
if !strings.Contains(qualityPath, testEvent) {
t.Errorf("GetQualityPath should contain event name '%s', got: %s", testEvent, qualityPath)
}
if !strings.Contains(qualityPath, testQuality) {
t.Errorf("GetQualityPath should contain quality '%s', got: %s", testQuality, qualityPath)
}
}
func TestConfig_DefaultValues(t *testing.T) {
cfg := MustGetConfig()
// Test that default values are reasonable
if cfg.Transfer.QueueSize != DefaultTransferQueueSize {
t.Errorf("Expected transfer queue size %d, got %d", DefaultTransferQueueSize, cfg.Transfer.QueueSize)
}
if cfg.Transfer.BatchSize != DefaultBatchSize {
t.Errorf("Expected transfer batch size %d, got %d", DefaultBatchSize, cfg.Transfer.BatchSize)
}
if cfg.Processing.WorkerCount != DefaultProcessWorkerCount {
t.Errorf("Expected processing worker count %d, got %d", DefaultProcessWorkerCount, cfg.Processing.WorkerCount)
}
if cfg.Cleanup.BatchSize != DefaultCleanupBatchSize {
t.Errorf("Expected cleanup batch size %d, got %d", DefaultCleanupBatchSize, cfg.Cleanup.BatchSize)
}
}

323
pkg/health/dashboard.html Normal file
View File

@ -0,0 +1,323 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>m3u8-downloader status</title>
<style>
:root {
--bg: #0f1115;
--panel: #161922;
--border: #262b38;
--text: #e6e8ee;
--muted: #8b909e;
--green: #3ecf8e;
--red: #f2545b;
--amber: #f2b705;
--blue: #4f8ef7;
}
* { box-sizing: border-box; }
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: var(--bg);
color: var(--text);
}
header {
display: flex;
align-items: center;
justify-content: space-between;
flex-wrap: wrap;
gap: 12px;
padding: 16px 24px;
border-bottom: 1px solid var(--border);
}
header h1 {
font-size: 18px;
margin: 0;
font-weight: 600;
}
.header-meta {
display: flex;
align-items: center;
gap: 16px;
color: var(--muted);
font-size: 13px;
}
.badge {
display: inline-flex;
align-items: center;
gap: 6px;
padding: 4px 12px;
border-radius: 999px;
font-size: 13px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.03em;
}
.badge-dot { width: 8px; height: 8px; border-radius: 50%; }
.badge.healthy { background: rgba(62,207,142,0.15); color: var(--green); }
.badge.healthy .badge-dot { background: var(--green); }
.badge.degraded { background: rgba(242,84,91,0.15); color: var(--red); }
.badge.degraded .badge-dot { background: var(--red); }
.badge.unknown { background: rgba(139,144,158,0.15); color: var(--muted); }
.badge.unknown .badge-dot { background: var(--muted); }
#conn-banner {
display: none;
background: var(--red);
color: #fff;
text-align: center;
padding: 8px;
font-size: 13px;
font-weight: 600;
}
#conn-banner.show { display: block; }
main {
padding: 24px;
display: grid;
grid-template-columns: repeat(auto-fit, minmax(340px, 1fr));
gap: 20px;
max-width: 1400px;
margin: 0 auto;
}
.card {
background: var(--panel);
border: 1px solid var(--border);
border-radius: 10px;
padding: 18px 20px;
}
.card h2 {
font-size: 14px;
text-transform: uppercase;
letter-spacing: 0.05em;
color: var(--muted);
margin: 0 0 14px 0;
display: flex;
align-items: center;
justify-content: space-between;
}
table { width: 100%; border-collapse: collapse; font-size: 13px; }
th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border); }
th { color: var(--muted); font-weight: 500; font-size: 12px; }
tr.failed-row { background: rgba(242,84,91,0.08); }
tr.inactive-row { opacity: 0.5; }
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 6px; }
.dot.on { background: var(--green); }
.dot.off { background: var(--red); }
.stat-grid {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 10px;
margin-bottom: 14px;
}
.stat {
background: rgba(255,255,255,0.03);
border-radius: 8px;
padding: 10px 12px;
}
.stat .value { font-size: 20px; font-weight: 600; }
.stat .label { font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: 0.03em; }
.stat.failed .value { color: var(--red); }
.state-pill {
display: inline-block;
padding: 3px 10px;
border-radius: 999px;
font-size: 12px;
font-weight: 600;
text-transform: uppercase;
}
.state-idle { background: rgba(139,144,158,0.15); color: var(--muted); }
.state-scanning, .state-running { background: rgba(79,142,247,0.15); color: var(--blue); }
.state-completed { background: rgba(62,207,142,0.15); color: var(--green); }
.state-failed { background: rgba(242,84,91,0.15); color: var(--red); }
.kv { display: flex; justify-content: space-between; padding: 4px 0; font-size: 13px; border-bottom: 1px solid var(--border); }
.kv:last-child { border-bottom: none; }
.kv .k { color: var(--muted); }
.error-list { margin: 10px 0 0 0; padding: 0; list-style: none; }
.error-list li {
font-size: 12px;
padding: 6px 8px;
margin-bottom: 4px;
background: rgba(242,84,91,0.08);
border-left: 3px solid var(--red);
border-radius: 4px;
color: var(--text);
}
.error-list .err-time { color: var(--muted); margin-right: 6px; }
.empty-note { color: var(--muted); font-size: 13px; font-style: italic; }
</style>
</head>
<body>
<div id="conn-banner">Lost connection to status endpoint - retrying...</div>
<header>
<h1>m3u8-downloader</h1>
<div class="header-meta">
<span id="overall-badge" class="badge unknown"><span class="badge-dot"></span>unknown</span>
<span id="uptime">uptime: -</span>
<span id="last-updated">last updated: -</span>
</div>
</header>
<main>
<section class="card">
<h2>Download</h2>
<table>
<thead>
<tr><th>Resolution</th><th>Active</th><th>OK</th><th>Failed</th><th>Last activity</th></tr>
</thead>
<tbody id="variant-rows"></tbody>
</table>
<div id="download-empty" class="empty-note" style="display:none;">No variants reporting yet.</div>
<ul id="download-errors" class="error-list"></ul>
</section>
<section class="card">
<h2>Transfer</h2>
<div class="kv"><span class="k">NAS connection</span><span><span id="nas-dot" class="dot off"></span><span id="nas-text">unknown</span></span></div>
<div class="stat-grid" style="margin-top:12px;">
<div class="stat"><div class="value" id="t-added">-</div><div class="label">Added</div></div>
<div class="stat"><div class="value" id="t-completed">-</div><div class="label">Completed</div></div>
<div class="stat failed"><div class="value" id="t-failed">-</div><div class="label">Failed</div></div>
<div class="stat"><div class="value" id="t-pending">-</div><div class="label">Pending</div></div>
<div class="stat"><div class="value" id="t-queue">-</div><div class="label">Queue size</div></div>
<div class="stat"><div class="value" id="t-cleanup">-</div><div class="label">Cleanup pending</div></div>
</div>
<div class="kv"><span class="k">Transferred</span><span id="t-bytes">-</span></div>
<ul id="transfer-errors" class="error-list"></ul>
</section>
<section class="card">
<h2>Processing</h2>
<div class="kv"><span class="k">State</span><span id="p-state" class="state-pill state-idle">idle</span></div>
<div class="kv"><span class="k">Event</span><span id="p-event">-</span></div>
<div class="kv"><span class="k">Segments found</span><span id="p-segments">-</span></div>
<div class="kv"><span class="k">Started</span><span id="p-started">-</span></div>
<div class="kv"><span class="k">Finished</span><span id="p-finished">-</span></div>
<ul id="processing-errors" class="error-list"></ul>
</section>
</main>
<script>
const POLL_INTERVAL_MS = 3000;
const connBanner = document.getElementById('conn-banner');
function fmtBytes(n) {
if (!n || n <= 0) return '0 B';
const units = ['B', 'KB', 'MB', 'GB', 'TB'];
let i = 0;
while (n >= 1024 && i < units.length - 1) { n /= 1024; i++; }
return n.toFixed(i === 0 ? 0 : 1) + ' ' + units[i];
}
function fmtDuration(sec) {
sec = Math.floor(sec || 0);
const h = Math.floor(sec / 3600);
const m = Math.floor((sec % 3600) / 60);
const s = sec % 60;
const parts = [];
if (h) parts.push(h + 'h');
if (m || h) parts.push(m + 'm');
parts.push(s + 's');
return parts.join(' ');
}
function fmtTime(iso) {
if (!iso) return '-';
const d = new Date(iso);
if (isNaN(d.getTime()) || d.getFullYear() < 1971) return '-';
return d.toLocaleTimeString();
}
function renderErrors(listEl, errors) {
listEl.innerHTML = '';
(errors || []).slice().reverse().forEach(e => {
const li = document.createElement('li');
const t = document.createElement('span');
t.className = 'err-time';
t.textContent = fmtTime(e.time);
li.appendChild(t);
li.appendChild(document.createTextNode(e.message));
listEl.appendChild(li);
});
}
function render(data) {
const badge = document.getElementById('overall-badge');
badge.className = 'badge ' + (data.status === 'healthy' ? 'healthy' : 'degraded');
badge.innerHTML = '<span class="badge-dot"></span>' + data.status;
document.getElementById('uptime').textContent = 'uptime: ' + fmtDuration(data.uptime_seconds);
document.getElementById('last-updated').textContent = 'last updated: ' + new Date().toLocaleTimeString();
// Download
const variants = (data.download && data.download.variants) || [];
const rows = document.getElementById('variant-rows');
rows.innerHTML = '';
document.getElementById('download-empty').style.display = variants.length ? 'none' : 'block';
variants.forEach(v => {
const tr = document.createElement('tr');
if (v.segments_failed > 0) tr.classList.add('failed-row');
if (!v.active) tr.classList.add('inactive-row');
tr.innerHTML =
'<td>' + v.resolution + '</td>' +
'<td><span class="dot ' + (v.active ? 'on' : 'off') + '"></span>' + (v.active ? 'active' : 'stopped') + '</td>' +
'<td>' + v.segments_ok + '</td>' +
'<td>' + v.segments_failed + '</td>' +
'<td>' + fmtTime(v.last_activity) + '</td>';
rows.appendChild(tr);
});
renderErrors(document.getElementById('download-errors'), data.download && data.download.recent_errors);
// Transfer
const t = data.transfer || {};
const nasDot = document.getElementById('nas-dot');
nasDot.className = 'dot ' + (t.nas_connected ? 'on' : 'off');
document.getElementById('nas-text').textContent = t.nas_connected ? 'connected' : 'disconnected';
document.getElementById('t-added').textContent = t.added ?? '-';
document.getElementById('t-completed').textContent = t.completed ?? '-';
document.getElementById('t-failed').textContent = t.failed ?? '-';
document.getElementById('t-pending').textContent = t.pending ?? '-';
document.getElementById('t-queue').textContent = t.queue_size ?? '-';
document.getElementById('t-cleanup').textContent = t.cleanup_pending ?? '-';
document.getElementById('t-bytes').textContent = fmtBytes(t.bytes_transferred);
renderErrors(document.getElementById('transfer-errors'), t.recent_errors);
// Processing
const p = data.processing || {};
const stateEl = document.getElementById('p-state');
const state = p.state || 'idle';
stateEl.textContent = state;
stateEl.className = 'state-pill state-' + state;
document.getElementById('p-event').textContent = p.event_name || '-';
document.getElementById('p-segments').textContent = p.segments_found ?? '-';
document.getElementById('p-started').textContent = fmtTime(p.started_at);
document.getElementById('p-finished').textContent = fmtTime(p.finished_at);
renderErrors(document.getElementById('processing-errors'), p.recent_errors);
}
async function poll() {
try {
const resp = await fetch('/status', { cache: 'no-store' });
if (!resp.ok) throw new Error('status ' + resp.status);
const data = await resp.json();
connBanner.classList.remove('show');
render(data);
} catch (e) {
connBanner.classList.add('show');
}
}
poll();
setInterval(poll, POLL_INTERVAL_MS);
</script>
</body>
</html>

352
pkg/health/registry.go Normal file
View File

@ -0,0 +1,352 @@
package health
import (
"sort"
"sync"
"time"
)
const maxRecentErrors = 10
type ErrorEntry struct {
Time time.Time `json:"time"`
Message string `json:"message"`
}
type VariantStatus struct {
Resolution string `json:"resolution"`
Bandwidth uint32 `json:"bandwidth"`
Active bool `json:"active"`
SegmentsOK int `json:"segments_ok"`
SegmentsFailed int `json:"segments_failed"`
LastActivity time.Time `json:"last_activity"`
LastError string `json:"last_error,omitempty"`
}
// StatsProvider is satisfied by *transfer.QueueStats without pkg/health
// needing to import pkg/transfer (which would create an import cycle, since
// pkg/transfer calls into pkg/health to report status).
type StatsProvider interface {
GetStats() (added, completed, failed, pending int, bytes int64)
}
type downloadState struct {
variants map[string]*VariantStatus
recentErrors []ErrorEntry
}
type transferState struct {
statsProvider StatsProvider
queueSizeFn func() int
cleanupPendingFn func() int
nasConnectedFn func() bool
nasConnected bool
recentErrors []ErrorEntry
}
type processingState struct {
state string // idle | scanning | running | completed | failed
eventName string
segmentsFound int
startedAt time.Time
finishedAt time.Time
lastError string
recentErrors []ErrorEntry
}
// Registry is the process-wide health/metrics store. All subsystems report
// into it via the Record* methods; the health HTTP server reads it via
// Snapshot(). Safe for concurrent use.
type Registry struct {
mu sync.RWMutex
startedAt time.Time
download downloadState
transfer transferState
processing processingState
}
var (
globalRegistry *Registry
registryOnce sync.Once
)
// MustGetRegistry returns the process-wide registry, mirroring the
// constants.MustGetConfig() / logging.MustGetFactory() singleton style.
func MustGetRegistry() *Registry {
registryOnce.Do(func() {
globalRegistry = New()
})
return globalRegistry
}
func New() *Registry {
return &Registry{
startedAt: time.Now(),
download: downloadState{
variants: make(map[string]*VariantStatus),
},
processing: processingState{
state: "idle",
},
}
}
func appendBounded(errs []ErrorEntry, e ErrorEntry) []ErrorEntry {
errs = append(errs, e)
if len(errs) > maxRecentErrors {
errs = errs[len(errs)-maxRecentErrors:]
}
return errs
}
func errString(err error) string {
if err == nil {
return ""
}
return err.Error()
}
// --- Download ---
func (r *Registry) RecordVariantStarted(resolution string, bandwidth uint32) {
r.mu.Lock()
defer r.mu.Unlock()
r.download.variants[resolution] = &VariantStatus{
Resolution: resolution,
Bandwidth: bandwidth,
Active: true,
LastActivity: time.Now(),
}
}
func (r *Registry) RecordVariantStopped(resolution string) {
r.mu.Lock()
defer r.mu.Unlock()
if v, ok := r.download.variants[resolution]; ok {
v.Active = false
}
}
func (r *Registry) RecordSegmentDownloaded(resolution string) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.SegmentsOK++
v.LastActivity = time.Now()
}
func (r *Registry) RecordSegmentFailed(resolution string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.SegmentsFailed++
v.LastActivity = time.Now()
v.LastError = errString(err)
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
Time: time.Now(),
Message: resolution + ": " + errString(err),
})
}
func (r *Registry) RecordPlaylistError(resolution string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
v := r.ensureVariantLocked(resolution)
v.LastError = errString(err)
r.download.recentErrors = appendBounded(r.download.recentErrors, ErrorEntry{
Time: time.Now(),
Message: resolution + " playlist: " + errString(err),
})
}
func (r *Registry) ensureVariantLocked(resolution string) *VariantStatus {
v, ok := r.download.variants[resolution]
if !ok {
v = &VariantStatus{Resolution: resolution, Active: true}
r.download.variants[resolution] = v
}
return v
}
// --- Transfer ---
// AttachTransfer wires the registry to the live counters/state already
// maintained by pkg/transfer instead of duplicating them. Called once from
// TransferService construction.
func (r *Registry) AttachTransfer(stats StatsProvider, queueSizeFn, cleanupPendingFn func() int, nasConnectedFn func() bool) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.statsProvider = stats
r.transfer.queueSizeFn = queueSizeFn
r.transfer.cleanupPendingFn = cleanupPendingFn
r.transfer.nasConnectedFn = nasConnectedFn
}
func (r *Registry) RecordNASConnectivity(connected bool) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.nasConnected = connected
}
// RecordTransferError captures failures QueueStats' cumulative counters
// don't individually retain (pre-queue failures, cleanup failures) so they
// still surface in the bounded recent-errors list.
func (r *Registry) RecordTransferError(sourcePath string, err error) {
r.mu.Lock()
defer r.mu.Unlock()
r.transfer.recentErrors = appendBounded(r.transfer.recentErrors, ErrorEntry{
Time: time.Now(),
Message: sourcePath + ": " + errString(err),
})
}
// --- Processing ---
func (r *Registry) RecordProcessingStageChange(state, eventName string) {
r.mu.Lock()
defer r.mu.Unlock()
r.processing.state = state
if eventName != "" {
r.processing.eventName = eventName
}
switch state {
case "scanning":
r.processing.startedAt = time.Now()
r.processing.finishedAt = time.Time{}
r.processing.lastError = ""
case "completed", "failed":
r.processing.finishedAt = time.Now()
}
}
func (r *Registry) RecordProcessingSegmentsFound(count int) {
r.mu.Lock()
defer r.mu.Unlock()
r.processing.segmentsFound = count
}
func (r *Registry) RecordProcessingError(err error) {
r.mu.Lock()
defer r.mu.Unlock()
msg := errString(err)
r.processing.lastError = msg
r.processing.recentErrors = appendBounded(r.processing.recentErrors, ErrorEntry{
Time: time.Now(),
Message: msg,
})
}
// --- Snapshot ---
type DownloadSnapshot struct {
Variants []*VariantStatus `json:"variants"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type TransferSnapshot struct {
NASConnected bool `json:"nas_connected"`
QueueSize int `json:"queue_size"`
CleanupPending int `json:"cleanup_pending"`
Added int `json:"added"`
Completed int `json:"completed"`
Failed int `json:"failed"`
Pending int `json:"pending"`
BytesTransferred int64 `json:"bytes_transferred"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type ProcessingSnapshot struct {
State string `json:"state"`
EventName string `json:"event_name,omitempty"`
SegmentsFound int `json:"segments_found"`
StartedAt *time.Time `json:"started_at,omitempty"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
LastError string `json:"last_error,omitempty"`
RecentErrors []ErrorEntry `json:"recent_errors"`
}
type Snapshot struct {
Status string `json:"status"`
UptimeSec float64 `json:"uptime_seconds"`
Download DownloadSnapshot `json:"download"`
Transfer TransferSnapshot `json:"transfer"`
Processing ProcessingSnapshot `json:"processing"`
}
func (r *Registry) Snapshot() Snapshot {
r.mu.RLock()
defer r.mu.RUnlock()
variants := make([]*VariantStatus, 0, len(r.download.variants))
for _, v := range r.download.variants {
vCopy := *v
variants = append(variants, &vCopy)
}
sort.Slice(variants, func(i, j int) bool { return variants[i].Resolution < variants[j].Resolution })
dl := DownloadSnapshot{
Variants: variants,
RecentErrors: append([]ErrorEntry(nil), r.download.recentErrors...),
}
tr := TransferSnapshot{
NASConnected: r.transfer.nasConnected,
RecentErrors: append([]ErrorEntry(nil), r.transfer.recentErrors...),
}
if r.transfer.statsProvider != nil {
added, completed, failed, pending, bytes := r.transfer.statsProvider.GetStats()
tr.Added, tr.Completed, tr.Failed, tr.Pending, tr.BytesTransferred = added, completed, failed, pending, bytes
}
if r.transfer.queueSizeFn != nil {
tr.QueueSize = r.transfer.queueSizeFn()
}
if r.transfer.cleanupPendingFn != nil {
tr.CleanupPending = r.transfer.cleanupPendingFn()
}
if r.transfer.nasConnectedFn != nil {
tr.NASConnected = r.transfer.nasConnectedFn()
}
proc := ProcessingSnapshot{
State: r.processing.state,
EventName: r.processing.eventName,
SegmentsFound: r.processing.segmentsFound,
LastError: r.processing.lastError,
RecentErrors: append([]ErrorEntry(nil), r.processing.recentErrors...),
}
if !r.processing.startedAt.IsZero() {
t := r.processing.startedAt
proc.StartedAt = &t
}
if !r.processing.finishedAt.IsZero() {
t := r.processing.finishedAt
proc.FinishedAt = &t
}
snap := Snapshot{
UptimeSec: time.Since(r.startedAt).Seconds(),
Download: dl,
Transfer: tr,
Processing: proc,
}
snap.Status = computeStatus(dl, tr, proc)
return snap
}
// computeStatus gives one glanceable field: "degraded" the moment anything
// worth investigating shows up, "healthy" otherwise. Transfer is only
// considered for degradation once it's actually in use (Attached + has seen
// activity), so a download-only run with NAS transfer disabled doesn't
// falsely report degraded.
func computeStatus(dl DownloadSnapshot, tr TransferSnapshot, proc ProcessingSnapshot) string {
if len(dl.RecentErrors) > 0 || len(tr.RecentErrors) > 0 || len(proc.RecentErrors) > 0 {
return "degraded"
}
if !tr.NASConnected && (tr.Added > 0 || tr.QueueSize > 0) {
return "degraded"
}
if proc.State == "failed" {
return "degraded"
}
return "healthy"
}

60
pkg/health/server.go Normal file
View File

@ -0,0 +1,60 @@
package health
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/logging"
"net"
"net/http"
)
//go:embed dashboard.html
var dashboardHTML []byte
// Start binds the health/status HTTP server and returns immediately once the
// listener is bound, so callers see bind failures (e.g. port already in use)
// synchronously rather than losing them in a background goroutine. The
// returned *http.Server should be shut down by the caller via Shutdown(ctx).
//
// Security note: cfg.Addr is expected to be a bind-all address (e.g. ":8090")
// so status can be checked from other devices on the local network during a
// live event. There is no authentication or TLS on these endpoints - this is
// only safe on a trusted local network, never exposed to the open internet.
func Start(ctx context.Context, cfg *config.HealthConfig, reg *Registry) (*http.Server, error) {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Write(dashboardHTML)
})
mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(reg.Snapshot()); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
})
ln, err := net.Listen("tcp", cfg.Addr)
if err != nil {
return nil, fmt.Errorf("failed to bind health server to %s: %w", cfg.Addr, err)
}
srv := &http.Server{Handler: mux}
go func() {
if err := srv.Serve(ln); err != nil && err != http.ErrServerClosed {
logging.MustGetFactory().App.Error("health server stopped unexpectedly", logging.KeyError, err.Error())
}
}()
return srv, nil
}

View File

@ -5,6 +5,50 @@ import (
"fmt" "fmt"
) )
// HTTPError represents an HTTP error with status code and message
type HTTPError struct {
StatusCode int
Message string
}
// Error returns the string representation of the HTTP error
func (e *HTTPError) Error() string {
return fmt.Sprintf("HTTP %d: %s", e.StatusCode, e.Message)
}
// Is implements error comparison for errors.Is
func (e *HTTPError) Is(target error) bool {
var httpErr *HTTPError
if errors.As(target, &httpErr) {
return e.StatusCode == httpErr.StatusCode
}
return false
}
// NewHTTPError creates a new HTTP error
func NewHTTPError(statusCode int, message string) error {
return &HTTPError{
StatusCode: statusCode,
Message: message,
}
}
// IsHTTPError checks if an error is an HTTP error
func IsHTTPError(err error) bool {
var httpErr *HTTPError
return errors.As(err, &httpErr)
}
// GetHTTPStatusCode extracts the status code from an HTTP error
func GetHTTPStatusCode(err error) int {
var httpErr *HTTPError
if errors.As(err, &httpErr) {
return httpErr.StatusCode
}
return 0
}
// Legacy support for existing code
type HttpError struct { type HttpError struct {
Code int Code int
} }

View File

@ -0,0 +1,318 @@
package httpClient
import (
"fmt"
"net/http"
"strings"
"testing"
)
func TestHTTPError_Error(t *testing.T) {
tests := []struct {
name string
statusCode int
message string
want string
}{
{
name: "basic http error",
statusCode: 404,
message: "Not Found",
want: "HTTP 404: Not Found",
},
{
name: "server error",
statusCode: 500,
message: "Internal Server Error",
want: "HTTP 500: Internal Server Error",
},
{
name: "unauthorized error",
statusCode: 401,
message: "Unauthorized",
want: "HTTP 401: Unauthorized",
},
{
name: "empty message",
statusCode: 400,
message: "",
want: "HTTP 400: ",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := &HTTPError{
StatusCode: tt.statusCode,
Message: tt.message,
}
got := err.Error()
if got != tt.want {
t.Errorf("HTTPError.Error() = %q, want %q", got, tt.want)
}
})
}
}
func TestHTTPError_Is(t *testing.T) {
err404 := &HTTPError{StatusCode: 404, Message: "Not Found"}
err500 := &HTTPError{StatusCode: 500, Message: "Server Error"}
otherErr404 := &HTTPError{StatusCode: 404, Message: "Different message"}
regularError := fmt.Errorf("regular error")
tests := []struct {
name string
err error
target error
want bool
}{
{
name: "same error instance",
err: err404,
target: err404,
want: true,
},
{
name: "different HTTP errors with same status",
err: err404,
target: otherErr404,
want: true,
},
{
name: "different HTTP errors with different status",
err: err404,
target: err500,
want: false,
},
{
name: "HTTP error vs regular error",
err: err404,
target: regularError,
want: false,
},
{
name: "regular error vs HTTP error",
err: regularError,
target: err404,
want: false,
},
{
name: "nil target",
err: err404,
target: nil,
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var got bool
if httpErr, ok := tt.err.(*HTTPError); ok {
got = httpErr.Is(tt.target)
} else {
got = false // Non-HTTP errors return false
}
if got != tt.want {
t.Errorf("HTTPError.Is() = %v, want %v", got, tt.want)
}
})
}
}
func TestIsHTTPError(t *testing.T) {
httpErr := &HTTPError{StatusCode: 404, Message: "Not Found"}
regularErr := fmt.Errorf("regular error")
tests := []struct {
name string
err error
want bool
}{
{
name: "http error",
err: httpErr,
want: true,
},
{
name: "regular error",
err: regularErr,
want: false,
},
{
name: "nil error",
err: nil,
want: false,
},
{
name: "wrapped http error",
err: fmt.Errorf("wrapped: %w", httpErr),
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := IsHTTPError(tt.err)
if got != tt.want {
t.Errorf("IsHTTPError() = %v, want %v", got, tt.want)
}
})
}
}
func TestGetHTTPStatusCode(t *testing.T) {
tests := []struct {
name string
err error
want int
}{
{
name: "http error 404",
err: &HTTPError{StatusCode: 404, Message: "Not Found"},
want: 404,
},
{
name: "http error 500",
err: &HTTPError{StatusCode: 500, Message: "Server Error"},
want: 500,
},
{
name: "wrapped http error",
err: fmt.Errorf("wrapped: %w", &HTTPError{StatusCode: 403, Message: "Forbidden"}),
want: 403,
},
{
name: "regular error",
err: fmt.Errorf("regular error"),
want: 0,
},
{
name: "nil error",
err: nil,
want: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := GetHTTPStatusCode(tt.err)
if got != tt.want {
t.Errorf("GetHTTPStatusCode() = %v, want %v", got, tt.want)
}
})
}
}
func TestNewHTTPError(t *testing.T) {
statusCode := 404
message := "Page not found"
err := NewHTTPError(statusCode, message)
// Check type
httpErr, ok := err.(*HTTPError)
if !ok {
t.Fatalf("NewHTTPError should return *HTTPError, got %T", err)
}
// Check fields
if httpErr.StatusCode != statusCode {
t.Errorf("Expected StatusCode=%d, got %d", statusCode, httpErr.StatusCode)
}
if httpErr.Message != message {
t.Errorf("Expected Message=%q, got %q", message, httpErr.Message)
}
// Check error string
expectedErrorString := fmt.Sprintf("HTTP %d: %s", statusCode, message)
if httpErr.Error() != expectedErrorString {
t.Errorf("Expected error string=%q, got %q", expectedErrorString, httpErr.Error())
}
}
func TestHTTPError_StatusCodeChecks(t *testing.T) {
tests := []struct {
name string
statusCode int
isClient bool
isServer bool
}{
{"200 OK", 200, false, false},
{"400 Bad Request", 400, true, false},
{"401 Unauthorized", 401, true, false},
{"404 Not Found", 404, true, false},
{"499 Client Error", 499, true, false},
{"500 Server Error", 500, false, true},
{"502 Bad Gateway", 502, false, true},
{"599 Server Error", 599, false, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := &HTTPError{StatusCode: tt.statusCode, Message: "test"}
isClient := err.StatusCode >= 400 && err.StatusCode < 500
isServer := err.StatusCode >= 500 && err.StatusCode < 600
if isClient != tt.isClient {
t.Errorf("Status %d: expected isClient=%v, got %v", tt.statusCode, tt.isClient, isClient)
}
if isServer != tt.isServer {
t.Errorf("Status %d: expected isServer=%v, got %v", tt.statusCode, tt.isServer, isServer)
}
})
}
}
func TestHTTPError_Integration(t *testing.T) {
// Test that HTTPError integrates well with standard error handling
err := NewHTTPError(http.StatusNotFound, "Resource not found")
// Should be able to use with errors.Is
target := &HTTPError{StatusCode: http.StatusNotFound}
if !err.(*HTTPError).Is(target) {
t.Error("HTTPError should match target with same status code")
}
// Should be detectable as HTTPError
if !IsHTTPError(err) {
t.Error("Should be detectable as HTTPError")
}
// Should return correct status code
if GetHTTPStatusCode(err) != http.StatusNotFound {
t.Error("Should return correct status code")
}
// Should have meaningful string representation
errorString := err.Error()
if !strings.Contains(errorString, "404") {
t.Error("Error string should contain status code")
}
if !strings.Contains(errorString, "Resource not found") {
t.Error("Error string should contain message")
}
}
func TestHTTPError_EdgeCases(t *testing.T) {
// Test with zero status code
err := NewHTTPError(0, "Zero status")
if err.Error() != "HTTP 0: Zero status" {
t.Errorf("Unexpected error string for zero status: %s", err.Error())
}
// Test with very long message
longMessage := strings.Repeat("a", 1000)
err = NewHTTPError(500, longMessage)
if !strings.Contains(err.Error(), longMessage) {
t.Error("Long message should be preserved")
}
// Test status code boundaries
for _, code := range []int{399, 400, 499, 500, 599, 600} {
err := &HTTPError{StatusCode: code}
// Should not panic
_ = err.Error()
}
}

72
pkg/logging/factory.go Normal file
View File

@ -0,0 +1,72 @@
package logging
import (
"log/slog"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/constants"
"os"
"path/filepath"
"strings"
"sync"
)
// Factory holds one structured logger per pipeline stage, each writing JSON
// lines to its own file under LoggingConfig.Dir. These are additive to the
// existing stdlib log.* calls, not a replacement for them.
type Factory struct {
App *slog.Logger
Download *slog.Logger
Transfer *slog.Logger
Processing *slog.Logger
}
var (
globalFactory *Factory
factoryOnce sync.Once
)
// MustGetFactory returns the process-wide logging factory, building it from
// the config singleton on first use. Mirrors constants.MustGetConfig().
func MustGetFactory() *Factory {
factoryOnce.Do(func() {
globalFactory = New(&constants.MustGetConfig().Logging)
})
return globalFactory
}
func New(cfg *config.LoggingConfig) *Factory {
level := parseLevel(cfg.Level)
return &Factory{
App: newSubsystemLogger(cfg.Dir, "app.log", "app", level),
Download: newSubsystemLogger(cfg.Dir, "download.log", "download", level),
Transfer: newSubsystemLogger(cfg.Dir, "transfer.log", "transfer", level),
Processing: newSubsystemLogger(cfg.Dir, "processing.log", "processing", level),
}
}
// newSubsystemLogger opens (or creates) the subsystem's log file in append
// mode. A logging misconfiguration must never take down an active recording,
// so failure to open the file falls back to stderr instead of panicking.
func newSubsystemLogger(dir, filename, component string, level slog.Level) *slog.Logger {
path := filepath.Join(dir, filename)
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
fallback := slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: level})).With("component", component)
fallback.Error("failed to open log file, falling back to stderr", KeyFile, path, KeyError, err.Error())
return fallback
}
return slog.New(slog.NewJSONHandler(f, &slog.HandlerOptions{Level: level})).With("component", component)
}
func parseLevel(level string) slog.Level {
switch strings.ToLower(strings.TrimSpace(level)) {
case "debug":
return slog.LevelDebug
case "warn", "warning":
return slog.LevelWarn
case "error":
return slog.LevelError
default:
return slog.LevelInfo
}
}

13
pkg/logging/keys.go Normal file
View File

@ -0,0 +1,13 @@
package logging
const (
KeyResolution = "resolution"
KeyBandwidth = "bandwidth"
KeySegment = "segment"
KeySeq = "seq"
KeyFile = "file"
KeyBytes = "bytes"
KeyEvent = "event"
KeyError = "error"
KeyAttempt = "attempt"
)

90
pkg/media/manifest.go Normal file
View File

@ -0,0 +1,90 @@
package media
import (
"encoding/json"
"log"
"m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/utils"
"os"
"sort"
)
type ManifestWriter struct {
ManifestPath string
Segments []ManifestItem
Index map[string]*ManifestItem
}
type ManifestItem struct {
SeqNo string `json:"seqNo"`
Resolution string `json:"resolution"`
}
func NewManifestWriter(eventName string) *ManifestWriter {
cfg := constants.MustGetConfig()
return &ManifestWriter{
ManifestPath: cfg.GetManifestPath(eventName),
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
}
}
func (m *ManifestWriter) AddOrUpdateSegment(seqNo string, resolution string) {
if m.Index == nil {
m.Index = make(map[string]*ManifestItem)
}
if m.Segments == nil {
m.Segments = make([]ManifestItem, 0)
}
if existing, ok := m.Index[seqNo]; ok {
if resolution > existing.Resolution {
existing.Resolution = resolution
}
return
} else {
item := ManifestItem{
SeqNo: seqNo,
Resolution: resolution,
}
m.Segments = append(m.Segments, item)
m.Index[seqNo] = &item
}
}
func (m *ManifestWriter) WriteManifest() {
sort.Slice(m.Segments, func(i, j int) bool {
return m.Segments[i].SeqNo < m.Segments[j].SeqNo
})
data, err := json.MarshalIndent(m.Segments, "", " ")
if err != nil {
log.Printf("Failed to marshal manifest: %v", err)
logging.MustGetFactory().Download.Error("failed to marshal manifest", logging.KeyError, err.Error())
return
}
if err := utils.ValidateWritablePath(m.ManifestPath); err != nil {
log.Printf("Manifest path validation failed: %v", err)
logging.MustGetFactory().Download.Error("manifest path validation failed", logging.KeyError, err.Error())
return
}
file, err := os.Create(m.ManifestPath)
if err != nil {
log.Printf("Failed to create manifest file: %v", err)
logging.MustGetFactory().Download.Error("failed to create manifest file", logging.KeyError, err.Error())
return
}
defer file.Close()
_, err = file.Write(data)
if err != nil {
log.Printf("Failed to write manifest file: %v", err)
logging.MustGetFactory().Download.Error("failed to write manifest file", logging.KeyError, err.Error())
return
}
}

241
pkg/media/manifest_test.go Normal file
View File

@ -0,0 +1,241 @@
package media
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
func TestManifestWriter_NewManifestWriter(t *testing.T) {
// Set up temporary environment for testing
tempDir, err := os.MkdirTemp("", "manifest_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Set environment variable to use temp directory
os.Setenv("LOCAL_OUTPUT_DIR", tempDir)
defer os.Unsetenv("LOCAL_OUTPUT_DIR")
eventName := "test-event"
writer := NewManifestWriter(eventName)
if writer == nil {
t.Fatal("NewManifestWriter() returned nil")
}
if writer.Segments == nil {
t.Error("Segments should be initialized")
}
if writer.Index == nil {
t.Error("Index should be initialized")
}
if len(writer.Segments) != 0 {
t.Errorf("Segments should be empty, got %d items", len(writer.Segments))
}
}
func TestManifestWriter_AddOrUpdateSegment(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "test.json",
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
}
// Test adding new segment
writer.AddOrUpdateSegment("1001", "1080p")
if len(writer.Segments) != 1 {
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
}
if writer.Segments[0].SeqNo != "1001" {
t.Errorf("Expected SeqNo '1001', got '%s'", writer.Segments[0].SeqNo)
}
if writer.Segments[0].Resolution != "1080p" {
t.Errorf("Expected Resolution '1080p', got '%s'", writer.Segments[0].Resolution)
}
// Test updating existing segment with higher resolution
writer.AddOrUpdateSegment("1001", "1440p")
if len(writer.Segments) != 1 {
t.Errorf("Segments count should remain 1 after update, got %d", len(writer.Segments))
}
if writer.Segments[0].Resolution != "1440p" {
t.Errorf("Expected updated resolution '1440p', got '%s'", writer.Segments[0].Resolution)
}
// Test updating existing segment with lower resolution (should not change)
writer.AddOrUpdateSegment("1001", "720p")
if writer.Segments[0].Resolution != "1440p" {
t.Errorf("Resolution should remain '1440p', got '%s'", writer.Segments[0].Resolution)
}
// Test adding different segment
writer.AddOrUpdateSegment("1002", "720p")
if len(writer.Segments) != 2 {
t.Errorf("Expected 2 segments, got %d", len(writer.Segments))
}
}
func TestManifestWriter_AddOrUpdateSegment_NilFields(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "test.json",
}
// Test with nil fields (should initialize them)
writer.AddOrUpdateSegment("1001", "1080p")
if writer.Segments == nil {
t.Error("Segments should be initialized")
}
if writer.Index == nil {
t.Error("Index should be initialized")
}
if len(writer.Segments) != 1 {
t.Errorf("Expected 1 segment, got %d", len(writer.Segments))
}
}
func TestManifestWriter_WriteManifest(t *testing.T) {
tempDir, err := os.MkdirTemp("", "manifest_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
manifestPath := filepath.Join(tempDir, "test-manifest.json")
writer := &ManifestWriter{
ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
}
// Add some test segments out of order
writer.AddOrUpdateSegment("1003", "1080p")
writer.AddOrUpdateSegment("1001", "720p")
writer.AddOrUpdateSegment("1002", "1080p")
// Write manifest
writer.WriteManifest()
// Verify file was created
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
t.Fatalf("Manifest file was not created: %s", manifestPath)
}
// Read and verify content
content, err := os.ReadFile(manifestPath)
if err != nil {
t.Fatalf("Failed to read manifest file: %v", err)
}
var segments []ManifestItem
err = json.Unmarshal(content, &segments)
if err != nil {
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
}
// Verify segments are sorted by sequence number
if len(segments) != 3 {
t.Errorf("Expected 3 segments in manifest, got %d", len(segments))
}
expectedOrder := []string{"1001", "1002", "1003"}
for i, segment := range segments {
if segment.SeqNo != expectedOrder[i] {
t.Errorf("Segment %d: expected SeqNo '%s', got '%s'", i, expectedOrder[i], segment.SeqNo)
}
}
// Verify content structure
if segments[0].Resolution != "720p" {
t.Errorf("Expected first segment resolution '720p', got '%s'", segments[0].Resolution)
}
if segments[1].Resolution != "1080p" {
t.Errorf("Expected second segment resolution '1080p', got '%s'", segments[1].Resolution)
}
}
func TestManifestWriter_WriteManifest_EmptySegments(t *testing.T) {
tempDir, err := os.MkdirTemp("", "manifest_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
manifestPath := filepath.Join(tempDir, "empty-manifest.json")
writer := &ManifestWriter{
ManifestPath: manifestPath,
Segments: make([]ManifestItem, 0),
Index: make(map[string]*ManifestItem),
}
// Write empty manifest
writer.WriteManifest()
// Verify file was created
if _, err := os.Stat(manifestPath); os.IsNotExist(err) {
t.Fatalf("Manifest file was not created: %s", manifestPath)
}
// Read and verify content is empty array
content, err := os.ReadFile(manifestPath)
if err != nil {
t.Fatalf("Failed to read manifest file: %v", err)
}
var segments []ManifestItem
err = json.Unmarshal(content, &segments)
if err != nil {
t.Fatalf("Failed to unmarshal manifest JSON: %v", err)
}
if len(segments) != 0 {
t.Errorf("Expected empty segments array, got %d items", len(segments))
}
}
func TestManifestWriter_WriteManifest_InvalidPath(t *testing.T) {
writer := &ManifestWriter{
ManifestPath: "/invalid/path/that/does/not/exist/manifest.json",
Segments: []ManifestItem{{SeqNo: "1001", Resolution: "1080p"}},
Index: make(map[string]*ManifestItem),
}
// This should not panic, just fail gracefully
writer.WriteManifest()
// Test passes if no panic occurs
}
func TestManifestItem_JSONSerialization(t *testing.T) {
item := ManifestItem{
SeqNo: "1001",
Resolution: "1080p",
}
// Test marshaling
data, err := json.Marshal(item)
if err != nil {
t.Fatalf("Failed to marshal ManifestItem: %v", err)
}
// Test unmarshaling
var unmarshaled ManifestItem
err = json.Unmarshal(data, &unmarshaled)
if err != nil {
t.Fatalf("Failed to unmarshal ManifestItem: %v", err)
}
if unmarshaled.SeqNo != item.SeqNo {
t.Errorf("SeqNo mismatch: expected '%s', got '%s'", item.SeqNo, unmarshaled.SeqNo)
}
if unmarshaled.Resolution != item.Resolution {
t.Errorf("Resolution mismatch: expected '%s', got '%s'", item.Resolution, unmarshaled.Resolution)
}
}

View File

@ -6,6 +6,7 @@ import (
"io" "io"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/httpClient" "m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@ -33,6 +34,7 @@ func (j SegmentJob) Key() string {
func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error { func DownloadSegment(ctx context.Context, client *http.Client, segmentURL string, outputDir string) error {
for attempt := 0; attempt < 2; attempt++ { for attempt := 0; attempt < 2; attempt++ {
if attempt > 0 { if attempt > 0 {
logging.MustGetFactory().Download.Debug("retrying segment download", logging.KeyAttempt, attempt, "url", segmentURL)
time.Sleep(300 * time.Millisecond) time.Sleep(300 * time.Millisecond)
} }
req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil) req, err := http.NewRequestWithContext(ctx, "GET", segmentURL, nil)

View File

@ -2,11 +2,14 @@ package media
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"github.com/grafov/m3u8" "github.com/grafov/m3u8"
"log" "log"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/httpClient" "m3u8-downloader/pkg/httpClient"
"m3u8-downloader/pkg/logging"
"net/http" "net/http"
"net/url" "net/url"
"path" "path"
@ -21,6 +24,7 @@ type StreamVariant struct {
ID int ID int
Resolution string Resolution string
OutputDir string OutputDir string
Writer *ManifestWriter
} }
func extractResolution(variant *m3u8.Variant) string { func extractResolution(variant *m3u8.Variant) string {
@ -44,19 +48,21 @@ func extractResolution(variant *m3u8.Variant) string {
} }
} }
func GetAllVariants(masterURL string) ([]*StreamVariant, error) { func GetAllVariants(masterURL string, outputDir string, writer *ManifestWriter) ([]*StreamVariant, error) {
client := &http.Client{} client := &http.Client{}
req, _ := http.NewRequest("GET", masterURL, nil) req, _ := http.NewRequest("GET", masterURL, nil)
req.Header.Set("User-Agent", constants.HTTPUserAgent) req.Header.Set("User-Agent", constants.HTTPUserAgent)
req.Header.Set("Referer", constants.REFERRER) req.Header.Set("Referer", constants.REFERRER)
resp, err := client.Do(req) resp, err := client.Do(req)
if err != nil { if err != nil {
logging.MustGetFactory().Download.Error("failed to fetch master playlist", logging.KeyError, err.Error())
return nil, err return nil, err
} }
defer resp.Body.Close() defer resp.Body.Close()
playlist, listType, err := m3u8.DecodeFrom(resp.Body, true) playlist, listType, err := m3u8.DecodeFrom(resp.Body, true)
if err != nil { if err != nil {
logging.MustGetFactory().Download.Error("failed to decode master playlist", logging.KeyError, err.Error())
return nil, err return nil, err
} }
@ -69,7 +75,8 @@ func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
BaseURL: base, BaseURL: base,
ID: 0, ID: 0,
Resolution: "unknown", Resolution: "unknown",
OutputDir: path.Join(constants.NASPath, "unknown"), OutputDir: path.Join(outputDir, "unknown"),
Writer: writer,
}}, nil }}, nil
} }
@ -83,7 +90,7 @@ func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
vURL, _ := url.Parse(v.URI) vURL, _ := url.Parse(v.URI)
fullURL := base.ResolveReference(vURL).String() fullURL := base.ResolveReference(vURL).String()
resolution := extractResolution(v) resolution := extractResolution(v)
outputDir := path.Join(constants.NASPath, resolution) outputDir := path.Join(outputDir, resolution)
variants = append(variants, &StreamVariant{ variants = append(variants, &StreamVariant{
URL: fullURL, URL: fullURL,
Bandwidth: v.Bandwidth, Bandwidth: v.Bandwidth,
@ -93,11 +100,17 @@ func GetAllVariants(masterURL string) ([]*StreamVariant, error) {
OutputDir: outputDir, OutputDir: outputDir,
}) })
} }
logging.MustGetFactory().Download.Info("discovered variants", "count", len(variants))
return variants, nil return variants, nil
} }
func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}) { func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan struct{}, manifest *ManifestWriter) {
log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth) log.Printf("Starting %s variant downloader (bandwidth: %d)", variant.Resolution, variant.Bandwidth)
dlLog := logging.MustGetFactory().Download.With(logging.KeyResolution, variant.Resolution)
registry := health.MustGetRegistry()
dlLog.Info("variant downloader started", logging.KeyBandwidth, variant.Bandwidth)
registry.RecordVariantStarted(variant.Resolution, variant.Bandwidth)
ticker := time.NewTicker(constants.RefreshDelay) ticker := time.NewTicker(constants.RefreshDelay)
defer ticker.Stop() defer ticker.Stop()
client := &http.Client{} client := &http.Client{}
@ -106,16 +119,20 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
registry.RecordVariantStopped(variant.Resolution)
return return
default: default:
} }
var seq uint64
playlist, err := LoadMediaPlaylist(variant.URL) playlist, err := LoadMediaPlaylist(variant.URL)
seq := playlist.SeqNo
if err != nil { if err != nil {
log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err) log.Printf("%s: Error loading playlist playlist: %v", variant.Resolution, err)
dlLog.Warn("failed to load media playlist", logging.KeyError, err.Error())
registry.RecordPlaylistError(variant.Resolution, err)
goto waitTick goto waitTick
} }
seq = playlist.SeqNo
for _, seg := range playlist.Segments { for _, seg := range playlist.Segments {
if seg == nil { if seg == nil {
@ -142,25 +159,42 @@ func VariantDownloader(ctx context.Context, variant *StreamVariant, sem chan str
err := DownloadSegment(ctx, client, j.AbsoluteURL(), j.Variant.OutputDir) err := DownloadSegment(ctx, client, j.AbsoluteURL(), j.Variant.OutputDir)
name := strings.TrimSuffix(path.Base(j.Key()), path.Ext(path.Base(j.Key()))) name := strings.TrimSuffix(path.Base(j.Key()), path.Ext(path.Base(j.Key())))
if err == nil { if err == nil {
log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name) log.Printf("✓ %s downloaded segment %s", j.Variant.Resolution, name)
} else if httpClient.IsHTTPStatus(err, 403) { dlLog.Debug("segment downloaded", logging.KeySegment, name)
registry.RecordSegmentDownloaded(variant.Resolution)
return
}
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
// Suppress log: shutdown in progress
return
}
if httpClient.IsHTTPStatus(err, 403) {
log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name) log.Printf("✗ %s failed to download segment %s (403)", j.Variant.Resolution, name)
dlLog.Warn("segment download failed (403)", logging.KeySegment, name)
} else { } else {
log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err) log.Printf("✗ %s failed to download segment %s: %v", j.Variant.Resolution, name, err)
dlLog.Error("segment download failed", logging.KeySegment, name, logging.KeyError, err.Error())
} }
registry.RecordSegmentFailed(variant.Resolution, err)
}(job) }(job)
seq++ seq++
} }
if playlist.Closed { if playlist.Closed {
log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution) log.Printf("%s: Playlist closed (#EXT-X-ENDLIST)", variant.Resolution)
dlLog.Info("playlist closed")
registry.RecordVariantStopped(variant.Resolution)
return return
} }
waitTick: waitTick:
select { select {
case <-ctx.Done(): case <-ctx.Done():
registry.RecordVariantStopped(variant.Resolution)
return return
case <-ticker.C: case <-ticker.C:
} }

12
pkg/nas/config.go Normal file
View File

@ -0,0 +1,12 @@
package nas
import "time"
type NASConfig struct {
Path string
Username string
Password string
Timeout time.Duration
RetryLimit int
VerifySize bool
}

202
pkg/nas/nas.go Normal file
View File

@ -0,0 +1,202 @@
package nas
import (
"context"
"fmt"
"io"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
)
type NASService struct {
Config NASConfig
connected bool
}
func NewNASService(config NASConfig) *NASService {
nt := &NASService{
Config: config,
}
// Establish network connection with credentials before accessing the path
if err := nt.EstablishConnection(); err != nil {
log.Fatalf("Failed to establish network connection to %s: %v", nt.Config.Path, err)
}
err := nt.EnsureDirectoryExists(nt.Config.Path)
if err != nil {
log.Fatalf("Failed to create directory %s: %v", nt.Config.Path, err)
}
return nt
}
func (nt *NASService) CopyFile(ctx context.Context, srcPath, destPath string) error {
src, err := os.Open(srcPath)
if err != nil {
return fmt.Errorf("Failed to open source file: %w", err)
}
defer src.Close()
dest, err := os.Create(destPath)
if err != nil {
return fmt.Errorf("Failed to create destination file: %w", err)
}
defer dest.Close()
done := make(chan error, 1)
go func() {
_, err := io.Copy(dest, src)
done <- err
}()
select {
case <-ctx.Done():
return ctx.Err()
case err := <-done:
if err != nil {
return err
}
return dest.Sync()
}
}
func (nt *NASService) VerifyTransfer(srcPath, destPath string) error {
srcInfo, err := os.Stat(srcPath)
if err != nil {
return fmt.Errorf("Failed to stat source file: %w", err)
}
destInfo, err := os.Stat(destPath)
if err != nil {
return fmt.Errorf("Failed to stat destination file: %w", err)
}
if srcInfo.Size() != destInfo.Size() {
return fmt.Errorf("size mismatch: source=%d, dest=%d", srcInfo.Size(), destInfo.Size())
}
return nil
}
func (nt *NASService) EnsureDirectoryExists(path string) error {
if err := os.MkdirAll(path, 0755); err != nil {
return fmt.Errorf("Failed to create directory: %w", err)
}
return nil
}
func (nt *NASService) EstablishConnection() error {
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
return nil // local path, no network mount needed
}
log.Printf("Establishing network connection to %s with user %s", networkPath, nt.Config.Username)
var cmd *exec.Cmd
if nt.Config.Username != "" && nt.Config.Password != "" {
cmd = exec.Command("net", "use", networkPath, "/user:"+nt.Config.Username, nt.Config.Password, "/persistent:no")
} else {
cmd = exec.Command("net", "use", networkPath, "/persistent:no")
}
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to establish network connection: %w\nOutput: %s", err, string(output))
}
log.Printf("Network connection established successfully")
return nil
}
func (nt *NASService) ExtractNetworkPath(fullPath string) string {
// Extract \\server\share from paths like \\server\share\folder\subfolder
if !strings.HasPrefix(fullPath, "\\\\") {
return "" // Not a UNC path
}
parts := strings.Split(fullPath[2:], "\\") // Remove leading \\
if len(parts) < 2 {
return "" // Invalid UNC path
}
return "\\\\" + parts[0] + "\\" + parts[1]
}
func (nt *NASService) TestConnection() error {
testFile := filepath.Join(nt.Config.Path, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("Failed to create test file: %w", err)
}
f.Close()
os.Remove(testFile)
nt.connected = true
log.Printf("Connected to NAS at %s", nt.Config.Path)
return nil
}
func (nt *NASService) IsConnected() bool {
return nt.connected
}
// Disconnect removes the network connection
func (nt *NASService) Disconnect() error {
networkPath := nt.ExtractNetworkPath(nt.Config.Path)
if networkPath == "" {
return nil // Local path, nothing to disconnect
}
cmd := exec.Command("net", "use", networkPath, "/delete")
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("Warning: failed to disconnect from %s: %v\nOutput: %s", networkPath, err, string(output))
// Don't return error since this is cleanup
} else {
log.Printf("Disconnected from network path: %s", networkPath)
}
nt.connected = false
return nil
}
// FileExists checks if a file already exists on the NAS and optionally verifies size
func (nt *NASService) FileExists(destinationPath string, expectedSize int64) (bool, error) {
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
destInfo, err := os.Stat(fullDestPath)
if err != nil {
if os.IsNotExist(err) {
return false, nil // File doesn't exist, no error
}
return false, fmt.Errorf("failed to stat NAS file: %w", err)
}
// File exists, check size if expected size is provided
if expectedSize > 0 && destInfo.Size() != expectedSize {
log.Printf("NAS file size mismatch for %s: expected=%d, actual=%d",
fullDestPath, expectedSize, destInfo.Size())
return false, nil // File exists but wrong size, treat as not existing
}
return true, nil
}
// GetFileSize returns the size of a file on the NAS
func (nt *NASService) GetFileSize(destinationPath string) (int64, error) {
fullDestPath := filepath.Join(nt.Config.Path, destinationPath)
destInfo, err := os.Stat(fullDestPath)
if err != nil {
return 0, fmt.Errorf("failed to stat NAS file: %w", err)
}
return destInfo.Size(), nil
}

View File

@ -0,0 +1,7 @@
package processing
type SegmentInfo struct {
Name string
SeqNo int
Resolution string
}

365
pkg/processing/service.go Normal file
View File

@ -0,0 +1,365 @@
package processing
import (
"bufio"
"context"
"fmt"
"log"
"m3u8-downloader/pkg/config"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas"
"m3u8-downloader/pkg/utils"
"os"
"os/exec"
"path/filepath"
"regexp"
"runtime"
"sort"
"strconv"
"strings"
"sync"
)
type ProcessingService struct {
config *config.Config
eventName string
nas *nas.NASService
}
func NewProcessingService(eventName string, cfg *config.Config) (*ProcessingService, error) {
if cfg == nil {
return nil, fmt.Errorf("configuration is required")
}
nasConfig := nas.NASConfig{
Path: cfg.NAS.OutputPath,
Username: cfg.NAS.Username,
Password: cfg.NAS.Password,
Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true,
}
nasService := nas.NewNASService(nasConfig)
if err := nasService.TestConnection(); err != nil {
logging.MustGetFactory().Processing.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
}
return &ProcessingService{
config: cfg,
eventName: eventName,
nas: nasService,
}, nil
}
func (ps *ProcessingService) GetSegmentInfo() (map[int]string, error) {
return nil, nil
}
func (ps *ProcessingService) GetEventDirs() ([]string, error) {
if ps.eventName == "" {
sourcePath := ps.config.NAS.OutputPath
dirs, err := os.ReadDir(sourcePath)
if err != nil {
return nil, fmt.Errorf("failed to read directory %s: %w", sourcePath, err)
}
var eventDirs []string
for _, dir := range dirs {
if dir.IsDir() {
eventDirs = append(eventDirs, dir.Name())
}
}
return eventDirs, nil
} else {
return []string{ps.eventName}, nil
}
}
func (ps *ProcessingService) Start(ctx context.Context) error {
procLog := logging.MustGetFactory().Processing
registry := health.MustGetRegistry()
if !ps.config.Processing.Enabled {
log.Println("Processing service disabled")
return nil
}
registry.RecordProcessingStageChange("scanning", ps.eventName)
// fail centralizes the log/registry bookkeeping for every error return
// site below instead of repeating it at each one.
fail := func(err error) error {
procLog.Error("processing failed", logging.KeyEvent, ps.eventName, logging.KeyError, err.Error())
registry.RecordProcessingStageChange("failed", ps.eventName)
registry.RecordProcessingError(err)
return err
}
if ps.eventName == "" {
events, err := ps.GetEventDirs()
if err != nil {
return fail(fmt.Errorf("failed to get event directories: %w", err))
}
if len(events) == 0 {
return fail(fmt.Errorf("no events found"))
}
if len(events) > 1 {
fmt.Println("Multiple events found, please select one:")
for i, event := range events {
fmt.Printf("%d. %s\n", i+1, event)
}
reader := bufio.NewReader(os.Stdin)
input, _ := reader.ReadString('\n')
input = strings.TrimSpace(input)
index, err := strconv.Atoi(input)
if err != nil {
return fail(fmt.Errorf("failed to parse input: %w", err))
}
if index < 1 || index > len(events) {
return fail(fmt.Errorf("invalid input"))
}
ps.eventName = events[index-1]
} else {
ps.eventName = events[0]
}
registry.RecordProcessingStageChange("scanning", ps.eventName)
}
procLog.Info("processing started", logging.KeyEvent, ps.eventName)
//Get all present resolutions
dirs, err := ps.GetResolutions()
if err != nil {
return fail(fmt.Errorf("Failed to get resolutions: %w", err))
}
//Spawn a worker per resolution
ch := make(chan SegmentInfo, 100)
var wg sync.WaitGroup
for _, resolution := range dirs {
wg.Add(1)
go ps.ParseResolutionDirectory(resolution, ch, &wg)
}
go func() {
wg.Wait()
close(ch)
}()
segments, err := ps.AggregateSegmentInfo(ch)
if err != nil {
return fail(fmt.Errorf("Failed to aggregate segment info: %w", err))
}
registry.RecordProcessingSegmentsFound(len(segments))
aggFile, err := ps.WriteConcatFile(segments)
if err != nil {
return fail(fmt.Errorf("Failed to write concat file: %w", err))
}
// Feed info to ffmpeg to stitch files together
outPath := ps.config.GetProcessOutputPath(ps.eventName)
if err := utils.EnsureDir(outPath); err != nil {
return fail(fmt.Errorf("failed to create output directory: %w", err))
}
registry.RecordProcessingStageChange("running", ps.eventName)
concatErr := ps.RunFFmpeg(aggFile, outPath)
if concatErr != nil {
return fail(concatErr)
}
procLog.Info("processing completed", logging.KeyEvent, ps.eventName, "segments", len(segments))
registry.RecordProcessingStageChange("completed", ps.eventName)
return nil
}
func (ps *ProcessingService) GetResolutions() ([]string, error) {
eventPath := ps.config.GetNASEventPath(ps.eventName)
dirs, err := os.ReadDir(eventPath)
if err != nil {
return nil, fmt.Errorf("failed to read source directory %s: %w", eventPath, err)
}
re := regexp.MustCompile(`^\d+p$`)
var resolutions []string
for _, dir := range dirs {
if dir.IsDir() && re.MatchString(dir.Name()) {
resolutions = append(resolutions, dir.Name())
}
}
return resolutions, nil
}
func (ps *ProcessingService) ParseResolutionDirectory(resolution string, ch chan<- SegmentInfo, wg *sync.WaitGroup) {
defer wg.Done()
resolutionPath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), resolution)
files, err := os.ReadDir(resolutionPath)
if err != nil {
log.Printf("Failed to read resolution directory %s: %v", resolutionPath, err)
return
}
for _, file := range files {
if !file.IsDir() {
if !strings.HasSuffix(strings.ToLower(file.Name()), ".ts") {
continue
}
no, err := strconv.Atoi(file.Name()[6:10])
if err != nil {
log.Printf("Failed to parse segment number: %v", err)
continue
}
ch <- SegmentInfo{
Name: file.Name(),
SeqNo: no,
Resolution: resolution,
}
}
}
}
func (ps *ProcessingService) AggregateSegmentInfo(ch <-chan SegmentInfo) (map[int]SegmentInfo, error) {
segmentMap := make(map[int]SegmentInfo)
rank := map[string]int{
"1080p": 1,
"720p": 2,
"540p": 3,
"480p": 4,
"450p": 5,
"360p": 6,
"270p": 7,
"240p": 8,
}
for segment := range ch {
fmt.Printf("Received segment %s in resolution %s \n", segment.Name, segment.Resolution)
logging.MustGetFactory().Processing.Debug("segment received", logging.KeySegment, segment.Name, logging.KeyResolution, segment.Resolution)
current, exists := segmentMap[segment.SeqNo]
if !exists || rank[segment.Resolution] > rank[current.Resolution] {
segmentMap[segment.SeqNo] = segment
}
}
return segmentMap, nil
}
func (ps *ProcessingService) WriteConcatFile(segmentMap map[int]SegmentInfo) (string, error) {
concatPath := ps.config.GetProcessOutputPath(ps.eventName)
if err := utils.EnsureDir(concatPath); err != nil {
return "", fmt.Errorf("failed to create directories for concat path: %w", err)
}
concatFilePath := utils.SafeJoin(concatPath, ps.eventName+".txt")
f, err := os.Create(concatFilePath)
if err != nil {
return "", fmt.Errorf("failed to create concat file: %w", err)
}
defer f.Close()
// Sort keys to preserve order
keys := make([]int, 0, len(segmentMap))
for k := range segmentMap {
keys = append(keys, k)
}
sort.Ints(keys)
for _, seq := range keys {
segment := segmentMap[seq]
filePath := utils.SafeJoin(ps.config.GetNASEventPath(ps.eventName), segment.Resolution, segment.Name)
line := fmt.Sprintf("file '%s'\n", filePath)
if _, err := f.WriteString(line); err != nil {
return "", fmt.Errorf("failed to write to concat file: %w", err)
}
}
return concatFilePath, nil
}
func (ps *ProcessingService) getFFmpegPath() (string, error) {
// First try the configured path
configuredPath := ps.config.Processing.FFmpegPath
if configuredPath != "" {
// Check if it's just the command name or a full path
if filepath.IsAbs(configuredPath) {
return configuredPath, nil
}
// Try to find it in PATH
if fullPath, err := exec.LookPath(configuredPath); err == nil {
return fullPath, nil
}
}
// Fallback: try local bin directory
var baseDir string
exePath, err := os.Executable()
if err == nil {
baseDir = filepath.Dir(exePath)
} else {
baseDir, err = os.Getwd()
if err != nil {
return "", err
}
}
ffmpeg := utils.SafeJoin(baseDir, "bin", "ffmpeg")
if runtime.GOOS == "windows" {
ffmpeg += ".exe"
}
if utils.PathExists(ffmpeg) {
return ffmpeg, nil
}
// Try current working directory
cwd, err := os.Getwd()
if err != nil {
return "", err
}
ffmpeg = utils.SafeJoin(cwd, "bin", "ffmpeg")
if runtime.GOOS == "windows" {
ffmpeg += ".exe"
}
if utils.PathExists(ffmpeg) {
return ffmpeg, nil
}
return "", fmt.Errorf("FFmpeg not found. Please install FFmpeg or set FFMPEG_PATH environment variable")
}
func (ps *ProcessingService) RunFFmpeg(inputPath, outputPath string) error {
fmt.Println("Running ffmpeg...")
fileOutPath := utils.SafeJoin(outputPath, ps.eventName+".mp4")
fmt.Println("Input path:", inputPath)
fmt.Println("Output path:", fileOutPath)
logging.MustGetFactory().Processing.Info("running ffmpeg", "input", inputPath, "output", fileOutPath)
path, err := ps.getFFmpegPath()
if err != nil {
logging.MustGetFactory().Processing.Error("failed to find ffmpeg", logging.KeyError, err.Error())
return fmt.Errorf("failed to find FFmpeg: %w", err)
}
cmd := exec.Command(path, "-f", "concat", "-safe", "0", "-i", inputPath, "-c", "copy", fileOutPath)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
logging.MustGetFactory().Processing.Error("ffmpeg run failed", logging.KeyError, err.Error())
return fmt.Errorf("failed to run ffmpeg: %w", err)
}
fmt.Println("FFmpeg completed successfully")
logging.MustGetFactory().Processing.Info("ffmpeg completed successfully", "output", fileOutPath)
return nil
}

View File

@ -0,0 +1,384 @@
package processing
import (
"m3u8-downloader/pkg/config"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
)
func createTestConfig(tempDir string) *config.Config {
return &config.Config{
Core: config.CoreConfig{
WorkerCount: 2,
RefreshDelay: 1 * time.Second,
},
NAS: config.NASConfig{
OutputPath: filepath.Join(tempDir, "nas"),
Username: "testuser",
Password: "testpass",
Timeout: 10 * time.Second,
RetryLimit: 2,
EnableTransfer: false, // Disable to avoid NAS connection
},
Processing: config.ProcessingConfig{
Enabled: true,
AutoProcess: true,
WorkerCount: 1,
FFmpegPath: "echo", // Use echo command for testing
},
Paths: config.PathsConfig{
LocalOutput: filepath.Join(tempDir, "data"),
ProcessOutput: filepath.Join(tempDir, "out"),
ManifestDir: filepath.Join(tempDir, "data"),
PersistenceFile: filepath.Join(tempDir, "queue.json"),
},
}
}
func TestNewProcessingService_Success(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
cfg.NAS.EnableTransfer = false // Disable NAS to avoid connection
// We can't test actual NAS connection, so we'll skip the constructor test
// that requires NAS connectivity. Instead, test the configuration handling.
if cfg.Processing.FFmpegPath != "echo" {
t.Errorf("Expected FFmpegPath='echo', got '%s'", cfg.Processing.FFmpegPath)
}
}
func TestNewProcessingService_NilConfig(t *testing.T) {
_, err := NewProcessingService("test-event", nil)
if err == nil {
t.Error("Expected error for nil config")
}
if !strings.Contains(err.Error(), "configuration is required") {
t.Errorf("Expected 'configuration is required' error, got: %v", err)
}
}
func TestProcessingService_GetEventDirs(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
// Create mock NAS directory structure
nasDir := cfg.NAS.OutputPath
os.MkdirAll(filepath.Join(nasDir, "event1"), 0755)
os.MkdirAll(filepath.Join(nasDir, "event2"), 0755)
os.MkdirAll(filepath.Join(nasDir, "event3"), 0755)
// Create a file (should be ignored)
os.WriteFile(filepath.Join(nasDir, "not_a_dir.txt"), []byte("test"), 0644)
ps := &ProcessingService{
config: cfg,
eventName: "", // Empty to test directory discovery
}
dirs, err := ps.GetEventDirs()
if err != nil {
t.Fatalf("GetEventDirs() failed: %v", err)
}
if len(dirs) != 3 {
t.Errorf("Expected 3 event directories, got %d", len(dirs))
}
expectedDirs := []string{"event1", "event2", "event3"}
for _, expected := range expectedDirs {
found := false
for _, actual := range dirs {
if actual == expected {
found = true
break
}
}
if !found {
t.Errorf("Expected to find directory '%s' in results: %v", expected, dirs)
}
}
}
func TestProcessingService_GetEventDirs_WithEventName(t *testing.T) {
cfg := createTestConfig("/tmp")
eventName := "specific-event"
ps := &ProcessingService{
config: cfg,
eventName: eventName,
}
dirs, err := ps.GetEventDirs()
if err != nil {
t.Fatalf("GetEventDirs() failed: %v", err)
}
if len(dirs) != 1 {
t.Errorf("Expected 1 directory, got %d", len(dirs))
}
if dirs[0] != eventName {
t.Errorf("Expected directory '%s', got '%s'", eventName, dirs[0])
}
}
func TestProcessingService_GetResolutions(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
eventName := "test-event"
// Create mock event directory with quality subdirectories
eventPath := filepath.Join(cfg.NAS.OutputPath, eventName)
os.MkdirAll(filepath.Join(eventPath, "1080p"), 0755)
os.MkdirAll(filepath.Join(eventPath, "720p"), 0755)
os.MkdirAll(filepath.Join(eventPath, "480p"), 0755)
os.MkdirAll(filepath.Join(eventPath, "not_resolution"), 0755) // Should be ignored
os.WriteFile(filepath.Join(eventPath, "file.txt"), []byte("test"), 0644) // Should be ignored
ps := &ProcessingService{
config: cfg,
eventName: eventName,
}
resolutions, err := ps.GetResolutions()
if err != nil {
t.Fatalf("GetResolutions() failed: %v", err)
}
expectedResolutions := []string{"1080p", "720p", "480p"}
if len(resolutions) != len(expectedResolutions) {
t.Errorf("Expected %d resolutions, got %d: %v", len(expectedResolutions), len(resolutions), resolutions)
}
for _, expected := range expectedResolutions {
found := false
for _, actual := range resolutions {
if actual == expected {
found = true
break
}
}
if !found {
t.Errorf("Expected to find resolution '%s' in results: %v", expected, resolutions)
}
}
}
func TestProcessingService_AggregateSegmentInfo(t *testing.T) {
ps := &ProcessingService{}
// Create test channel with segments
ch := make(chan SegmentInfo, 5)
// Add segments with different qualities for same sequence
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"}
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "1080p"} // Higher quality, should win
ch <- SegmentInfo{Name: "seg_1002.ts", SeqNo: 1002, Resolution: "480p"}
ch <- SegmentInfo{Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"}
ch <- SegmentInfo{Name: "seg_1001.ts", SeqNo: 1001, Resolution: "540p"} // Lower than 1080p, should not replace
close(ch)
segmentMap, err := ps.AggregateSegmentInfo(ch)
if err != nil {
t.Fatalf("AggregateSegmentInfo() failed: %v", err)
}
// Should have 3 unique sequence numbers
if len(segmentMap) != 3 {
t.Errorf("Expected 3 unique segments, got %d", len(segmentMap))
}
// Check sequence 1001 has the highest quality (1080p)
seg1001, exists := segmentMap[1001]
if !exists {
t.Fatal("Segment 1001 should exist")
}
if seg1001.Resolution != "1080p" {
t.Errorf("Expected segment 1001 to have resolution '1080p', got '%s'", seg1001.Resolution)
}
// Check sequence 1002 has 480p
seg1002, exists := segmentMap[1002]
if !exists {
t.Fatal("Segment 1002 should exist")
}
if seg1002.Resolution != "480p" {
t.Errorf("Expected segment 1002 to have resolution '480p', got '%s'", seg1002.Resolution)
}
// Check sequence 1003 has 1080p
seg1003, exists := segmentMap[1003]
if !exists {
t.Fatal("Segment 1003 should exist")
}
if seg1003.Resolution != "1080p" {
t.Errorf("Expected segment 1003 to have resolution '1080p', got '%s'", seg1003.Resolution)
}
}
func TestProcessingService_WriteConcatFile(t *testing.T) {
tempDir, err := os.MkdirTemp("", "processing_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
cfg := createTestConfig(tempDir)
eventName := "test-event"
ps := &ProcessingService{
config: cfg,
eventName: eventName,
}
// Create test segment map
segmentMap := map[int]SegmentInfo{
1003: {Name: "seg_1003.ts", SeqNo: 1003, Resolution: "1080p"},
1001: {Name: "seg_1001.ts", SeqNo: 1001, Resolution: "720p"},
1002: {Name: "seg_1002.ts", SeqNo: 1002, Resolution: "1080p"},
}
concatFilePath, err := ps.WriteConcatFile(segmentMap)
if err != nil {
t.Fatalf("WriteConcatFile() failed: %v", err)
}
// Verify file was created
if _, err := os.Stat(concatFilePath); os.IsNotExist(err) {
t.Fatalf("Concat file was not created: %s", concatFilePath)
}
// Read and verify content
content, err := os.ReadFile(concatFilePath)
if err != nil {
t.Fatalf("Failed to read concat file: %v", err)
}
contentStr := string(content)
lines := strings.Split(strings.TrimSpace(contentStr), "\n")
if len(lines) != 3 {
t.Errorf("Expected 3 lines in concat file, got %d", len(lines))
}
// Verify segments are sorted by sequence number
expectedOrder := []string{"seg_1001.ts", "seg_1002.ts", "seg_1003.ts"}
for i, line := range lines {
if !strings.Contains(line, expectedOrder[i]) {
t.Errorf("Line %d should contain '%s', got: %s", i, expectedOrder[i], line)
}
if !strings.HasPrefix(line, "file '") {
t.Errorf("Line %d should start with 'file ', got: %s", i, line)
}
}
}
func TestProcessingService_getFFmpegPath(t *testing.T) {
cfg := createTestConfig("/tmp")
tests := []struct {
name string
ffmpegPath string
shouldFind bool
expectedError string
}{
{
name: "echo command (should be found in PATH)",
ffmpegPath: "echo",
shouldFind: true,
},
{
name: "absolute path test",
ffmpegPath: func() string {
if runtime.GOOS == "windows" {
return "C:\\Windows\\System32\\cmd.exe"
}
return "/bin/echo"
}(),
shouldFind: true,
},
{
name: "nonexistent command",
ffmpegPath: "nonexistent_ffmpeg_command_12345",
shouldFind: false,
expectedError: "FFmpeg not found",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
testCfg := *cfg
testCfg.Processing.FFmpegPath = tt.ffmpegPath
ps := &ProcessingService{
config: &testCfg,
eventName: "test",
}
path, err := ps.getFFmpegPath()
if tt.shouldFind {
if err != nil {
t.Errorf("Expected to find FFmpeg, but got error: %v", err)
}
if path == "" {
t.Error("Expected non-empty path")
}
} else {
if err == nil {
t.Error("Expected error for nonexistent FFmpeg")
}
if tt.expectedError != "" && !strings.Contains(err.Error(), tt.expectedError) {
t.Errorf("Expected error containing '%s', got: %v", tt.expectedError, err)
}
}
})
}
}
func TestSegmentInfo_Structure(t *testing.T) {
segment := SegmentInfo{
Name: "test_segment.ts",
SeqNo: 1001,
Resolution: "1080p",
}
if segment.Name != "test_segment.ts" {
t.Errorf("Expected Name='test_segment.ts', got '%s'", segment.Name)
}
if segment.SeqNo != 1001 {
t.Errorf("Expected SeqNo=1001, got %d", segment.SeqNo)
}
if segment.Resolution != "1080p" {
t.Errorf("Expected Resolution='1080p', got '%s'", segment.Resolution)
}
}
func TestProcessJob_Structure(t *testing.T) {
job := ProcessJob{
EventName: "test-event",
}
if job.EventName != "test-event" {
t.Errorf("Expected EventName='test-event', got '%s'", job.EventName)
}
}

5
pkg/processing/types.go Normal file
View File

@ -0,0 +1,5 @@
package processing
type ProcessJob struct {
EventName string
}

View File

@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/logging"
"os" "os"
"sync" "sync"
"time" "time"
@ -41,6 +42,7 @@ func (cs *CleanupService) Start(ctx context.Context) error {
} }
log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize) log.Printf("Cleanup service started (retention: %v, batch: %d)", cs.config.RetentionPeriod, cs.config.BatchSize)
logging.MustGetFactory().Transfer.Info("cleanup service started", "retention", cs.config.RetentionPeriod.String(), "batch_size", cs.config.BatchSize)
ticker := time.NewTicker(cs.config.CheckInterval) ticker := time.NewTicker(cs.config.CheckInterval)
defer ticker.Stop() defer ticker.Stop()
@ -69,7 +71,6 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
if batchSize > len(cs.pendingFiles) { if batchSize > len(cs.pendingFiles) {
batchSize = len(cs.pendingFiles) batchSize = len(cs.pendingFiles)
} }
cs.mu.Unlock()
log.Printf("Executing cleanup batch (size: %d)", batchSize) log.Printf("Executing cleanup batch (size: %d)", batchSize)
@ -98,6 +99,7 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
} }
log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors)) log.Printf("Cleanup batch completed (cleaned: %d, errors: %d)", cleanedCount, len(errors))
logging.MustGetFactory().Transfer.Info("cleanup batch completed", "cleaned", cleanedCount, "errors", len(errors))
if len(errors) > 0 { if len(errors) > 0 {
for i, err := range errors { for i, err := range errors {
@ -106,6 +108,7 @@ func (cs *CleanupService) ExecuteCleanup(ctx context.Context) error {
break break
} }
log.Printf("Error: %v", err) log.Printf("Error: %v", err)
logging.MustGetFactory().Transfer.Warn("cleanup error", logging.KeyError, err.Error())
} }
} }

View File

@ -3,39 +3,29 @@ package transfer
import ( import (
"context" "context"
"fmt" "fmt"
"io"
"log" "log"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas"
"os" "os"
"path/filepath" "path/filepath"
) )
type NASTransfer struct { func TransferFile(nt *nas.NASService, ctx context.Context, item *TransferItem) error {
config NASConfig destPath := filepath.Join(nt.Config.Path, item.DestinationPath)
connected bool
}
func NewNASTransfer(config NASConfig) *NASTransfer {
return &NASTransfer{
config: config,
}
}
func (nt *NASTransfer) TransferFile(ctx context.Context, item *TransferItem) error {
destPath := filepath.Join(nt.config.Path, item.DestinationPath)
destDir := filepath.Dir(destPath) destDir := filepath.Dir(destPath)
if err := nt.ensureDirectoryExists(destDir); err != nil { if err := nt.EnsureDirectoryExists(destDir); err != nil {
return fmt.Errorf("Failed to create directory %s: %w", destDir, err) return fmt.Errorf("Failed to create directory %s: %w", destDir, err)
} }
transferCtx, cancel := context.WithTimeout(ctx, nt.config.Timeout) transferCtx, cancel := context.WithTimeout(ctx, nt.Config.Timeout)
defer cancel() defer cancel()
if err := nt.copyFile(transferCtx, item.SourcePath, destPath); err != nil { if err := nt.CopyFile(transferCtx, item.SourcePath, destPath); err != nil {
return fmt.Errorf("Failed to copy file %s to %s: %w", item.SourcePath, destPath, err) return fmt.Errorf("Failed to copy file %s to %s: %w", item.SourcePath, destPath, err)
} }
if nt.config.VerifySize { if nt.Config.VerifySize {
if err := nt.VerifyTransfer(item.SourcePath, destPath); err != nil { if err := nt.VerifyTransfer(item.SourcePath, destPath); err != nil {
os.Remove(destPath) os.Remove(destPath)
return fmt.Errorf("Failed to verify transfer: %w", err) return fmt.Errorf("Failed to verify transfer: %w", err)
@ -43,82 +33,7 @@ func (nt *NASTransfer) TransferFile(ctx context.Context, item *TransferItem) err
} }
log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath) log.Printf("File transfer completed: %s -> %s", item.SourcePath, destPath)
logging.MustGetFactory().Transfer.Debug("nas copy completed", logging.KeyFile, item.SourcePath, "dest", destPath)
return nil return nil
} }
func (nt *NASTransfer) copyFile(ctx context.Context, srcPath, destPath string) error {
src, err := os.Open(srcPath)
if err != nil {
return fmt.Errorf("Failed to open source file: %w", err)
}
defer src.Close()
dest, err := os.Create(destPath)
if err != nil {
return fmt.Errorf("Failed to create destination file: %w", err)
}
defer dest.Close()
done := make(chan error, 1)
go func() {
_, err := io.Copy(dest, src)
done <- err
}()
select {
case <-ctx.Done():
return ctx.Err()
case err := <-done:
if err != nil {
return err
}
return dest.Sync()
}
}
func (nt *NASTransfer) VerifyTransfer(srcPath, destPath string) error {
srcInfo, err := os.Stat(srcPath)
if err != nil {
return fmt.Errorf("Failed to stat source file: %w", err)
}
destInfo, err := os.Stat(destPath)
if err != nil {
return fmt.Errorf("Failed to stat destination file: %w", err)
}
if srcInfo.Size() != destInfo.Size() {
return fmt.Errorf("size mismatch: source=%d, dest=%d", srcInfo.Size(), destInfo.Size())
}
return nil
}
func (nt *NASTransfer) ensureDirectoryExists(path string) error {
if err := os.MkdirAll(path, 0755); err != nil {
return fmt.Errorf("Failed to create directory: %w", err)
}
return nil
}
func (nt *NASTransfer) TestConnection() error {
testFile := filepath.Join(nt.config.Path, ".connection_test")
f, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("Failed to create test file: %w", err)
}
f.Close()
os.Remove(testFile)
nt.connected = true
log.Printf("Connected to NAS at %s", nt.config.Path)
return nil
}
func (nt *NASTransfer) IsConnected() bool {
return nt.connected
}

View File

@ -6,6 +6,9 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"m3u8-downloader/pkg/nas"
"os" "os"
"sync" "sync"
"time" "time"
@ -15,7 +18,7 @@ type TransferQueue struct {
config QueueConfig config QueueConfig
items *PriorityQueue items *PriorityQueue
stats *QueueStats stats *QueueStats
nasTransfer *NASTransfer nasService *nas.NASService
cleanup *CleanupService cleanup *CleanupService
workers []chan TransferItem workers []chan TransferItem
mu sync.RWMutex mu sync.RWMutex
@ -48,7 +51,7 @@ func (pq *PriorityQueue) Pop() interface{} {
return item return item
} }
func NewTransferQueue(config QueueConfig, nasTransfer *NASTransfer, cleanup *CleanupService) *TransferQueue { func NewTransferQueue(config QueueConfig, nasTransfer *nas.NASService, cleanup *CleanupService) *TransferQueue {
pq := &PriorityQueue{} pq := &PriorityQueue{}
heap.Init(pq) heap.Init(pq)
@ -56,7 +59,7 @@ func NewTransferQueue(config QueueConfig, nasTransfer *NASTransfer, cleanup *Cle
config: config, config: config,
items: pq, items: pq,
stats: &QueueStats{}, stats: &QueueStats{},
nasTransfer: nasTransfer, nasService: nasTransfer,
cleanup: cleanup, cleanup: cleanup,
workers: make([]chan TransferItem, config.WorkerCount), workers: make([]chan TransferItem, config.WorkerCount),
} }
@ -73,7 +76,10 @@ func (tq *TransferQueue) Add(item TransferItem) error {
defer tq.mu.Unlock() defer tq.mu.Unlock()
if tq.items.Len() >= tq.config.MaxQueueSize { if tq.items.Len() >= tq.config.MaxQueueSize {
return fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize) err := fmt.Errorf("Queue is full (max size: %d)", tq.config.MaxQueueSize)
logging.MustGetFactory().Transfer.Error("queue full, dropping file", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
return err
} }
heap.Push(tq.items, &item) heap.Push(tq.items, &item)
@ -145,6 +151,24 @@ func (tq *TransferQueue) dispatchWork() {
} }
func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) { func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
// Check if file already exists on NAS before attempting transfer
if exists, err := tq.nasService.FileExists(item.DestinationPath, item.FileSize); err != nil {
log.Printf("Failed to check if file exists on NAS for %s: %v", item.SourcePath, err)
// Continue with transfer attempt on error
} else if exists {
log.Printf("File already exists on NAS, skipping transfer: %s", item.SourcePath)
item.Status = StatusCompleted
tq.stats.IncrementCompleted(item.FileSize)
// Schedule for cleanup
if tq.cleanup != nil {
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
log.Printf("Failed to schedule cleanup for existing file %s: %v", item.SourcePath, err)
}
}
return
}
maxRetries := 3 maxRetries := 3
for attempt := 1; attempt <= maxRetries; attempt++ { for attempt := 1; attempt <= maxRetries; attempt++ {
@ -160,7 +184,7 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
} }
} }
err := tq.nasTransfer.TransferFile(ctx, &item) err := TransferFile(tq.nasService, ctx, &item)
if err == nil { if err == nil {
item.Status = StatusCompleted item.Status = StatusCompleted
tq.stats.IncrementCompleted(item.FileSize) tq.stats.IncrementCompleted(item.FileSize)
@ -168,9 +192,11 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
if tq.cleanup != nil { if tq.cleanup != nil {
if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil { if err := tq.cleanup.ScheduleCleanup(item.SourcePath); err != nil {
log.Printf("Failed to add file to cleanup list: %v", err) log.Printf("Failed to add file to cleanup list: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to schedule cleanup", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
} }
} }
log.Printf("File transfer completed: %s", item.SourcePath) log.Printf("File transfer completed: %s", item.SourcePath)
logging.MustGetFactory().Transfer.Info("file transfer completed", logging.KeyFile, item.SourcePath, logging.KeyBytes, item.FileSize)
return return
} }
@ -178,11 +204,14 @@ func (tq *TransferQueue) processItem(ctx context.Context, item TransferItem) {
item.RetryCount++ item.RetryCount++
log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err) log.Printf("File transfer failed: %s (attempt %d/%d): %v", item.SourcePath, item.RetryCount, maxRetries, err)
logging.MustGetFactory().Transfer.Warn("file transfer attempt failed", logging.KeyFile, item.SourcePath, logging.KeyAttempt, item.RetryCount, logging.KeyError, err.Error())
if attempt == maxRetries { if attempt == maxRetries {
item.Status = StatusFailed item.Status = StatusFailed
tq.stats.IncrementFailed() tq.stats.IncrementFailed()
log.Printf("Transfer permanently failed for file: %s", item.SourcePath) log.Printf("Transfer permanently failed for file: %s", item.SourcePath)
logging.MustGetFactory().Transfer.Error("transfer permanently failed", logging.KeyFile, item.SourcePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(item.SourcePath, err)
return return
} }
} }
@ -206,10 +235,12 @@ func (tq *TransferQueue) SaveState() error {
"timestamp": time.Now(), "timestamp": time.Now(),
}, "", " ") }, "", " ")
if err != nil { if err != nil {
logging.MustGetFactory().Transfer.Error("failed to marshal queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to marshal queue state: %w", err) return fmt.Errorf("Failed to marshal queue state: %w", err)
} }
if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil { if err := os.WriteFile(tq.config.PersistencePath, data, 0644); err != nil {
logging.MustGetFactory().Transfer.Error("failed to save queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to save queue state: %w", err) return fmt.Errorf("Failed to save queue state: %w", err)
} }
@ -222,6 +253,7 @@ func (tq *TransferQueue) LoadState() error {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return nil return nil
} }
logging.MustGetFactory().Transfer.Error("failed to load queue state", logging.KeyError, err.Error())
return fmt.Errorf("Failed to load queue state: %w", err) return fmt.Errorf("Failed to load queue state: %w", err)
} }

View File

@ -5,6 +5,13 @@ import (
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/constants" "m3u8-downloader/pkg/constants"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
nas2 "m3u8-downloader/pkg/nas"
"m3u8-downloader/pkg/utils"
"os"
"path/filepath"
"strings"
"sync" "sync"
"time" "time"
) )
@ -12,47 +19,60 @@ import (
type TransferService struct { type TransferService struct {
watcher *FileWatcher watcher *FileWatcher
queue *TransferQueue queue *TransferQueue
nas *NASTransfer nas *nas2.NASService
cleanup *CleanupService cleanup *CleanupService
stats *QueueStats stats *QueueStats
} }
func NewTrasferService(outputDir string) (*TransferService, error) { func NewTrasferService(outputDir string, eventName string) (*TransferService, error) {
nasConfig := NASConfig{ cfg := constants.MustGetConfig()
Path: constants.NASPath,
Username: constants.NASUsername, nasConfig := nas2.NASConfig{
Password: constants.NASPassword, Path: outputDir,
Timeout: constants.TransferTimeout, Username: cfg.NAS.Username,
RetryLimit: constants.TransferRetryLimit, Password: cfg.NAS.Password,
Timeout: cfg.NAS.Timeout,
RetryLimit: cfg.NAS.RetryLimit,
VerifySize: true, VerifySize: true,
} }
nas := NewNASTransfer(nasConfig) nas := nas2.NewNASService(nasConfig)
if err := nas.TestConnection(); err != nil { if err := nas.TestConnection(); err != nil {
return nil, fmt.Errorf("Failed to connect to NAS: %w", err) logging.MustGetFactory().Transfer.Error("failed to connect to NAS", logging.KeyError, err.Error())
return nil, fmt.Errorf("failed to connect to NAS: %w", err)
} }
logging.MustGetFactory().Transfer.Info("connected to NAS", "path", outputDir)
health.MustGetRegistry().RecordNASConnectivity(true)
cleanupConfig := CleanupConfig{ cleanupConfig := CleanupConfig{
Enabled: constants.CleanupAfterTransfer, Enabled: cfg.Cleanup.AfterTransfer,
RetentionPeriod: constants.RetainLocalHours, RetentionPeriod: time.Duration(cfg.Cleanup.RetainHours) * time.Hour,
BatchSize: constants.CleanupBatchSize, BatchSize: cfg.Cleanup.BatchSize,
CheckInterval: constants.FileSettlingDelay, CheckInterval: cfg.Transfer.FileSettlingDelay,
} }
cleanup := NewCleanupService(cleanupConfig) cleanup := NewCleanupService(cleanupConfig)
queueConfig := QueueConfig{ queueConfig := QueueConfig{
WorkerCount: constants.TransferWorkerCount, WorkerCount: cfg.Transfer.WorkerCount,
PersistencePath: constants.PersistencePath, PersistencePath: cfg.Paths.PersistenceFile,
MaxQueueSize: constants.TransferQueueSize, MaxQueueSize: cfg.Transfer.QueueSize,
BatchSize: constants.BatchSize, BatchSize: cfg.Transfer.BatchSize,
} }
queue := NewTransferQueue(queueConfig, nas, cleanup) queue := NewTransferQueue(queueConfig, nas, cleanup)
watcher, err := NewFileWatcher(outputDir, queue) // Create local output directory if it doesn't exist
if err != nil { localOutputPath := cfg.GetEventPath(eventName)
return nil, fmt.Errorf("Failed to create file watcher: %w", err) if err := utils.EnsureDir(localOutputPath); err != nil {
return nil, fmt.Errorf("failed to create local output directory: %w", err)
} }
watcher, err := NewFileWatcher(localOutputPath, queue, cfg.Transfer.FileSettlingDelay)
if err != nil {
return nil, fmt.Errorf("failed to create file watcher: %w", err)
}
health.MustGetRegistry().AttachTransfer(queue.stats, queue.GetQueueSize, cleanup.GetPendingCount, nas.IsConnected)
return &TransferService{ return &TransferService{
watcher: watcher, watcher: watcher,
queue: queue, queue: queue,
@ -96,6 +116,7 @@ func (ts *TransferService) Start(ctx context.Context) error {
}() }()
log.Println("Transfer service started") log.Println("Transfer service started")
logging.MustGetFactory().Transfer.Info("transfer service started")
wg.Wait() wg.Wait()
return nil return nil
@ -115,6 +136,9 @@ func (ts *TransferService) reportStats(ctx context.Context) {
cleanupPending := ts.cleanup.GetPendingCount() cleanupPending := ts.cleanup.GetPendingCount()
log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending) log.Printf("Transfer Stats: Added: %d, Completed: %d, Failed: %d, Pending: %d, Bytes: %d, Queue Size: %d, Cleanup Pending: %d", added, completed, failed, pending, bytes, queueSize, cleanupPending)
logging.MustGetFactory().Transfer.Info("transfer stats",
"added", added, "completed", completed, "failed", failed, "pending", pending,
logging.KeyBytes, bytes, "queue_size", queueSize, "cleanup_pending", cleanupPending)
} }
} }
} }
@ -130,7 +154,119 @@ func (ts *TransferService) Shutdown(ctx context.Context) error {
return fmt.Errorf("Failed to force cleanup: %w", err) return fmt.Errorf("Failed to force cleanup: %w", err)
} }
// Disconnect from NAS
if err := ts.nas.Disconnect(); err != nil {
log.Printf("Warning: failed to disconnect from NAS: %v", err)
logging.MustGetFactory().Transfer.Warn("failed to disconnect from NAS", logging.KeyError, err.Error())
}
health.MustGetRegistry().RecordNASConnectivity(false)
log.Println("Transfer service shut down") log.Println("Transfer service shut down")
logging.MustGetFactory().Transfer.Info("transfer service shut down")
return nil return nil
} }
// QueueExistingFiles scans a directory for .ts files and queues them for transfer
func (ts *TransferService) QueueExistingFiles(localEventPath string) error {
cfg := constants.MustGetConfig()
log.Printf("Scanning for existing files in: %s", localEventPath)
var fileCount, alreadyTransferred, scheduledForCleanup int
// Extract event name from path for NAS destination
eventName := filepath.Base(localEventPath)
err := filepath.Walk(localEventPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
log.Printf("Error accessing path %s: %v", path, err)
return nil // Continue walking
}
// Only process .ts files
if !info.IsDir() && strings.HasSuffix(strings.ToLower(info.Name()), ".ts") {
// Extract resolution from directory path
resolution := ts.extractResolutionFromPath(path)
// Get relative path from event directory
relPath, err := filepath.Rel(localEventPath, path)
if err != nil {
log.Printf("Failed to get relative path for %s: %v", path, err)
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, path, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(path, err)
return nil
}
// Build NAS destination path (eventName/relPath)
nasDestPath := filepath.Join(eventName, relPath)
// Check if file already exists on NAS with matching size
exists, err := ts.nas.FileExists(nasDestPath, info.Size())
if err != nil {
log.Printf("Failed to check NAS file existence for %s: %v", path, err)
// Continue with transfer attempt on error
} else if exists {
log.Printf("File already exists on NAS: %s (%s, %d bytes)", path, resolution, info.Size())
alreadyTransferred++
// Schedule for cleanup if cleanup is enabled
if cfg.Cleanup.AfterTransfer {
if err := ts.cleanup.ScheduleCleanup(path); err != nil {
log.Printf("Failed to schedule cleanup for already-transferred file %s: %v", path, err)
} else {
scheduledForCleanup++
}
}
return nil // Skip queuing this file
}
// Create transfer item
item := TransferItem{
ID: ts.generateTransferID(),
SourcePath: path,
DestinationPath: nasDestPath,
Resolution: resolution,
Timestamp: info.ModTime(),
Status: StatusPending,
FileSize: info.Size(),
}
// Add to queue
if err := ts.queue.Add(item); err != nil {
log.Printf("Failed to queue file %s: %v", path, err)
logging.MustGetFactory().Transfer.Error("failed to queue file", logging.KeyFile, path, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(path, err)
} else {
log.Printf("Queued file: %s (%s, %d bytes)", path, resolution, info.Size())
fileCount++
}
}
return nil
})
if err != nil {
return fmt.Errorf("failed to walk directory: %w", err)
}
log.Printf("File scan completed - Queued: %d, Already transferred: %d, Scheduled for cleanup: %d",
fileCount, alreadyTransferred, scheduledForCleanup)
return nil
}
func (ts *TransferService) extractResolutionFromPath(filePath string) string {
dir := filepath.Dir(filePath)
parts := strings.Split(dir, string(filepath.Separator))
for _, part := range parts {
if strings.HasSuffix(part, "p") {
return part
}
}
return "unknown"
}
func (ts *TransferService) generateTransferID() string {
return fmt.Sprintf("transfer_existing_%d", time.Now().UnixNano())
}

View File

@ -44,15 +44,6 @@ func (s TransferStatus) String() string {
} }
} }
type NASConfig struct {
Path string
Username string
Password string
Timeout time.Duration
RetryLimit int
VerifySize bool
}
type QueueConfig struct { type QueueConfig struct {
WorkerCount int WorkerCount int
PersistencePath string PersistencePath string

View File

@ -4,6 +4,8 @@ import (
"context" "context"
"fmt" "fmt"
"log" "log"
"m3u8-downloader/pkg/health"
"m3u8-downloader/pkg/logging"
"math/rand" "math/rand"
"os" "os"
"path/filepath" "path/filepath"
@ -23,7 +25,7 @@ type FileWatcher struct {
mu sync.Mutex mu sync.Mutex
} }
func NewFileWatcher(outputDir string, queue *TransferQueue) (*FileWatcher, error) { func NewFileWatcher(outputDir string, queue *TransferQueue, settlingDelay time.Duration) (*FileWatcher, error) {
watcher, err := fsnotify.NewWatcher() watcher, err := fsnotify.NewWatcher()
if err != nil { if err != nil {
return nil, err return nil, err
@ -32,7 +34,7 @@ func NewFileWatcher(outputDir string, queue *TransferQueue) (*FileWatcher, error
outputDir: outputDir, outputDir: outputDir,
queue: queue, queue: queue,
watcher: watcher, watcher: watcher,
settingDelay: time.Second, settingDelay: settlingDelay,
pendingFiles: make(map[string]*time.Timer), pendingFiles: make(map[string]*time.Timer),
}, nil }, nil
} }
@ -45,11 +47,13 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
} }
log.Printf("Starting file watcher on %s", fw.outputDir) log.Printf("Starting file watcher on %s", fw.outputDir)
logging.MustGetFactory().Transfer.Info("file watcher started", "dir", fw.outputDir)
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
log.Println("File watcher shutting down...") log.Println("File watcher shutting down...")
logging.MustGetFactory().Transfer.Info("file watcher shutting down")
return ctx.Err() return ctx.Err()
case event, ok := <-fw.watcher.Events: case event, ok := <-fw.watcher.Events:
@ -63,6 +67,7 @@ func (fw *FileWatcher) Start(ctx context.Context) error {
return fmt.Errorf("Watcher errors channel closed") return fmt.Errorf("Watcher errors channel closed")
} }
log.Printf("Watcher error: %v", err) log.Printf("Watcher error: %v", err)
logging.MustGetFactory().Transfer.Error("watcher error", logging.KeyError, err.Error())
} }
} }
} }
@ -144,6 +149,8 @@ func (fw *FileWatcher) processFile(filePath string) {
info, err := os.Stat(filePath) info, err := os.Stat(filePath)
if err != nil { if err != nil {
log.Printf("Failed to stat file %s: %v", filePath, err) log.Printf("Failed to stat file %s: %v", filePath, err)
logging.MustGetFactory().Transfer.Warn("failed to stat file", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
return return
} }
@ -152,6 +159,8 @@ func (fw *FileWatcher) processFile(filePath string) {
relPath, err := filepath.Rel(fw.outputDir, filePath) relPath, err := filepath.Rel(fw.outputDir, filePath)
if err != nil { if err != nil {
log.Printf("Failed to get relative path for file %s: %v", filePath, err) log.Printf("Failed to get relative path for file %s: %v", filePath, err)
logging.MustGetFactory().Transfer.Warn("failed to get relative path", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
return return
} }
@ -167,8 +176,11 @@ func (fw *FileWatcher) processFile(filePath string) {
if err := fw.queue.Add(item); err != nil { if err := fw.queue.Add(item); err != nil {
log.Printf("Failed to add file to queue: %v", err) log.Printf("Failed to add file to queue: %v", err)
logging.MustGetFactory().Transfer.Error("failed to add file to queue", logging.KeyFile, filePath, logging.KeyError, err.Error())
health.MustGetRegistry().RecordTransferError(filePath, err)
} else { } else {
log.Printf("Added file to queue: %s", filePath) log.Printf("Added file to queue: %s", filePath)
logging.MustGetFactory().Transfer.Debug("file queued", logging.KeyFile, filePath)
} }
} }

62
pkg/utils/paths.go Normal file
View File

@ -0,0 +1,62 @@
package utils
import (
"fmt"
"os"
"path/filepath"
"strings"
)
func SafeJoin(base string, elements ...string) string {
path := filepath.Join(append([]string{base}, elements...)...)
return filepath.Clean(path)
}
func EnsureDir(path string) error {
if err := os.MkdirAll(path, 0755); err != nil {
return fmt.Errorf("failed to create directory %s: %w", path, err)
}
return nil
}
func PathExists(path string) bool {
_, err := os.Stat(path)
return !os.IsNotExist(err)
}
func IsValidPath(path string) bool {
if path == "" {
return false
}
return !strings.ContainsAny(path, "<>:\"|?*")
}
func NormalizePath(path string) string {
return filepath.Clean(strings.ReplaceAll(path, "\\", string(filepath.Separator)))
}
func GetRelativePath(basePath, targetPath string) (string, error) {
rel, err := filepath.Rel(basePath, targetPath)
if err != nil {
return "", fmt.Errorf("failed to get relative path: %w", err)
}
return rel, nil
}
func ValidateWritablePath(path string) error {
dir := filepath.Dir(path)
if err := EnsureDir(dir); err != nil {
return err
}
testFile := filepath.Join(dir, ".write_test")
file, err := os.Create(testFile)
if err != nil {
return fmt.Errorf("path %s is not writable: %w", dir, err)
}
file.Close()
os.Remove(testFile)
return nil
}

235
pkg/utils/paths_test.go Normal file
View File

@ -0,0 +1,235 @@
package utils
import (
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
func TestSafeJoin(t *testing.T) {
tests := []struct {
name string
base string
elements []string
want string
}{
{
name: "basic join",
base: "data",
elements: []string{"events", "test-event"},
want: filepath.Join("data", "events", "test-event"),
},
{
name: "empty elements",
base: "data",
elements: []string{},
want: "data",
},
{
name: "with path separators",
base: "data/events",
elements: []string{"test-event", "1080p"},
want: filepath.Join("data", "events", "test-event", "1080p"),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := SafeJoin(tt.base, tt.elements...)
if got != tt.want {
t.Errorf("SafeJoin() = %v, want %v", got, tt.want)
}
})
}
}
func TestEnsureDir(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
testPath := filepath.Join(tempDir, "test", "nested", "directory")
// Test creating nested directories
err = EnsureDir(testPath)
if err != nil {
t.Errorf("EnsureDir() failed: %v", err)
}
// Verify directory was created
if _, err := os.Stat(testPath); os.IsNotExist(err) {
t.Errorf("Directory was not created: %s", testPath)
}
// Test with existing directory (should not fail)
err = EnsureDir(testPath)
if err != nil {
t.Errorf("EnsureDir() failed on existing directory: %v", err)
}
}
func TestPathExists(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Test existing path
if !PathExists(tempDir) {
t.Errorf("PathExists() should return true for existing path: %s", tempDir)
}
// Test non-existing path
nonExistentPath := filepath.Join(tempDir, "does-not-exist")
if PathExists(nonExistentPath) {
t.Errorf("PathExists() should return false for non-existent path: %s", nonExistentPath)
}
// Test with file
testFile := filepath.Join(tempDir, "test.txt")
f, err := os.Create(testFile)
if err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
f.Close()
if !PathExists(testFile) {
t.Errorf("PathExists() should return true for existing file: %s", testFile)
}
}
func TestIsValidPath(t *testing.T) {
tests := []struct {
name string
path string
want bool
}{
{"empty path", "", false},
{"valid path", "data/events/test", true},
{"path with colon", "data:events", false},
{"path with pipe", "data|events", false},
{"path with question mark", "data?events", false},
{"path with asterisk", "data*events", false},
{"path with quotes", "data\"events", false},
{"path with angle brackets", "data<events>", false},
{"normal windows path", "C:\\data\\events", true}, // Windows path separators are actually OK
{"unix path", "/home/user/data", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := IsValidPath(tt.path)
if got != tt.want {
t.Errorf("IsValidPath(%q) = %v, want %v", tt.path, got, tt.want)
}
})
}
}
func TestNormalizePath(t *testing.T) {
tests := []struct {
name string
path string
want string
}{
{
name: "windows backslashes",
path: "data\\events\\test",
want: filepath.Join("data", "events", "test"),
},
{
name: "unix forward slashes",
path: "data/events/test",
want: filepath.Join("data", "events", "test"),
},
{
name: "mixed slashes",
path: "data\\events/test\\file",
want: filepath.Join("data", "events", "test", "file"),
},
{
name: "redundant separators",
path: "data//events\\\\test",
want: filepath.Join("data", "events", "test"),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := NormalizePath(tt.path)
if got != tt.want {
t.Errorf("NormalizePath(%q) = %q, want %q", tt.path, got, tt.want)
}
})
}
}
func TestGetRelativePath(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
basePath := tempDir
targetPath := filepath.Join(tempDir, "subdir", "file.txt")
rel, err := GetRelativePath(basePath, targetPath)
if err != nil {
t.Errorf("GetRelativePath() failed: %v", err)
}
expected := filepath.Join("subdir", "file.txt")
if rel != expected {
t.Errorf("GetRelativePath() = %q, want %q", rel, expected)
}
// Test with invalid paths
_, err = GetRelativePath("", "")
if err == nil {
t.Error("GetRelativePath() should fail with empty paths")
}
}
func TestValidateWritablePath(t *testing.T) {
tempDir, err := os.MkdirTemp("", "utils_test_*")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)
// Test writable path
writablePath := filepath.Join(tempDir, "test", "file.txt")
err = ValidateWritablePath(writablePath)
if err != nil {
t.Errorf("ValidateWritablePath() failed for writable path: %v", err)
}
// Verify directory was created
dir := filepath.Dir(writablePath)
if _, err := os.Stat(dir); os.IsNotExist(err) {
t.Errorf("Directory should have been created: %s", dir)
}
// Test with read-only directory (if supported by OS)
if runtime.GOOS != "windows" { // Skip on Windows as it's more complex
readOnlyDir := filepath.Join(tempDir, "readonly")
os.MkdirAll(readOnlyDir, 0755)
os.Chmod(readOnlyDir, 0444) // Read-only
defer os.Chmod(readOnlyDir, 0755) // Restore permissions for cleanup
readOnlyPath := filepath.Join(readOnlyDir, "file.txt")
err = ValidateWritablePath(readOnlyPath)
if err == nil {
t.Error("ValidateWritablePath() should fail for read-only directory")
}
if !strings.Contains(err.Error(), "not writable") {
t.Errorf("Expected 'not writable' error, got: %v", err)
}
}
}

146
test_runner.go Normal file
View File

@ -0,0 +1,146 @@
package main
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
func main() {
fmt.Println("🧪 StreamRecorder Test Suite")
fmt.Println("============================")
startTime := time.Now()
// Set test environment to avoid interfering with real data
originalEnv := setupTestEnvironment()
defer restoreEnvironment(originalEnv)
// Create temporary directory for test data
tempDir, err := os.MkdirTemp("", "streamrecorder_test_*")
if err != nil {
fmt.Printf("❌ Failed to create temp directory: %v\n", err)
os.Exit(1)
}
defer func() {
fmt.Printf("🧹 Cleaning up test directory: %s\n", tempDir)
os.RemoveAll(tempDir)
}()
// Set test-specific environment variables
os.Setenv("LOCAL_OUTPUT_DIR", filepath.Join(tempDir, "data"))
os.Setenv("PROCESS_OUTPUT_DIR", filepath.Join(tempDir, "out"))
os.Setenv("ENABLE_NAS_TRANSFER", "false") // Disable NAS for tests
os.Setenv("PROCESSING_ENABLED", "false") // Disable processing that needs FFmpeg
fmt.Printf("📁 Using temporary directory: %s\n", tempDir)
fmt.Println()
// Run tests for each package
packages := []string{
"./pkg/config",
"./pkg/utils",
"./pkg/constants",
"./pkg/httpClient",
"./pkg/media",
"./pkg/processing",
}
var failedPackages []string
totalTests := 0
passedTests := 0
for _, pkg := range packages {
fmt.Printf("🔍 Testing package: %s\n", pkg)
cmd := exec.Command("go", "test", "-v", pkg)
cmd.Dir = "."
output, err := cmd.CombinedOutput()
outputStr := string(output)
// Count tests
testCount := strings.Count(outputStr, "=== RUN")
passCount := strings.Count(outputStr, "--- PASS:")
totalTests += testCount
passedTests += passCount
if err != nil {
fmt.Printf("❌ FAILED: %s (%d/%d tests passed)\n", pkg, passCount, testCount)
failedPackages = append(failedPackages, pkg)
// Show failure details
lines := strings.Split(outputStr, "\n")
for _, line := range lines {
if strings.Contains(line, "FAIL:") ||
strings.Contains(line, "Error:") ||
strings.Contains(line, "panic:") {
fmt.Printf(" %s\n", line)
}
}
} else {
fmt.Printf("✅ PASSED: %s (%d tests)\n", pkg, testCount)
}
fmt.Println()
}
// Print summary
duration := time.Since(startTime)
fmt.Println("📊 Test Summary")
fmt.Println("===============")
fmt.Printf("Total packages: %d\n", len(packages))
fmt.Printf("Passed packages: %d\n", len(packages)-len(failedPackages))
fmt.Printf("Failed packages: %d\n", len(failedPackages))
fmt.Printf("Total tests: %d\n", totalTests)
fmt.Printf("Passed tests: %d\n", passedTests)
fmt.Printf("Failed tests: %d\n", totalTests-passedTests)
fmt.Printf("Duration: %v\n", duration.Round(time.Millisecond))
if len(failedPackages) > 0 {
fmt.Println()
fmt.Println("❌ Failed packages:")
for _, pkg := range failedPackages {
fmt.Printf(" - %s\n", pkg)
}
os.Exit(1)
} else {
fmt.Println()
fmt.Println("🎉 All tests passed!")
}
}
func setupTestEnvironment() map[string]string {
// Save original environment variables that we'll modify
originalEnv := make(map[string]string)
envVars := []string{
"LOCAL_OUTPUT_DIR",
"PROCESS_OUTPUT_DIR",
"ENABLE_NAS_TRANSFER",
"PROCESSING_ENABLED",
"NAS_OUTPUT_PATH",
"FFMPEG_PATH",
"WORKER_COUNT",
}
for _, envVar := range envVars {
originalEnv[envVar] = os.Getenv(envVar)
}
return originalEnv
}
func restoreEnvironment(originalEnv map[string]string) {
fmt.Println("🔄 Restoring original environment...")
for envVar, originalValue := range originalEnv {
if originalValue == "" {
os.Unsetenv(envVar)
} else {
os.Setenv(envVar, originalValue)
}
}
}